Current Status
Paco Nathan edited this page Jan 17, 2014
·
34 revisions
- 2014-01-17 r0.1.1 supports distributed Py workflows, based on Pandas, NumPy, scikit-learn, etc.
- 2013-12-22 refactored to show the example monoid use
- 2013-12-17 resolved
reify()
high false positive rate using datrie in lieu of bloom filter - 2013-12-08 first draft for the tutorial and diagrams
- 2013-12-03 added logging, building tarball/container for Executor downloads via HDFS
- 2013-12-01 first complete run on Elastic Mesos with >1 workers in a hash ring
- 2013-11-30 two-phase synchronization to determine
UnitOfWork
task_queue completion - 2013-11-29 running on multiple shards (w/o long-polling yet to determine generation end)
- 2013-11-26 integrated the required Apache Mesos methods to manage a remote service
- 2013-11-23 first successful launch of customized scheduler/executor on Elastic Mesos
- 2013-11-21 running one master/one slave only (e.g., on a laptop)
- de-Ubuntu-fy the installer scripts, via Anaconda installations on the cluster
- download the tarball (or container) for executor/worker launch from HDFS
- serialize evicted Individuals to HDFS using Snakebite
- resume
UnitOfWork
iteration from a starting state checkpointed in HDFS - shard checkpoint to HDFS
- shard recovery from HDFS
- integrate GA classes+methods more closely with Pyevolve
- save/recover Framework state in Zookeeper
- make service port configurable / discover port availability / run 2+ services per slave