Skip to content

Current Status

Paco Nathan edited this page Jan 17, 2014 · 34 revisions
  • 2014-01-17 r0.1.1 supports distributed Py workflows, based on Pandas, NumPy, scikit-learn, etc.
  • 2013-12-22 refactored to show the example monoid use
  • 2013-12-17 resolved reify() high false positive rate using datrie in lieu of bloom filter
  • 2013-12-08 first draft for the tutorial and diagrams
  • 2013-12-03 added logging, building tarball/container for Executor downloads via HDFS
  • 2013-12-01 first complete run on Elastic Mesos with >1 workers in a hash ring
  • 2013-11-30 two-phase synchronization to determine UnitOfWork task_queue completion
  • 2013-11-29 running on multiple shards (w/o long-polling yet to determine generation end)
  • 2013-11-26 integrated the required Apache Mesos methods to manage a remote service
  • 2013-11-23 first successful launch of customized scheduler/executor on Elastic Mesos
  • 2013-11-21 running one master/one slave only (e.g., on a laptop)

TODO

  • de-Ubuntu-fy the installer scripts, via Anaconda installations on the cluster
  • download the tarball (or container) for executor/worker launch from HDFS
  • serialize evicted Individuals to HDFS using Snakebite
  • resume UnitOfWork iteration from a starting state checkpointed in HDFS
  • shard checkpoint to HDFS
  • shard recovery from HDFS
  • integrate GA classes+methods more closely with Pyevolve
  • save/recover Framework state in Zookeeper
  • make service port configurable / discover port availability / run 2+ services per slave