Support for Amazon Elastic Container Service?

[sthomp]

Just wondering if there's any plan to support Amazon ECS? I saw some discussion about potentially using docker containers for memory management. ECS would be a nice way to abstract away other details like selecting which server should run a given task.

[mistercrunch]

I need to talk to people in the data infrastructure team at Airbnb as to how we are planning on going about our array of workers. Clearly Celery was a first stab at it that works pretty great, and having the CeleryExecutor is an asset, but moving forward we need containment, and to reuse existing infrastructure. I don't know much about Yarn and its APIs, but I thought that would be the way to go. Seems like ECS is another option.

Airbnb will build and share a solution that works for us and hopefully for others, if your stack is different or take a different path (say we go Yarn and you want to go ECS), we'll collaborate to make sure we build an interface that works for both, but you'll have to implement that interface.

If you want to be part of the conversation on how to distribute tasks to workers I can reach out when the time comes.

[sthomp]

Definitely. Thanks!