Recommended workflow for Elastic Beanstalk

[ruuts]

Hi!

We are about to switch over to Elastic Beanstalk.
Currently we use delayed_job with a hacky method to instantiate workers.
You don't know when Amazon drops the (undocumented) support for post deploy scripts.

The right solution would be deploying a separate worker environment alongside the web environment.
Another important factor is that workers processes can auto scale separate from the web processes.

The worker environment works by:

1. Spawning a web server
2. And spawning a daemon that will poll for SQS messages.
3. When there is a SQS message it will send that message to a web server endpoint.
4. The endpoint processes the SQS message.
5. When the job is finished it will respond with a 200.
6. The polling process will than mark the actual SQS as finished.

How should the web server process the SQS message?
I can imagine instantiating Shoryuken manually and feeding it the message.

An alternative could be using the AWS sdk but I like a more abstracted version so that we are less tied to Amazon.

Thanks for helping out!

[jeromedalbert]

I'm also interested in this! The Deployment page on the wiki explains things very well for Heroku, but I am unsure about how to do something similar for AWS EB.

[Senjai]

I'm not sure what exactly you're asking.

How should the web server process the SQS message

Why not load the daemon within the environment of your application? Similar to how delayed job or resque works.

[farski]

Trying to use Shoryuken to process jobs coming from the webhook is probably not the right strategy. The Shoryuken server and the EB worker daemon are both trying to accomplish the same thing, so using both really isn't a great idea. You can still use Shoryuken to create the SQS message, but no matter what you do after that Shoryuken isn't natively going to be designed to handle messages coming off the queue through EB.

How you set things up will depend on what kind of app you have, but assuming it's a Rails app on the web environment, a good way to approach it would be to just add a controller that is specifically used in the worker environment, and deploy the same app to both. Have the daemon hit that endpoint with the webhook, and then have the controller action do the little bit of work necessary to construct and perform a job from the message, and respond appropriately. There will be some overlap between that code and the code already in Shoryuken that parses SQS messages, so you could try to reuse that if you want, but there's a lot going on in there with Celluloid and timers and managers that won't be that beneficial since every message is already going to be a separate HTTP request. A lot of that can be handled by the web or app server you will need in the container anyway (assuming you're using containers).

The way EB is set up currently, it's really just not a good place to deploy a Shoryuken server. If you want to deploy your web app on EB, it may make more sense to use ECS to handle the Shoryuken servers. Those containers end up being fairly simple (they don't really have any other container dependencies), so it's easy to configure their task definitions, and also handle the load balancing and auto-scaling if that's something you need.

[dyatlov]

I'll answer how we do this.
We have elasticbeanstslk and have shoryuken running in a separate worker env.
@ruuts you're right about preinstalled daemon which poll for messages. But.. it has it's disadvantage, you can poll just one queue; whereas with shoryuken you can poll from different queues with defined priority. Also shoryuken can't and won't work with that daemon.
How we use it? We left default worker queue as is and created another named SQS queue and have set shoryuken to listen from that named queue.
So shoryuken reads from needed queue and worker daemon does nothing because default queue is always empty.
It's also possible to use web tier env instead of worker env for shoryuken. This way you don't have to worry about preinstalled daemon (but we don't worry about it in worker env too) but should properly configure server(s) to not be accessible from outside.
Deployment is simple..

eb deploy - deploy to default web tier env
eb deploy worker - deploy to worker env

In order to know when code is inside worker/web env we have an environment variable defined in each configuration like so: WORKER_MODE=0

[ruuts]

Thanks all for the answers. Sorry for my late reply. [1]

@Senjai I meant how could shoryuken transform the SQS message, received on my worker http endpoint, into a job that could be performed?

@farski to me indeed shoryuken sounds like to much code for little benefit. I guess I should my make own ActiveJob adapter for just serializing the jobs as SQS messages and some middleware that catches the SQS message on the worker side, deserialize it and process it.

@dyatlov if I understand you well this means I should spawn a shoryuken worker process? (bundle exec shoryuken -r worker.rb ....). I think EB has no documented features for starting worker processes after deploy. So then I would still end up with a hacky solution ?

[1] I had a filter in Gmail configured that skips my inbox when i'm not @mentioned

[dyatlov]

@ruuts right, it means that you have to run the process manually. but you would anyway need to do it, that's the way how worker env is designed.

This is my config:

https://gist.github.com/dyatlov/4ca6ab8eb291d520036e

It's copied from current project so it has some unneeded info, im lazy :)

site.config should be in .ebextensions directory.. that's official way for working with instances.

[ruuts]

@dyatlov Thanks! I see what you mean. But it uses the /opt/elasticbeanstalk/hooks/appdeploy/post/ directory. As far as I know it's undocumented and thus has an unreliable future. Therefore I would not use this in production. Correct me if i'm wrong.

[dyatlov]

Correcting:

https://forums.aws.amazon.com/thread.jspa?messageID=493887

Directories and all the things are totally the same for all containers if you use the same platform version for staging and production. My scripts are for platform 2015.03 but I don't think anything changed with those directories since then.

@ruuts ^^

[ruuts]

@dyatlov thanks.

The post by Kenta@AWS is correct by David@AWS saying:

Dropping files directly into the hooks directories is risky as this is not the documented method, is different in some containers and may change in the future.

Therefore I think it's not wise to use the hooks directory before Amazon documented it themselves.

Anyways I think I will make a custom ActiveJob adapter and middleware that responds to the endpoint.
This way it's compliant with the documented Amazon features.

[ruuts]

https://github.com/tawan/active-elastic-job seems to be the correct solution