WorkerServlet

[thomasturrell]

I don't understand how it would be possible to create workerservlets that could execute jobs that take longer that 60 seconds.

if the example at
https://github.com/aws/aws-toolkit-eclipse/blob/master/com.amazonaws.eclipse.elasticbeanstalk/templates/worker/src/WorkerServlet.java
is modified to contain Thread.sleep(120 * 1000); then the task gets submitted 10 times and then ends up in the dead letter queue.

Any help would be gratefully received.

[david-at-aws]

Hey Thomas, you should be able to configure this via the InactivityTimeout setting for your environment (search for "InactivityTimeout" on this page for more info).

[thomasturrell]

Hey David

Thank you for taking the time on this.

I think I have isolated a bug.

I changed the InactivityTimeout setting as you suggest however it doesn't behave as documented.

InactivityTimeout values up to 60 behave exactly as documented. Tasks that take a longer time than the InactivityTimeout value fail. Tasks that take less time than the InactivityTimeout succeed.

However (this is the bug), InactivityTimeout values of over 60 seconds do not behave as documented. Tasks that take a shorter time than the InactivityTimeout value but more than 60 seconds fail.

Basically there seems to be an undocumented upper value to the InactivityTimeout of 60 seconds.

Here are my logs when the task takes 120 seconds and the InactivityTimeout is set to 30:

2015-06-19T09:37:42Z init: initializing aws-sqsd 2.0 (2015-02-18)
2015-06-19T09:37:42Z start: polling https://sqs.eu-central-1.amazonaws.com/295923482971/awseb-e-zhmup3kvhm-stack-AWSEBWorkerQueue-1LUJ3UFSPYQGY
2015-06-19T09:38:26Z message: sent to %[http://localhost:80]
2015-06-19T09:38:56Z socket-err: 46502f2a-beda-4744-8fcf-f51ea6ad95a9 (1) Errno::ETIMEDOUT - 30.001
2015-06-19T09:38:58Z message: sent to %[http://localhost:80]
2015-06-19T09:39:29Z socket-err: 46502f2a-beda-4744-8fcf-f51ea6ad95a9 (2) Errno::ETIMEDOUT - 30.132

Repeated 8 more times (10 is the MaxRetries value)

This is the log when the when the task takes 120 seconds and the InactivityTimeout is set to 50:

2015-06-19T09:45:04Z init: initializing aws-sqsd 2.0 (2015-02-18)
2015-06-19T09:45:04Z start: polling https://sqs.eu-central-1.amazonaws.com/295923482971/awseb-e-zhmup3kvhm-stack-AWSEBWorkerQueue-1LUJ3UFSPYQGY
2015-06-19T09:46:41Z message: sent to %[http://localhost:80]
2015-06-19T09:47:31Z socket-err: 2f1cd567-5bee-4611-8d92-74cc0b9463dc (1) Errno::ETIMEDOUT - 50.002
2015-06-19T09:47:33Z message: sent to %[http://localhost:80]
2015-06-19T09:48:23Z socket-err: 2f1cd567-5bee-4611-8d92-74cc0b9463dc (2) Errno::ETIMEDOUT - 50.094

Repeated 8 more times (10 is the MaxRetries value)

This is the log when the task takes 120 seconds and the InactivityTimeout is set to 180:

2015-06-19T09:56:21Z init: initializing aws-sqsd 2.0 (2015-02-18)
2015-06-19T09:56:21Z start: polling https://sqs.eu-central-1.amazonaws.com/295923482971/awseb-e-zhmup3kvhm-stack-AWSEBWorkerQueue-1LUJ3UFSPYQGY
2015-06-19T09:58:24Z message: sent to %[http://localhost:80]
2015-06-19T09:59:24Z http-err: ecac994e-cee2-4665-b3df-b03585cf121d (1) 502 - 60.060
2015-06-19T09:59:26Z message: sent to %[http://localhost:80]
2015-06-19T10:00:26Z http-err: ecac994e-cee2-4665-b3df-b03585cf121d (2) 502 - 60.063

Repeated 8 more times (10 is the MaxRetries value)

Notice how the error has changed from a Errno::ETIMEDOUT to a 502. But also notice that the tasks fail at 60 seconds even though the InactivityTimeout is set to 180.

[thomasturrell]

I can prepose a workaround

the file /etc/httpd/conf.d/elasticbeanstalk.conf should set the ProxyTimeout to 1800.

1800 is the maximum value for InactivityTimeout as documented in http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/command-options.html.

The question in my mind is what has caused this bug. Perhaps something in the SQS stack (Apache?) has changed recently.

[david-at-aws]

Hey Thomas, thanks for the detailed report and the workaround! I've passed this on to the Elastic Beanstalk team to look into.

[zhangzhx]

Closing this issue due to no response for a long time. Feel free to reopen it if you still have more questions.