New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiple concurrent requests causes java lambda containers to fail #154

Closed
Inkromind opened this Issue Oct 2, 2017 · 4 comments

Comments

Projects
None yet
4 participants
@Inkromind
Contributor

Inkromind commented Oct 2, 2017

We are using sam-local locally/in testing to run a java8 lambda with 192MB of memory and a rest endpoint that is called by our application to trigger the lambda.
If our application however sends out multiple requests concurrently and multiple java8 lambda containers are ran at the same time, one or more of the containers quits resulting in sam-local throwing the following error because it did not get a proper response:

Function returned an invalid response (must include one of: body, headers or statusCode in the response object): json parsing error syntax error unexpected end of input

The api endpoint then returns a 502.

This only happens when multiple java8 lambda containers are run at the same time. If they are triggered in sequence, we do not experience this problem.
The containers themself do not log any error. They simply quit. Because they also get removed afterwards, we have not succeeded in getting more info from them after they have quit to see why they might have been killed.

This error occurs both with docker for mac (with a memory limit for docker high enough to accomodate the multiple containers) and on linux.

This is a pretty critical problem for us as it means we cannot use the lambdas in local dev or testing.

@PaulMaddox

This comment has been minimized.

Show comment
Hide comment
@PaulMaddox

PaulMaddox Nov 25, 2017

Contributor

I'm afraid SAM Local is limited to a single invocation concurrency at the moment.

Contributor

PaulMaddox commented Nov 25, 2017

I'm afraid SAM Local is limited to a single invocation concurrency at the moment.

@PaulMaddox PaulMaddox closed this Nov 25, 2017

@Inkromind

This comment has been minimized.

Show comment
Hide comment
@Inkromind

Inkromind Nov 25, 2017

Contributor

Could this be enforced? e.g. by queueing requests. Now SAM Local executes all incoming requests concurrently. If it does not support that, I would expect it to not execute them concurrently, rather than doing it concurrently and knowing it can/will fail.

Contributor

Inkromind commented Nov 25, 2017

Could this be enforced? e.g. by queueing requests. Now SAM Local executes all incoming requests concurrently. If it does not support that, I would expect it to not execute them concurrently, rather than doing it concurrently and knowing it can/will fail.

@deckingravish

This comment has been minimized.

Show comment
Hide comment
@deckingravish

deckingravish Jan 19, 2018

I am also facing the same issue. If we make multiple concurrent calls, it spins all but one container for sure, which gives an even false impression of concurrency working fine to some extent. I had agree with @Inkromind that it must enforce single invocation if that's what it supports.

deckingravish commented Jan 19, 2018

I am also facing the same issue. If we make multiple concurrent calls, it spins all but one container for sure, which gives an even false impression of concurrency working fine to some extent. I had agree with @Inkromind that it must enforce single invocation if that's what it supports.

@emes001

This comment has been minimized.

Show comment
Hide comment
@emes001

emes001 Jan 31, 2018

I just ran into this same issue and it was driving me mad. Debating on if I have the bandwidth to fork and run multiple concurrent containers..

emes001 commented Jan 31, 2018

I just ran into this same issue and it was driving me mad. Debating on if I have the bandwidth to fork and run multiple concurrent containers..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment