-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Why are requests so slow? #134
Comments
Are you using a Java Jar for the code? It unzips the jar on every invoke which might be the problem. We need to implement caching of the unzipped folder to make it faster. |
No, js a hello world js
|
What OS are you on? And what version of Docker? From what I observed most of the startup latency is in spinning up Docker itself. We need a better mechanism to cache and reuse containers so you don't take this penalty. |
Ubuntu 17.04 and current release version of docker. Does it start a new container for each request? |
Hi @ztolley, We do create a new container for each invocation - we deliberately chose to not re-use containers for simplicity in the implementation. An overhead of 10 seconds is not normal though.
If you change the |
@PaulMaddox I have the same issue as @ztolley. If anything mine is worse...
Similar delay on each invocation. I notice that during most of this time I'm also using a simple node lambda function. It runs fine after the delay. I'm running Ubuntu 16.04.3 LTS in a VirtualBox VM on Windows.
I'm building aws-sam-local from source:
I'll try with the release version now... EDIT: same results with v0.2.2 EDIT: |
I increased the docker daemon debug level and can see the first pull of the image. This only happens on the first invocation.
On each invocation there is nothing in the docker log until 20s later when I see:
You can see that this whole sequence takes <1s so it seems docker is not the culprit. Not sure where to look next? |
I also tried running the hello-world sample using invoke with similar results:
|
@ztolley, @jugglingcats, do you have AWS credentials configured? If not, see if adding them speeds up your execution time. Before each execution, sam attempts to gather credentials. If no local credentials are found, it falls back to gathering credentials from ec2 instance metadata, which can take a very long time to fail if you are not running on an ec2 instance. |
Thanks @m5 that fixed it for me! Worth adding to the docs. |
Looks like this issue is resolved by supplying credentials. I will close the ticket.. |
Thanks for this thread... I had 6-7s total times, and adding the AWS creds environment variables brought the total times down to about ~1.3s ... I would love to see the response times be even lower, but honestly, I'll take aws sam local in any form. Incredibly helpful. |
I am facing the same issue. |
@aranvinds03. The way I worked around this was to extract my jar to a temp dir, then point CodeUri at that temp dir. Decompressing each time I make an API call is also pretty slow for me since my jar file is pretty large. |
I tried that as well. I extracted jar and run with CodeUri pointing to folder. Still, the requests are too slow. If i upload the jar to lambda, then requests hitting aws endpoint is much faster than local box. I realized that "decompressing jar" step is not time consuming. There is something else which takes more time in local box. |
@aravinds03 That's true. There is a lot more going on. Each request involves spinning up a new container (they are not reused), and then running a java process. For me, since my Lambda is Clojure, it also involves loading the Clojure runtime. This all adds up to my requests taking several seconds at a minimum. At this point, I am more or less just living with that limitation. |
Is it planned (or possible) to keep the containers warm?
… On Dec 4, 2017, at 3:43 PM, Jeb Beich ***@***.***> wrote:
@aravinds03 That's true. There is a lot more going on. Each request involves spinning up a new container (they are not reused), and then running a java process. For me, since my Lambda is Clojure, it also involves loading the Clojure runtime. This all adds up to my requests taking several seconds at a minimum. At this point, I am more or less just living with that limitation.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.
|
Just adding to this issue for future searchers: If you configured your credentials with a profile, as of sam v0.2.2, you need to pass This allowed me to get the speedups described in this thread. |
@jamescgibson , |
maybe it's obvious in hindsight, but sam couldn't find my credentials even using the
|
I find that the missing AWS credentials is not the issue I or seems are seeing which is causing the delay. Its clear that each invocation causes a decompressing of the JAR and then cleanup following the invocation. I saw a post where someone used a folder in the CodeUri property but could not get that working -- IS THERE a way to point to a folder which has already been decompressed / unzipped and keeps it "warm" essentially. Clearly, there needs to be a way to keep the lambda warm between invocations. I would be will to assist if given direction on this. |
AWS Credential solved for me, but I think this still would worth some investigation, why the SAM local looks for the credentials ? Some of the developers in larger organizations may not have access to the AWS environment. |
I test lambda by running SAM local in centos7 with vagrant, ~/.aws/config and ~/.aws/credentials were configured.But it always cost 30s. |
It's the same @zhawei described problems from a Aws-sam-local run by use normal user in the local environment. I change normal to the root user. Response message send from binary running with aws-sam-local fast more run than normal user. |
in my case sam local start-api --skip-pull-image |
I couldn't find it in the sam-local docs, but following the AWS SAM docs, you can increase the ram usage of your local service too. Took me a while to find this answer, so I'll leave this here. In your
You can view the actual usage of this too using |
Although I wasn't suffering delays as bad as other reports, each request was taking over 2s but after setting the |
I did an experiment with container reuse. This is just with a lambda in python, I'm developing on ubuntu 16.04. In summary, docker container spinning up only takes an extra second. So it is not worth making the feature for container reuse. Link to my code https://github.com/kevanpng/aws-sam-local . For a fixed query, both my colleague and I have 4s invocation time on sam local. His is a windows machine. With giving the My colleague is running on mac and when he tried the same query with lambda reuse and profile flag, he still had 11-14 seconds to run. Maybe it could be that docker is slow on mac? |
For those finding this issue first, also useful is the Ref #239 |
I had this exact issue today, but did not find any indication what caused is. It turns out the fix in #159 only applies to golang, so the python environment does not warn about it. Also, being able to run without credentials is important to me as I want to be certain nothing affects what is in the cloud. The problem is that the credentials cache is basically broken. It will try to get the credentials and then store them for use in later invocations. However, if no credentials are returned it keeps trying every time it is invoked because it does not know the difference between not initialized and not found. I fixed this in a fork, but don't know if the solution breaks something else so I didn't post it as a pull-request. In case someone else has the capacity to do so, here is my fix: bjorn-ove@b9e5afe |
@BearOve, has this solution been stable for you since the time of this post? |
Yes, no issues since related to this. |
i follow @skinofstars to use --warm-containers flag but the value is LAZY, and now the response time is decreased. |
I installed the package globally
Created a simple template and js file to say hello, copied from samples
Started with
sam local start-api
After the first start which downloads the docker images I can fire a request to the server from a browser or command line.
Immediately says invoking index.handler
waits 10 seconds then Then START, END and REPORT.
Whats the gap between Invoke and START, and what can you do to bring that down? Every request that follows the first has the same delay.
The text was updated successfully, but these errors were encountered: