New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature Request: Allow warming up >1 function instances #24

Open
vladgolubev opened this Issue Jul 26, 2017 · 17 comments

Comments

Projects
None yet
7 participants
@vladgolubev

vladgolubev commented Jul 26, 2017

Thanks for the plugin! Currently the only limitation which stops me from using it instead of custom solution is that it warms up only 1 function, but I want to have ~ 10 warm functions w/ VPC connections, so it can handle a sudden burst.

What do you think? Is this a unique use case which doesn't belong to this plugin or you can consider this feature?

@goncaloneves

This comment has been minimized.

Show comment
Hide comment
@goncaloneves

goncaloneves Jul 26, 2017

Member

Hi @vladgolubev.

Yes it is something that I could work on and add to the plugin if there is actually a good way to do this.

Have you looked on how this would work?

Member

goncaloneves commented Jul 26, 2017

Hi @vladgolubev.

Yes it is something that I could work on and add to the plugin if there is actually a good way to do this.

Have you looked on how this would work?

@goncaloneves

This comment has been minimized.

Show comment
Hide comment
@goncaloneves

goncaloneves Jul 26, 2017

Member

What I have in my mind is blocking the lambda callback for let's say 5/10 seconds and invoking n times.

Member

goncaloneves commented Jul 26, 2017

What I have in my mind is blocking the lambda callback for let's say 5/10 seconds and invoking n times.

@vladgolubev

This comment has been minimized.

Show comment
Hide comment
@vladgolubev

vladgolubev Jul 26, 2017

@goncaloneves

If I understand correctly, I'd do smth like wrapping lines 236-L241 with

_.times(this.warmup.warmersCount, () => invokes.push(lambda.invoke(params)/* ... */)

vladgolubev commented Jul 26, 2017

@goncaloneves

If I understand correctly, I'd do smth like wrapping lines 236-L241 with

_.times(this.warmup.warmersCount, () => invokes.push(lambda.invoke(params)/* ... */)
@goncaloneves

This comment has been minimized.

Show comment
Hide comment
@goncaloneves

goncaloneves Jul 26, 2017

Member
Member

goncaloneves commented Jul 26, 2017

@der-flo

This comment has been minimized.

Show comment
Hide comment
@der-flo

der-flo Jul 27, 2017

I'm also interested in this topic. When Amazon scales up the lambda, it immediately routes requests to the new ones?

The coldstart problem could easily be fixed if we could signal the readyness for requests inside the booted lambda and Amazon waits with requests-routing until this signal. I did not read of such a feature in the AWS docs so I assume the route requests quite earlier to the new container, but not until the container (not the app / node) is up?

der-flo commented Jul 27, 2017

I'm also interested in this topic. When Amazon scales up the lambda, it immediately routes requests to the new ones?

The coldstart problem could easily be fixed if we could signal the readyness for requests inside the booted lambda and Amazon waits with requests-routing until this signal. I did not read of such a feature in the AWS docs so I assume the route requests quite earlier to the new container, but not until the container (not the app / node) is up?

@goncaloneves

This comment has been minimized.

Show comment
Hide comment
@goncaloneves

goncaloneves Jul 29, 2017

Member

@der-flo I don't think so.

Let's say you have no lambda container initialised, if you invoke 5 times concurrently, you will probably get 5 new containers initialised.
But after those have been initialised if you invoke again 10 minutes later, you will probably only hit one of the containers. And you will only hit a second one with high volume concurrency that has a higher price tag. All the others ones would be dropped unless you can keep that high volume coming in.

So this begs my question, what if we hold the container callback response to the warmup once it's initialised, could this make AWS spawn new containers? I have no idea and I have started writing some code to test this logic, but I still didn't have time to go through the testing.

If this is feasible then we could have a way of making this work, if not we need more ideas.

Member

goncaloneves commented Jul 29, 2017

@der-flo I don't think so.

Let's say you have no lambda container initialised, if you invoke 5 times concurrently, you will probably get 5 new containers initialised.
But after those have been initialised if you invoke again 10 minutes later, you will probably only hit one of the containers. And you will only hit a second one with high volume concurrency that has a higher price tag. All the others ones would be dropped unless you can keep that high volume coming in.

So this begs my question, what if we hold the container callback response to the warmup once it's initialised, could this make AWS spawn new containers? I have no idea and I have started writing some code to test this logic, but I still didn't have time to go through the testing.

If this is feasible then we could have a way of making this work, if not we need more ideas.

@juanjoDiaz

This comment has been minimized.

Show comment
Hide comment
@juanjoDiaz

juanjoDiaz Sep 15, 2017

Collaborator

A bit late to the party but I'm curious to know if got anywhere with this?

AFAIK AWS doesn't give any visibility on lambda concurrency and no guarantees of how they scale. So it seems almost impossible to me to add this functionality in a solid and predictable way.

I kind of refuse to believe that if you make 5 request to a non-instantiated lambda you'll get 5 lambdas. Amazon can definitely do better than that.

Regarding your proposal of holding the response in the event loop. I don't think that that will have any effect since, regardless on when the callback is executed, most probably the response is sent and the connection closed asap.

Collaborator

juanjoDiaz commented Sep 15, 2017

A bit late to the party but I'm curious to know if got anywhere with this?

AFAIK AWS doesn't give any visibility on lambda concurrency and no guarantees of how they scale. So it seems almost impossible to me to add this functionality in a solid and predictable way.

I kind of refuse to believe that if you make 5 request to a non-instantiated lambda you'll get 5 lambdas. Amazon can definitely do better than that.

Regarding your proposal of holding the response in the event loop. I don't think that that will have any effect since, regardless on when the callback is executed, most probably the response is sent and the connection closed asap.

@goncaloneves

This comment has been minimized.

Show comment
Hide comment
@goncaloneves

goncaloneves Sep 18, 2017

Member

I still didn't do this test yet.

I found these guys offering warmup to multiple lambda containers as a service. https://lambdacult.com/spark/faq

Fetching CW logs for tracking usage and using lambda versions with corresponding alias to switch during warmups.

Member

goncaloneves commented Sep 18, 2017

I still didn't do this test yet.

I found these guys offering warmup to multiple lambda containers as a service. https://lambdacult.com/spark/faq

Fetching CW logs for tracking usage and using lambda versions with corresponding alias to switch during warmups.

@ryan-roemer

This comment has been minimized.

Show comment
Hide comment
@ryan-roemer

ryan-roemer Aug 5, 2018

I have this exact same use case, and I was thinking that we could delay the completion of the lambda if the warmup task is detected. A somewhat reverse of the guidance in the README 😛

module.exports.lambdaToWarm = function(event, context, callback) {
  /** Delayed response for WarmUP plugin for concurrent warmups... */
  if (event.source === 'serverless-plugin-warmup') {
    setTimeout(function () {
      console.log('WarmUP - Lambda is warm!');
      return callback(null, 'Lambda is warm!');
    }, 1 * 1000); // wait 1 secs to force concurrent executions
  }

  // ... add lambda logic after
};

(I'm picking 1 secs, because I'm using VPC and we're seeing cold starts in the 10-14 seconds latency range, so that should definitely "hit it". Could probably pick more or less).

The problem with this hack, however, is that all the real traffic might spin up a lot more concurrent Lambdas during that explicit delay period. Then on the periodic refresh, we probably hit the same problem again...

But, I think it would potentially guarantee multiple concurrent executions (with some side effects / costs).

Thoughts? (Happy to PR this up if desired...)

ryan-roemer commented Aug 5, 2018

I have this exact same use case, and I was thinking that we could delay the completion of the lambda if the warmup task is detected. A somewhat reverse of the guidance in the README 😛

module.exports.lambdaToWarm = function(event, context, callback) {
  /** Delayed response for WarmUP plugin for concurrent warmups... */
  if (event.source === 'serverless-plugin-warmup') {
    setTimeout(function () {
      console.log('WarmUP - Lambda is warm!');
      return callback(null, 'Lambda is warm!');
    }, 1 * 1000); // wait 1 secs to force concurrent executions
  }

  // ... add lambda logic after
};

(I'm picking 1 secs, because I'm using VPC and we're seeing cold starts in the 10-14 seconds latency range, so that should definitely "hit it". Could probably pick more or less).

The problem with this hack, however, is that all the real traffic might spin up a lot more concurrent Lambdas during that explicit delay period. Then on the periodic refresh, we probably hit the same problem again...

But, I think it would potentially guarantee multiple concurrent executions (with some side effects / costs).

Thoughts? (Happy to PR this up if desired...)

@juanjoDiaz

This comment has been minimized.

Show comment
Hide comment
@juanjoDiaz

juanjoDiaz Aug 5, 2018

Collaborator

Hi @ryan-roemer,

That's a good theoretical idea. Except for the fact that you might be ~10 lambdas overprovisioned most of the time and the cost might increase a bit.

However, how do you know if AWS decides to spin up a new lambda or if it simply waits for your lambda to finish? As I mentioned above, Lambda concurrency is a mystery... Even, the tool that was posted above (https://lambdacult.com/spark/faq) to keep up multiple lambdas has shut down...

The only way of achieving something like this (iirc the way that spark used to do it) would be to hammer the API gateway and listen to the logs until you see the number different Lambda IDs that you want. Not a very nice solution, require the extra permission to read the logs, and looking at what happened to spark I assume that it doesn't quite work. 🙂

Collaborator

juanjoDiaz commented Aug 5, 2018

Hi @ryan-roemer,

That's a good theoretical idea. Except for the fact that you might be ~10 lambdas overprovisioned most of the time and the cost might increase a bit.

However, how do you know if AWS decides to spin up a new lambda or if it simply waits for your lambda to finish? As I mentioned above, Lambda concurrency is a mystery... Even, the tool that was posted above (https://lambdacult.com/spark/faq) to keep up multiple lambdas has shut down...

The only way of achieving something like this (iirc the way that spark used to do it) would be to hammer the API gateway and listen to the logs until you see the number different Lambda IDs that you want. Not a very nice solution, require the extra permission to read the logs, and looking at what happened to spark I assume that it doesn't quite work. 🙂

@ryan-roemer

This comment has been minimized.

Show comment
Hide comment
@ryan-roemer

ryan-roemer Aug 5, 2018

We can probably run some experiments in the CloudWatch dashboard / Lambda dashboards to just observe a constant concurrency-numbered warmer function and just see if it works out in practice...

I've got some decent introspection hooked up to some projects so could run some experiments if we wanted over like a day or so inspecting the logs. (Side note -- I'm weeks away from parental leave and will likely drop off the internet soon after -- so if I somehow don't pick up this thread until later, that's what happened)

ryan-roemer commented Aug 5, 2018

We can probably run some experiments in the CloudWatch dashboard / Lambda dashboards to just observe a constant concurrency-numbered warmer function and just see if it works out in practice...

I've got some decent introspection hooked up to some projects so could run some experiments if we wanted over like a day or so inspecting the logs. (Side note -- I'm weeks away from parental leave and will likely drop off the internet soon after -- so if I somehow don't pick up this thread until later, that's what happened)

@goncaloneves

This comment has been minimized.

Show comment
Hide comment
@goncaloneves

goncaloneves Aug 9, 2018

Member

@ryan-roemer any insight in your experiments of holding execution?

Member

goncaloneves commented Aug 9, 2018

@ryan-roemer any insight in your experiments of holding execution?

@ryan-roemer

This comment has been minimized.

Show comment
Hide comment
@ryan-roemer

ryan-roemer Aug 9, 2018

I haven't had time to fork this project to spin up multiple lambdas, but using a tweaked version of thsi plugin (had to manually copy / change the code script itself because we use package.artifact and a separate build script) I'm seeing lambdas stay warm on a 10 minute interval (a little more permissive than default 5).

I'm about to head out on baby leave, so I won't be able to PR a POC, but if someone else puts up a PR, I could pull it down and try it out in my current setup and report back!

ryan-roemer commented Aug 9, 2018

I haven't had time to fork this project to spin up multiple lambdas, but using a tweaked version of thsi plugin (had to manually copy / change the code script itself because we use package.artifact and a separate build script) I'm seeing lambdas stay warm on a 10 minute interval (a little more permissive than default 5).

I'm about to head out on baby leave, so I won't be able to PR a POC, but if someone else puts up a PR, I could pull it down and try it out in my current setup and report back!

@goncaloneves

This comment has been minimized.

Show comment
Hide comment
@goncaloneves

goncaloneves Aug 10, 2018

Member

Sure @ryan-roemer, I will revisit this soon I hope. Thanks for putting the time. Enjoy your leave.

Member

goncaloneves commented Aug 10, 2018

Sure @ryan-roemer, I will revisit this soon I hope. Thanks for putting the time. Enjoy your leave.

@aldarund

This comment has been minimized.

Show comment
Hide comment
@aldarund

aldarund Aug 21, 2018

there some solution for warming up multiple lambdas, but i didnt tested it yet but it might be good to see how they do it
https://medium.com/thundra/dealing-with-cold-starts-in-aws-lambda-a5e3aa8f532
https://github.com/thundra-io/thundra-lambda-nodejs-warmup

aldarund commented Aug 21, 2018

there some solution for warming up multiple lambdas, but i didnt tested it yet but it might be good to see how they do it
https://medium.com/thundra/dealing-with-cold-starts-in-aws-lambda-a5e3aa8f532
https://github.com/thundra-io/thundra-lambda-nodejs-warmup

@marcfielding1

This comment has been minimized.

Show comment
Hide comment
@marcfielding1

marcfielding1 Aug 24, 2018

I can work on this, I already have PoC code that runs of CloudWatch metrics although I also have an alternative idea which we can look at on PR. I'll have a look today tomorrow. :-) Could you assign this to me?

marcfielding1 commented Aug 24, 2018

I can work on this, I already have PoC code that runs of CloudWatch metrics although I also have an alternative idea which we can look at on PR. I'll have a look today tomorrow. :-) Could you assign this to me?

@marcfielding1

This comment has been minimized.

Show comment
Hide comment
@marcfielding1

marcfielding1 Aug 24, 2018

Container re-use isn't hard to monitor, for example in our code base I could do the following in my handler:

Excuse the pseudo code I'm on my mobile.

// inside my handler
  someLoggingFunction(isNewContainer)
  const isNewContainer = false

// loggingFunction

function someLogginFunction(isNewContainer) {
 // if isNewContainer is undefined then we log it as a new container with a timestamp
// else log it as a re-used container

}

Now the magic happens when your functions run, you check how many new containers your app used in the previous minute and multiply your requests by that.

This is a very simple version of what I have, you can get complicated by figuring out if it really was a new container by looking at the execution time per function and then figuring out if any overlapped.

Granted this does perhaps involve Dynamo or "another" method of storage, but with a simple switch you could have single warmpup or analytical warm up where it does this stuff, trust me when I say if you get this right it'll be hugely popular especially in financial services.

marcfielding1 commented Aug 24, 2018

Container re-use isn't hard to monitor, for example in our code base I could do the following in my handler:

Excuse the pseudo code I'm on my mobile.

// inside my handler
  someLoggingFunction(isNewContainer)
  const isNewContainer = false

// loggingFunction

function someLogginFunction(isNewContainer) {
 // if isNewContainer is undefined then we log it as a new container with a timestamp
// else log it as a re-used container

}

Now the magic happens when your functions run, you check how many new containers your app used in the previous minute and multiply your requests by that.

This is a very simple version of what I have, you can get complicated by figuring out if it really was a new container by looking at the execution time per function and then figuring out if any overlapped.

Granted this does perhaps involve Dynamo or "another" method of storage, but with a simple switch you could have single warmpup or analytical warm up where it does this stuff, trust me when I say if you get this right it'll be hugely popular especially in financial services.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment