Hi, we're experiencing very slow cold starts even when activating flash boot, and this didn't happen before, with the same model architecture (Llama-3.1-8B) (different custom versions though). In fact, when it's about 1' since the last request, the worker appears initializing again, downloading weights etc. We've tried attaching a data storage to it, thinking that would lower cold start times (our hypothesis was that weights were downloaded there once and then just loaded from that disk at subsequent requests). However, that made things even worse, the delay times going up to 2' for a request.
We're about to launch an AI-based app, and have been using Runpod for development for some months now, but these delay times are not acceptable for the app to run properly. We need to scale down to 0 to keep costs variable at the beginning until we have more customers (then we will use active workers, for sure).
Can you please provide a solution for this? Is there some way to configure Runpod serverless endpoints so that delay times come back to where they were a week or two ago (1-2s)? @alpayariyak @Jorghi12 @pandyamarut @justinmerrell @carlson-svg @mikljohansson @casper-hansen @joennlae @willsamu @rachfop @vladmihaisima
Thanks in advance.
Hi, we're experiencing very slow cold starts even when activating flash boot, and this didn't happen before, with the same model architecture (Llama-3.1-8B) (different custom versions though). In fact, when it's about 1' since the last request, the worker appears initializing again, downloading weights etc. We've tried attaching a data storage to it, thinking that would lower cold start times (our hypothesis was that weights were downloaded there once and then just loaded from that disk at subsequent requests). However, that made things even worse, the delay times going up to 2' for a request.
We're about to launch an AI-based app, and have been using Runpod for development for some months now, but these delay times are not acceptable for the app to run properly. We need to scale down to 0 to keep costs variable at the beginning until we have more customers (then we will use active workers, for sure).
Can you please provide a solution for this? Is there some way to configure Runpod serverless endpoints so that delay times come back to where they were a week or two ago (1-2s)? @alpayariyak @Jorghi12 @pandyamarut @justinmerrell @carlson-svg @mikljohansson @casper-hansen @joennlae @willsamu @rachfop @vladmihaisima
Thanks in advance.