-
Notifications
You must be signed in to change notification settings - Fork 280
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Clipper returns default predictions when a new version of a model is deployed but not connected #152
Comments
Maybe clipper_manager could have functions to Or Maybe a few new APIs
|
@dcrankshaw
I experienced that the time to initialize the new swarm service is very variable, so I think that Clipper might need some routines to check a new model's status before registering it to Clipper. |
Yeah agreed. Have you found that Swarm's container status is sufficient to indicate whether a container is running yet? We've found that for several deep learning models, especially when running on a GPU, there is a non-trivial amount of time after a container has started to initialize the model and connect to Clipper. Using the underlying container manager to detect when the containers ready as you do would be relatively simple to implement, but I was worried that that was not sufficient. |
@chester-leung For a first version of a fix, you should modify the |
@dcrankshaw According to the reference,
|
That's a good idea. @chester-leung for a first step, let's get a version working that just looks at the container state. As a second step, we can modify the RPC implementation to write a file somewhere once the container has connected to Clipper as the healthcheck. |
So far, I've implemented a fix to force the container manager to sleep until all added containers are deemed ready. This should fix the problem of querying a model in a new container before the new container is fully functional. |
When a user updates a model version, Clipper is informed that the model version has changed and it immediately starts to query the new version. However, it can take several minutes for the container for that new version to initialize and connect to Clipper. In that intervening period, Clipper attempts to query the latest version, cannot find any containers for that model version, and instead returns the default prediction.
Instead, it may be desirable for Clipper to wait until the new container has finished initializing and connects to Clipper before switching to the new version.
The text was updated successfully, but these errors were encountered: