Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ensure container/model is ready to be queried #447

Merged
merged 67 commits into from
Apr 8, 2018
Merged

Ensure container/model is ready to be queried #447

merged 67 commits into from
Apr 8, 2018

Conversation

chester-leung
Copy link
Member

Implemented Docker healtcheck and Kubernetes readiness probe to ensure that a container/model is ready to be queried. This change blocks deploy_model() until we get explicit notification that the container/model is running.

Also changes pull policy in query frontend and management frontend such that new Docker images are always pulled.

Fixes #436, Fixes #425, Fixes #152

chester-leung and others added 30 commits January 21, 2018 18:53
@chester-leung
created virtual env and added supporting files for mxnet
621216a
@chester-leung
added venv to .gitignore
da0346b
@chester-leung
Remove venv
86d1c44
@chester-leung
implemented serialization: save/load
b2695c4
@chester-leung
finished writing imxnet integration test
58a1963
@chester-leung
testing integration test
db9d8c3
@chester-leung
bind and fit model before saving
5c51a35
@chester-leung
Remove deps directory
7c93c59
load model changed
427526f
reformatted code
0fca8d5
removed duplicate code in clipper_admin/deployers/mxnet.py that was c…
2175c0e 
…ausing two models to register
fixed model loading
d9b4a29
reformatted code
09a507f
deleted commented out old code
ff91f8b
Merge branch 'develop' of https://github.com/chester-leung/clipper in…
9b55872 
…to develop

Merging reformatted code
deleted duplicate create_image line in build_docker_images.sh
03824f7
@chester-leung
synced fork
5aa718f
@chester-leung @dcrankshaw
new feature to wait for docker container to be running before registe…
cc059b4 
…ring model
@chester-leung @dcrankshaw
changed container starting wait time from 5 seconds to 3 seconds
63eccb7
@chester-leung @dcrankshaw
checking status of k8s deployment
ff44671
@chester-leung @dcrankshaw
k8s container manager sleeps when deployment is not ready
236d6e2
@chester-leung @dcrankshaw
remove k8s_start.py, test script for docker/k8s container manager edits
f716127
@chester-leung
changed to list append() function in docker container manager
5eb2ad5
@chester-leung
fixed docker_container_manager bug that caused set_num_replicas test …
ead3e8e 
…to fail
@chester-leung
reformatted code
e3ded43
@chester-leung
Merge branch 'container_check' into develop
f0e037b 
Added docker/k8s container checking
@chester-leung
fixed develop branch
8eb0e27
@chester-leung
Merge remote-tracking branch 'upstream/develop' into develop
9738772 
sync fork
@chester-leung
change data_shapes parameter in integration test to data_iter.provide…
58a2f90 
…_data, a DataDesc object
@chester-leung
updated documentation in deployers/mxnet.py to match changes
5a7b0ee
@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1190/
Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1192/
Test FAILed.

@dcrankshaw
Copy link
Contributor

Jenkins test this please

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1232/
Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1275/
Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1276/
Test FAILed.

@dcrankshaw
Copy link
Contributor

@chester-leung what's the status of this? Are you waiting on a review?

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1285/
Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1286/
Test FAILed.

@chester-leung
Copy link
Member Author

@dcrankshaw I'm trying to figure out why the build is failing - the tests are all passing when I run them locally. I encountered a problem locally similar to the current error when I hadn't rebuilt the py-rpc docker image to have the healthcheck. I think somehow the healthcheck isn't being set for the containers when Jenkins runs the tests.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1302/
Test FAILed.

@chester-leung
Copy link
Member Author

Jenkins test this please

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1305/
Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1311/
Test FAILed.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1316/
Test PASSed.

dcrankshaw
dcrankshaw approved these changes Apr 8, 2018
View reviewed changes
Copy link
Contributor

@dcrankshaw dcrankshaw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@dcrankshaw dcrankshaw merged commit 0d89cb2 into ucbrise:develop Apr 8, 2018
@chester-leung chester-leung deleted the docker_healthcheck branch April 8, 2018 20:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dcrankshaw dcrankshaw dcrankshaw approved these changes

Assignees
No one assigned
Labels
status: needs review
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@chester-leung @AmplabJenkins @dcrankshaw