New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TfServing uber tracking bug #64

Closed
jlewi opened this Issue Dec 22, 2017 · 9 comments

Comments

Projects
None yet
5 participants
@jlewi
Copy link
Contributor

jlewi commented Dec 22, 2017

Opening an uber tracking bug to keep track of various features/improvements to ensure TfServing is well supported in Kubeflow

  • Publish Docker images #50
  • Health/liveness checks #368
  • Monitoring #369
  • GPU support

/cc @rhaertel80 @kow3ns

@nkashy1

This comment has been minimized.

Copy link
Contributor

nkashy1 commented Jan 6, 2018

TensorFlow Serving client library support - right now TensorFlow Serving only publishes a package to PyPI for Python 2 (link).

This means that users can't run this library from JupyterHub if they just follow our standard procedure (only a Python 3 backend is available there)

@jlewi jlewi added the area/inference label Jan 7, 2018

@jlewi jlewi added this to the Kubecon Europe milestone Jan 9, 2018

jlewi added a commit that referenced this issue Feb 5, 2018

add http proxy for tf-serving (#183)
Since REST Api is still widely used and tf-serving the c++ binary don't support http request.

This pr use python tornado with async supported web server to do the proxy

relationship between this pr and #65
differences:

#65 providing a big application supports tf-serving, xgboost and sklearn. while this pr is just a thin layer on top of tf-serving
#65 is using paste web server instead of tornado.
#65 is spawning a tf-serving process inside the docker, this pr use tf-serving as a grpc provider
similar:

both utilizing tensorflow/serving.
both provide CMLE like REST api
since #65 is too big for one pr and the author intend to split it, this pr can potentially help the tf-serving part. as the code is only one file.

Related to:

Define common prediction API(#102)
TfServing Uber issue (#64)
@yupbank

This comment has been minimized.

Copy link
Member

yupbank commented Feb 22, 2018

Health/liveness checks can be done from http-proxy

jlewi added a commit that referenced this issue Feb 24, 2018

Integrate tf serving image building with prow (#281)
pre/post-submit will also build / push / test TF serving image using the workflow in
components/k8s-model-server/images/releaser

The image name is gcr.io/mlkube-testing/model-server, and tag is the workflow name.

Related to #64  TFServing Uber Tracking Bug
Related to #50 Publish and maintain TFServing Docker images.
@jlewi

This comment has been minimized.

Copy link
Contributor

jlewi commented Feb 24, 2018

@yupbank Can you explain what you mean by doing health and liveness checks from the http proxy? Would that be testing the correct thing?

@yupbank

This comment has been minimized.

Copy link
Member

yupbank commented Feb 25, 2018

oh... for that i was referring to k8s liveness probe and readiness probe ...

@jlewi

This comment has been minimized.

Copy link
Contributor

jlewi commented Feb 25, 2018

@yupbank That part I understand but how does the http proxy help us determine the liveness/readiness of TFServing? The http proxy could be live/ready but the TFServing might not.

@yupbank

This comment has been minimized.

Copy link
Member

yupbank commented Feb 25, 2018

since now we have this https://github.com/tensorflow/serving/blob/master/tensorflow_serving/apis/model_service.proto#L16

http proxy can be forced to only be ready when TFServing is ready.

@jlewi

This comment has been minimized.

Copy link
Contributor

jlewi commented Mar 6, 2018

Closing this bug because we have more specific bugs for individual issues.

@jlewi jlewi closed this Mar 6, 2018

@lluunn

This comment has been minimized.

Copy link
Contributor

lluunn commented Mar 6, 2018

Should we have one for logging too?

@zmoon111

This comment has been minimized.

Copy link

zmoon111 commented Apr 19, 2018

i use util percent of cpu&gpu to check if it is in live status

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment