Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tf-serving support model on NFS #688

Merged
merged 6 commits into from
May 2, 2018
Merged

tf-serving support model on NFS #688

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
6f344d0
tf-serving support model on NFS
Apr 19, 2018
8f384cb
modelLocate -> modelStorageType
Apr 20, 2018
61ce6b4
make default modelStorageType 'cloud'
Apr 23, 2018
a4ecab0
mistake om -> in
Apr 24, 2018
8602b6b
remove redundant if clause
Apr 26, 2018
0778285
make 'cloud' as default value of modelStorageType
Apr 27, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
97 changes: 97 additions & 0 deletions components/k8s-model-server/Serve a local model using Tensorflow Serving.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
# Serve a local model using Tensorflow Serving

- Build NFS Server and allow your k8s machine to access. (more detail in [Setup an NFS Server](https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nfs-mount-on-ubuntu-16-04)). Assume `/var/nfs/general` folder has exported now.

- Put your model into NFS. In this tutorial, we also use inception model. So first download the whole model from [here](https://console.cloud.google.com/storage/browser/kubeflow-models/inception). Assume your model is located on `/var/nfs/general/inception`.

- Install NFS Client Components on your k8s machine

```
$ sudo apt-get update
$ sudo apt-get install nfs-common
```

- Check if model is available *(Optional)*

```
$ sudo mkdir -p /nfs/general
$ sudo mount NFS_SERVER_IP:/var/nfs/general /nfs/general
$ ls /nfs/general
inception
```

- Create Persistent Volume(PV) and Persistent Volume Claim(PVC) *[learn more](https://github.com/kubernetes/examples/tree/master/staging/volumes/nfs)*

- Create PV. You need to modify `NFS_SERVER_IP` to yours


```
$ cat nfs-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: nfs
spec:
capacity:
storage: 1Mi
accessModes:
- ReadWriteMany
nfs:
server: NFS_SERVER_IP
path: "/"

$ kubectl create -f nfs-pv.yaml
```

- Create PVC

```
$ cat nfs-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: nfs
spec:
accessModes:
- ReadWriteMany
storageClassName: ""
resources:
requests:
storage: 1Mi

$ kubectl create -f nfs-pvc.yaml
```

- Check PV and PVC *(Optional)*

```
$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
nfs 1Mi RWX Retain Bound default/nfs 20h

$kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
nfs Bound nfs 1Mi RWX 20h
```

- Create a Component for your model. You need to add `/mnt` before model path

```
$ MODEL_COMPONENT=serveInceptionNFS
$ MODEL_NAME=inception-nfs
$ MODEL_PATH=/mnt/var/nfs/general/inception
$ MODEL_STORAGE_TYPE=nfs
$ NFS_PVC_NAME=nfs
$ ks generate tf-serving ${MODEL_COMPONENT} --name=${MODEL_NAME}
$ ks param set ${MODEL_COMPONENT} modelPath ${MODEL_PATH}
$ ks param set ${MODEL_COMPONENT} modelStorageType ${MODEL_STORAGE_TYPE}
$ ks param set ${MODEL_COMPONENT} nfsPVC ${NFS_PVC_NAME}
```

- Deploy the model component. Ksonnet will pick up existing parameters for your environment (e.g. cloud, nocloud) and customize the resulting deployment appropriately

```
$ ks apply ${KF_ENV} -c ${MODEL_COMPONENT}
```

- Use the served model see [here](https://github.com/kubeflow/kubeflow/tree/master/components/k8s-model-server#use-the-served-model)
17 changes: 17 additions & 0 deletions kubeflow/tf-serving/tf-serving.libsonnet
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,13 @@
runAsUser: 1000,
fsGroup: 1000,
},
volumeMounts+: if 'modelStorageType' om $.params then
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's the om here

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes! thanks for telling!

if $.params.modelStorageType == "nfs" then [{
name: "nfs",
mountPath: "/mnt",
}]
else []
else [],
}, // tfServingContainer

tfServingContainer+: $.parts.tfServingContainerBase +
Expand Down Expand Up @@ -217,6 +224,16 @@
if $.util.toBool($.params.deployHttpProxy) then
$.parts.httpProxyContainer,
],
volumes+: if 'modelStorageType' in $.params then
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reasonable. Yesterday I tried to make modelStorageType as a param for ks generate tf-serving but failed (so far). So I add storageType into params. When I use modelStorageType as a param, it shows some error (maybe meets some conflicts).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking just use a param, modelStorageType = cloud by default.
What's the error?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If using modelStorageType like this:

  params:: {
  ...
  ...(other params)
  modelName: $.params.name,
  modelPath: null,
  modelStorageType: if "modelStorageType" in $.params then
      $.params.modelStorageType
    else
      "cloud",

It will shows an error:

ERROR generate objects for namespace : unable to read /home/ciscoai/my-kubeflow/environments/default/main.jsonnet: RUNTIME ERROR: Max stack frames exceeded.
-------------------------------------------------
	/home/ciscoai/my-kubeflow/components/serveInception.jsonnet:(15:12)-(17:4)	object <anonymous>

  params+: updatedParams {
    name: name,
  },

-------------------------------------------------
	<builtin>	builtin function <operator+>

-------------------------------------------------
	<std>:966:25-26	thunk from <function <anonymous>>

        std.objectHasEx(o, f, true),

-------------------------------------------------
	<builtin>	builtin function <objectHasEx>

-------------------------------------------------
		

-------------------------------------------------
	/home/ciscoai/my-kubeflow/vendor/kubeflow/tf-serving/tf-serving.libsonnet:14:7-32	object <anonymous>

      $.params.modelStorageType

-------------------------------------------------
	/home/ciscoai/my-kubeflow/vendor/kubeflow/tf-serving/tf-serving.libsonnet:14:7-32	object <anonymous>

      $.params.modelStorageType

-------------------------------------------------
	/home/ciscoai/my-kubeflow/vendor/kubeflow/tf-serving/tf-serving.libsonnet:14:7-32	object <anonymous>

      $.params.modelStorageType

-------------------------------------------------
	/home/ciscoai/my-kubeflow/vendor/kubeflow/tf-serving/tf-serving.libsonnet:14:7-32	object <anonymous>

      $.params.modelStorageType

-------------------------------------------------
	/home/ciscoai/my-kubeflow/vendor/kubeflow/tf-serving/tf-serving.libsonnet:14:7-32	object <anonymous>

      $.params.modelStorageType

-------------------------------------------------
	... (skipped 481 frames)
-------------------------------------------------
	<std>:971:33-35	thunk from <function <anonymous>>

        if !std.primitiveEquals(ta, tb) then

-------------------------------------------------
	<builtin>	builtin function <primitiveEquals>

-------------------------------------------------
	<std>:971:12-40	function <anonymous>

        if !std.primitiveEquals(ta, tb) then

-------------------------------------------------
		

-------------------------------------------------
	<std>:1025:45-71	function <anonymous>

            for x in std.objectFields(a) if isContent(std.prune(a[x]))

-------------------------------------------------
	<builtin>	builtin function <flatMap>

-------------------------------------------------
	<builtin>	builtin function <$objectFlatMerge>

-------------------------------------------------
	/home/ciscoai/my-kubeflow/components/serveInception.jsonnet:20:1-52	$

std.prune(k.core.v1.list.new(tfServing.components))

-------------------------------------------------
	<extvar:__ksonnet/components>:2:19-86	object <anonymous>

  serveInception: import "/home/ciscoai/my-kubeflow/components/serveInception.jsonnet",

-------------------------------------------------
	During manifestation

When I replace modelStorageType to storageType(or other identifier):

    storageType: if "modelStorageType" in $.params then
      $.params.modelStorageType
    else
      "cloud",

It runs well.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean, just doing

params:: {
modelStorageType: "cloud",
}

It will get overridden if you pass the param modelStorageType

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh ok! thanks for telling!

if $.params.modelStorageType == "nfs" then
[{
name: "nfs",
persistentVolumeClaim: {
claimName: $.params.nfsPVC,
}
}]
else []
else [],
},
},
},
Expand Down
16 changes: 15 additions & 1 deletion user_guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -296,7 +296,7 @@ unsecured endpoint by default. For a production deployment with SSL and authenti

We treat each deployed model as a [component](https://ksonnet.io/docs/tutorial#2-generate-and-deploy-an-app-component) in your APP.

Create a component for your model
Create a component for your model located on cloud

```
MODEL_COMPONENT=serveInception
Expand All @@ -306,6 +306,20 @@ ks generate tf-serving ${MODEL_COMPONENT} --name=${MODEL_NAME}
ks param set ${MODEL_COMPONENT} modelPath ${MODEL_PATH}
```

*(Or)* create a component for your model located on nfs, learn more from `components/k8s-model-server`

```
MODEL_COMPONENT=serveInceptionNFS
MODEL_NAME=inception-nfs
MODEL_PATH=/mnt/var/nfs/general/inception
MODEL_STORAGE_TYPE=nfs
NFS_PVC_NAME=nfs
ks generate tf-serving ${MODEL_COMPONENT} --name=${MODEL_NAME}
ks param set ${MODEL_COMPONENT} modelPath ${MODEL_PATH}
ks param set ${MODEL_COMPONENT} modelStorageType ${MODEL_STORAGE_TYPE}
ks param set ${MODEL_COMPONENT} nfsPVC ${NFS_PVC_NAME}
```

Deploy the model component. Ksonnet will pick up existing parameters for your environment (e.g. cloud, nocloud) and customize the resulting deployment appropriately

```
Expand Down