From 5fa40f3442e98911249d0727d423557a602bae64 Mon Sep 17 00:00:00 2001 From: David Eliahu Date: Wed, 4 Mar 2020 13:51:09 -0800 Subject: [PATCH 1/2] Use tensorflow-cpu in images --- dev/versions.md | 26 +++++++++++-------- images/onnx-serve-gpu/Dockerfile | 3 ++- images/onnx-serve/Dockerfile | 3 ++- images/python-serve-gpu/Dockerfile | 3 ++- images/python-serve/Dockerfile | 3 ++- images/tf-api/Dockerfile | 14 ++++++++-- .../cortex/serve/tf.requirements.txt | 2 ++ 7 files changed, 37 insertions(+), 17 deletions(-) diff --git a/dev/versions.md b/dev/versions.md index 61ed6e6243..b69fcb0c45 100644 --- a/dev/versions.md +++ b/dev/versions.md @@ -86,23 +86,23 @@ Note: check their [install.md](https://github.com/kubernetes/client-go/blob/mast 1. `go mod tidy` 1. Check that the diff in `go.mod` is reasonable -## TensorFlow / TensorFlow Serving / Python / Python base operating system +## Python -The Python version in the base images for `tf-api` and `onnx-serve-gpu`/`python-serve-gpu` determines the Python version used throughout Cortex. +The same Python version should be used throughout Cortex (e.g. search for `3.6` and update all accordingly). -1. Update the `tensorflow/tensorflow` base image in `images/tf-api/Dockerfile` to the desired version ([Dockerhub](https://hub.docker.com/r/tensorflow/tensorflow)) -1. Update the `nvidia/cuda` base image in `images/python-serve-gpu/Dockerfile` and `images/onnx-serve-gpu/Dockerfile` (as well as `libnvinfer` in `images/python-serve-gpu/Dockerfile` and `images/tf-serve-gpu/Dockerfile`) to the desired version based on [TensorFlow's documentation](https://www.tensorflow.org/install/gpu#ubuntu_1804_cuda_101) ([Dockerhub](https://hub.docker.com/r/nvidia/cuda)) (it's possible these versions will diverge depending on ONNX runtime support) -1. Run `docker run --rm -it tensorflow/tensorflow:***`, and in the container run `python3 --version` and `cat /etc/lsb-release` -1. Run `docker run --rm -it nvidia/cuda:***`, and in the container run `cat /etc/lsb-release` -1. The Ubuntu versions should match; if they do not, downgrade whichever one is too advanced -1. The minor Python version in `tensorflow/tensorflow` must be used in all dockerfiles; search for e.g. `python3.6-dev` and update accordingly -1. Update TensorFlow version listed in `tensorflow.md` and `python.md` +It's probably safest to use the minor version of Python that you get when you run `apt-get install python3` ([currently that's what TensorFlow's Docker image does](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/dockerfiles/dockerfiles/cpu.Dockerfile)) + +## TensorFlow / TensorFlow Serving + +1. Find the latest release on [GitHub](https://github.com/tensorflow/tensorflow/releases) 1. Search the codebase for the current minor TensorFlow version (e.g. `2.1`) and update versions as appropriate -1. Search the codebase for the minor Python version (e.g. `3.6`) and update versions as appropriate -1. Search the codebase for `ubuntu` and update versions as appropriate Note: it's ok if example training notebooks aren't upgraded, as long as the exported model still works +## cuda + +1. Update the `nvidia/cuda` base image in `images/python-serve-gpu/Dockerfile` and `images/onnx-serve-gpu/Dockerfile` (as well as `libnvinfer` in `images/python-serve-gpu/Dockerfile` and `images/tf-serve-gpu/Dockerfile`) to the desired version based on [TensorFlow's documentation](https://www.tensorflow.org/install/gpu#ubuntu_1804_cuda_101) ([Dockerhub](https://hub.docker.com/r/nvidia/cuda)) (it's possible these versions will diverge depending on ONNX runtime support) + ## ONNX runtime 1. Update the version in `onnx-cpu.requirements.txt` and `onnx-gpu.requirements.txt` ([releases](https://github.com/microsoft/onnxruntime/releases)) @@ -204,6 +204,10 @@ Note: overriding horizontal-pod-autoscaler-sync-period on EKS is currently not s 1. Find the latest 2.X release on [GitHub](https://github.com/helm/helm/releases) (Istio does not work with helm 3) 1. Update the version in `images/manager/Dockerfile` +## Ubuntu base images + +1. Search the codebase for `ubuntu` and update versions as appropriate + ## Alpine base images 1. Find the latest release on [Dockerhub](https://hub.docker.com/_/alpine) diff --git a/images/onnx-serve-gpu/Dockerfile b/images/onnx-serve-gpu/Dockerfile index 01e87d56c8..207dbcc7e3 100644 --- a/images/onnx-serve-gpu/Dockerfile +++ b/images/onnx-serve-gpu/Dockerfile @@ -23,7 +23,8 @@ RUN apt-get update -qq && apt-get install -y -q \ curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \ python3.6 get-pip.py && \ pip install --upgrade pip && \ - rm -rf /root/.cache/pip* + rm -rf /root/.cache/pip* && \ + ln -s /usr/bin/python3.6 /usr/local/bin/python RUN locale-gen en_US.UTF-8 ENV LANG=en_US.UTF-8 LANGUAGE=en_US.en LC_ALL=en_US.UTF-8 diff --git a/images/onnx-serve/Dockerfile b/images/onnx-serve/Dockerfile index e2acc9af5c..61ba53eb84 100644 --- a/images/onnx-serve/Dockerfile +++ b/images/onnx-serve/Dockerfile @@ -23,7 +23,8 @@ RUN apt-get update -qq && apt-get install -y -q \ curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \ python3.6 get-pip.py && \ pip install --upgrade pip && \ - rm -rf /root/.cache/pip* + rm -rf /root/.cache/pip* && \ + ln -s /usr/bin/python3.6 /usr/local/bin/python RUN locale-gen en_US.UTF-8 ENV LANG=en_US.UTF-8 LANGUAGE=en_US.en LC_ALL=en_US.UTF-8 diff --git a/images/python-serve-gpu/Dockerfile b/images/python-serve-gpu/Dockerfile index 2e425272a9..44b587c467 100644 --- a/images/python-serve-gpu/Dockerfile +++ b/images/python-serve-gpu/Dockerfile @@ -25,7 +25,8 @@ RUN apt-get update -qq && apt-get install -y -q \ curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \ python3.6 get-pip.py && \ pip install --upgrade pip && \ - rm -rf /root/.cache/pip* + rm -rf /root/.cache/pip* && \ + ln -s /usr/bin/python3.6 /usr/local/bin/python RUN locale-gen en_US.UTF-8 ENV LANG=en_US.UTF-8 LANGUAGE=en_US.en LC_ALL=en_US.UTF-8 diff --git a/images/python-serve/Dockerfile b/images/python-serve/Dockerfile index 66361df370..4347e1f1e6 100644 --- a/images/python-serve/Dockerfile +++ b/images/python-serve/Dockerfile @@ -23,7 +23,8 @@ RUN apt-get update -qq && apt-get install -y -q \ curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \ python3.6 get-pip.py && \ pip install --upgrade pip && \ - rm -rf /root/.cache/pip* + rm -rf /root/.cache/pip* && \ + ln -s /usr/bin/python3.6 /usr/local/bin/python RUN locale-gen en_US.UTF-8 ENV LANG=en_US.UTF-8 LANGUAGE=en_US.en LC_ALL=en_US.UTF-8 diff --git a/images/tf-api/Dockerfile b/images/tf-api/Dockerfile index 5200826562..d54984f97c 100644 --- a/images/tf-api/Dockerfile +++ b/images/tf-api/Dockerfile @@ -1,20 +1,30 @@ -FROM tensorflow/tensorflow:2.1.0-py3 +FROM ubuntu:18.04 RUN apt-get update -qq && apt-get install -y -q \ build-essential \ + curl \ libfreetype6-dev \ libpng-dev \ libzmq3-dev \ pkg-config \ + rsync \ software-properties-common \ + unzip \ zlib1g-dev \ + python3.6-dev \ + python3.6-distutils \ git \ libsm6 \ libxext6 \ libxrender-dev \ libsndfile1 \ locales \ - && apt-get clean -qq && rm -rf /var/lib/apt/lists/* + && apt-get clean -qq && rm -rf /var/lib/apt/lists/* && \ + curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \ + python3.6 get-pip.py && \ + pip install --upgrade pip && \ + rm -rf /root/.cache/pip* && \ + ln -s /usr/bin/python3.6 /usr/local/bin/python RUN locale-gen en_US.UTF-8 ENV LANG=en_US.UTF-8 LANGUAGE=en_US.en LC_ALL=en_US.UTF-8 diff --git a/pkg/workloads/cortex/serve/tf.requirements.txt b/pkg/workloads/cortex/serve/tf.requirements.txt index 4f1a86305a..42db5119db 100644 --- a/pkg/workloads/cortex/serve/tf.requirements.txt +++ b/pkg/workloads/cortex/serve/tf.requirements.txt @@ -1,3 +1,5 @@ +tensorflow-cpu==2.1.0 + opencv-python==4.1.2.30 tensor2tensor==1.15.4 tensorflow-hub==0.7.0 From 7523d3865b99650b43728edc09cd8a8a9fe0ebd2 Mon Sep 17 00:00:00 2001 From: David Eliahu Date: Wed, 4 Mar 2020 14:08:25 -0800 Subject: [PATCH 2/2] Capitalize CUDA --- dev/versions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dev/versions.md b/dev/versions.md index b69fcb0c45..fac7b9f1c6 100644 --- a/dev/versions.md +++ b/dev/versions.md @@ -99,7 +99,7 @@ It's probably safest to use the minor version of Python that you get when you ru Note: it's ok if example training notebooks aren't upgraded, as long as the exported model still works -## cuda +## CUDA 1. Update the `nvidia/cuda` base image in `images/python-serve-gpu/Dockerfile` and `images/onnx-serve-gpu/Dockerfile` (as well as `libnvinfer` in `images/python-serve-gpu/Dockerfile` and `images/tf-serve-gpu/Dockerfile`) to the desired version based on [TensorFlow's documentation](https://www.tensorflow.org/install/gpu#ubuntu_1804_cuda_101) ([Dockerhub](https://hub.docker.com/r/nvidia/cuda)) (it's possible these versions will diverge depending on ONNX runtime support)