KFServing instructions fail on Docker build

[fbbradheintz]

Context

I'm attempting to follow the instructions at https://github.com/pytorch/serve/tree/master/kubernetes/kfserving but they fail on the second step:

DOCKER_BUILDKIT=1 docker build --no-cache --file Dockerfile_kf.dev -t <docker image name> .

(This is after substituting a suitable name for <docker image name>, of course.)

• torchserve version: Latest master, commit hash 3bc2556
• torch version: n/a (but the Docker build is attempting to install 1.7.1)
• torchvision version [if any]: n/a
• torchtext version [if any]: n/a
• torchaudio version [if any]: n/a
• java version:

openjdk 11.0.9.1 2020-11-04
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.9.1+1)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.9.1+1, mixed mode)

• Operating System and version: MacOS 10.15.7 building a Docker with ARG BASE_IMAGE=ubuntu:18.04

Your Environment

• Installed using source? [yes/no]: yes
• Are you planning to deploy it using docker container? [yes/no]: I'm trying to
• Is it a CPU or GPU environment?: CPU
• Using a default/custom handler? not relevant, but: serve/examples/image_classifier/mnist/mnist_handler.py
• What kind of model is it e.g. vision, text, audio?: not relevant, but mnist (image classifier)
• Are you planning to use local models from model-store or public url being used e.g. from S3 bucket etc.?
  [If public url then provide link.]: attempting local
• Provide config.properties, logs [ts.log] and parameters used for model registration/update APIs: n/a - haven't even gotten to that step
• Link to your project [if any]: n/a

Expected Behavior

Docker image created

Current Behavior

Docker image creation fails (see log below) after failing to install pygit2

Possible Solution

This mimicks a bug that used to exist with ts_scripts/install_dependencies.py. On Mac, running ts_scripts/install_dependencies.py --environment=dev installs pygit2 correctly; this command line is correctly updated in the Dockerfile, but still fails.

Attempting to change it to conda install pygit2 -c conda-forge or pip install pygit2 does not appear to help.

Steps to Reproduce

1. Follow the steps at https://github.com/pytorch/serve/tree/master/kubernetes/kfserving
2. The second step, DOCKER_BUILDKIT=1 docker build --no-cache --file Dockerfile_kf.dev -t <docker image name> ., fails, showing the log below.

Failure Logs [if any]

 => ERROR [dev-image 1/2] RUN if [ "cpu" = "gpu" ]; then export USE_CUDA=1; fi     && git clone -b master https://github.com/kubeflow/kfserving.git     && pip install -e .  136.8s
------
 > [dev-image 1/2] RUN if [ "cpu" = "gpu" ]; then export USE_CUDA=1; fi     && git clone -b master https://github.com/kubeflow/kfserving.git     && pip install -e ./kfserving/python/kfserving     && git clone https://github.com/pytorch/serve.git     && cd serve     && git checkout master     && python ts_scripts/install_dependencies.py --environment=dev --cuda latest     && python ts_scripts/install_from_src.py     && pip install captum     && pip install transformers    && pip install .     && useradd -m model-server     && mkdir -p /home/model-server/tmp     && cp kubernetes/kfserving/kf_predictor_docker/dockerd-entrypoint_kf.sh /usr/local/bin/dockerd-entrypoint_kf.sh     && chmod +x /usr/local/bin/dockerd-entrypoint_kf.sh     && chown -R model-server /home/model-server     && cp -R kubernetes/kfserving/kfserving_wrapper /home/model-server/kfserving_wrapper     && cp kubernetes/kfserving/kf_predictor_docker/config_kf.properties /home/model-server/config.properties     && mkdir /home/model-server/model-store && chown -R model-server /home/model-server/model-store:
#10 0.270 Cloning into 'kfserving'...
#10 9.051 Obtaining file:///kfserving/python/kfserving
#10 9.656 Requirement already satisfied: setuptools>=21.0.0 in /usr/local/lib/python3.6/dist-packages (from kfserving==0.5.0) (51.0.0)
#10 10.91 Collecting adal>=1.2.2
#10 11.00   Downloading adal-1.2.5-py2.py3-none-any.whl (55 kB)
#10 11.76 Collecting argparse>=1.4.0
#10 11.78   Downloading argparse-1.4.0-py2.py3-none-any.whl (23 kB)
#10 11.78 Collecting azure-storage-blob<=2.1.0,>=1.3.0
#10 11.80   Downloading azure_storage_blob-2.1.0-py2.py3-none-any.whl (88 kB)
#10 11.93 Collecting azure-common>=1.1.5
#10 11.96   Downloading azure_common-1.1.26-py2.py3-none-any.whl (12 kB)
#10 11.97 Collecting azure-storage-common~=2.1
#10 11.98   Downloading azure_storage_common-2.1.0-py2.py3-none-any.whl (47 kB)
#10 11.99 Collecting certifi>=14.05.14
#10 12.01   Downloading certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
#10 12.02 Collecting cryptography>=1.1.0
#10 12.04   Downloading cryptography-3.3.1-cp36-abi3-manylinux2010_x86_64.whl (2.6 MB)
#10 12.61 Collecting cffi>=1.12
#10 12.63   Downloading cffi-1.14.4-cp36-cp36m-manylinux1_x86_64.whl (401 kB)
#10 12.69 Collecting google-cloud-storage>=1.31.0
#10 12.84   Downloading google_cloud_storage-1.33.0-py2.py3-none-any.whl (92 kB)
#10 13.15 Collecting google-auth<2.0dev,>=1.11.0
#10 13.17   Downloading google_auth-1.23.0-py2.py3-none-any.whl (114 kB)
#10 13.40 Requirement already satisfied: setuptools>=21.0.0 in /usr/local/lib/python3.6/dist-packages (from kfserving==0.5.0) (51.0.0)
#10 13.41 Collecting cachetools<5.0,>=2.0.0
#10 13.43   Downloading cachetools-4.1.1-py3-none-any.whl (10 kB)
#10 13.44 Collecting google-cloud-core<2.0dev,>=1.4.1
#10 13.45   Downloading google_cloud_core-1.4.4-py2.py3-none-any.whl (27 kB)
#10 13.56 Collecting google-api-core<2.0.0dev,>=1.21.0
#10 13.57   Downloading google_api_core-1.23.0-py2.py3-none-any.whl (91 kB)
#10 13.59 Requirement already satisfied: setuptools>=21.0.0 in /usr/local/lib/python3.6/dist-packages (from kfserving==0.5.0) (51.0.0)
#10 14.15 Collecting google-resumable-media<2.0dev,>=1.0.0
#10 14.16   Downloading google_resumable_media-1.1.0-py2.py3-none-any.whl (75 kB)
#10 14.24 Collecting google-crc32c<2.0dev,>=1.0
#10 14.25   Downloading google_crc32c-1.0.0-cp36-cp36m-manylinux2010_x86_64.whl (39 kB)
#10 14.28 Collecting googleapis-common-protos<2.0dev,>=1.6.0
#10 14.30   Downloading googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
#10 14.33 Collecting kubernetes>=12.0.0
#10 14.35   Downloading kubernetes-12.0.1-py2.py3-none-any.whl (1.7 MB)
#10 14.47 Requirement already satisfied: setuptools>=21.0.0 in /usr/local/lib/python3.6/dist-packages (from kfserving==0.5.0) (51.0.0)
#10 14.77 Collecting minio>=4.0.9
#10 14.78   Downloading minio-7.0.0-py3-none-any.whl (72 kB)
#10 14.85 Collecting configparser
#10 14.87   Downloading configparser-5.0.1-py3-none-any.whl (22 kB)
#10 14.89 Collecting numpy>=1.17.3
#10 14.91   Downloading numpy-1.19.4-cp36-cp36m-manylinux2010_x86_64.whl (14.5 MB)
#10 15.53 Collecting protobuf>=3.12.0
#10 15.54   Downloading protobuf-3.14.0-cp36-cp36m-manylinux1_x86_64.whl (1.0 MB)
#10 15.58 Collecting pyasn1-modules>=0.2.1
#10 15.60   Downloading pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
#10 15.76 Collecting pyasn1<0.5.0,>=0.4.6
#10 15.78   Downloading pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
#10 15.79 Collecting pycparser
#10 15.80   Downloading pycparser-2.20-py2.py3-none-any.whl (112 kB)
#10 15.82 Collecting PyJWT>=1.0.0
#10 15.83   Downloading PyJWT-1.7.1-py2.py3-none-any.whl (18 kB)
#10 15.85 Collecting python_dateutil>=2.5.3
#10 15.87   Downloading python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
#10 15.90 Collecting pytz
#10 15.92   Downloading pytz-2020.4-py2.py3-none-any.whl (509 kB)
#10 15.95 Collecting pyyaml>=3.12
#10 15.96   Downloading PyYAML-5.3.1.tar.gz (269 kB)
#10 16.32 Collecting requests>=2.0.0
#10 16.34   Downloading requests-2.25.0-py2.py3-none-any.whl (61 kB)
#10 16.52 Collecting chardet<4,>=3.0.2
#10 16.59   Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)
#10 16.60 Collecting idna<3,>=2.5
#10 16.62   Downloading idna-2.10-py2.py3-none-any.whl (58 kB)
#10 16.63 Collecting requests-oauthlib
#10 16.65   Downloading requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
#10 16.72 Collecting oauthlib>=3.0.0
#10 16.74   Downloading oauthlib-3.1.0-py2.py3-none-any.whl (147 kB)
#10 16.76 Collecting rsa<5,>=3.1.4
#10 16.78   Downloading rsa-4.6-py3-none-any.whl (47 kB)
#10 16.83 Collecting six>=1.15
#10 16.85   Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
#10 16.86 Collecting table_logger>=0.3.5
#10 16.88   Downloading table_logger-0.3.6-py3-none-any.whl (14 kB)
#10 16.89 Collecting tornado>=6.0.0
#10 16.91   Downloading tornado-6.1-cp36-cp36m-manylinux2010_x86_64.whl (427 kB)
#10 16.94 Collecting urllib3>=1.15.1
#10 16.95   Downloading urllib3-1.26.2-py2.py3-none-any.whl (136 kB)
#10 17.00 Collecting websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0
#10 17.01   Downloading websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
#10 17.05 Building wheels for collected packages: pyyaml
#10 17.05   Building wheel for pyyaml (setup.py): started
#10 17.43   Building wheel for pyyaml (setup.py): finished with status 'done'
#10 17.43   Created wheel for pyyaml: filename=PyYAML-5.3.1-cp36-cp36m-linux_x86_64.whl size=44620 sha256=ea171036d0a403b963f1e9a352d2fd2a2893dbe5ae6e60ddec428c1aaf430db1
#10 17.43   Stored in directory: /root/.cache/pip/wheels/e5/9d/ad/2ee53cf262cba1ffd8afe1487eef788ea3f260b7e6232a80fc
#10 17.44 Successfully built pyyaml
#10 17.62 Installing collected packages: six, pyasn1, urllib3, rsa, pycparser, pyasn1-modules, protobuf, idna, chardet, certifi, cachetools, requests, pytz, googleapis-common-protos, google-auth, cffi, python-dateutil, oauthlib, google-crc32c, google-api-core, cryptography, azure-common, websocket-client, requests-oauthlib, pyyaml, PyJWT, numpy, google-resumable-media, google-cloud-core, configparser, azure-storage-common, tornado, table-logger, minio, kubernetes, google-cloud-storage, azure-storage-blob, argparse, adal, kfserving
#10 23.07   Running setup.py develop for kfserving
#10 23.50 Successfully installed PyJWT-1.7.1 adal-1.2.5 argparse-1.4.0 azure-common-1.1.26 azure-storage-blob-2.1.0 azure-storage-common-2.1.0 cachetools-4.1.1 certifi-2020.12.5 cffi-1.14.4 chardet-3.0.4 configparser-5.0.1 cryptography-3.3.1 google-api-core-1.23.0 google-auth-1.23.0 google-cloud-core-1.4.4 google-cloud-storage-1.33.0 google-crc32c-1.0.0 google-resumable-media-1.1.0 googleapis-common-protos-1.52.0 idna-2.10 kfserving kubernetes-12.0.1 minio-7.0.0 numpy-1.19.4 oauthlib-3.1.0 protobuf-3.14.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 python-dateutil-2.8.1 pytz-2020.4 pyyaml-5.3.1 requests-2.25.0 requests-oauthlib-1.3.0 rsa-4.6 six-1.15.0 table-logger-0.3.6 tornado-6.1 urllib3-1.26.2 websocket-client-0.57.0
#10 23.65 Cloning into 'serve'...
#10 31.52 Already on 'master'
#10 31.52 Your branch is up to date with 'origin/master'.
#10 31.58 sh: 1: nvidia-smi: not found
#10 31.58 sh: 1: sudo: not found
#10 31.58 sh: 1: sudo: not found
#10 31.58 sh: 1: conda: not found
#10 32.35 Collecting torch==1.6.0
#10 33.45   Downloading torch-1.6.0-cp36-cp36m-manylinux1_x86_64.whl (748.8 MB)
#10 64.85 Killed
#10 65.56 Requirement already satisfied: pip in /usr/local/lib/python3.6/dist-packages (20.3.1)
#10 67.61 Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (51.0.0)
#10 76.33 Requirement already satisfied: wheel in /usr/local/lib/python3.6/dist-packages (from -r requirements/common.txt (line 3)) (0.36.1)
#10 77.15 Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from -r requirements/common.txt (line 4)) (2.25.0)
#10 81.11 Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from -r requirements/developer.txt (line 8)) (1.19.4)
#10 83.68 Requirement already satisfied: protobuf in /usr/local/lib/python3.6/dist-packages (from -r requirements/developer.txt (line 10)) (3.14.0)
#10 85.86 Collecting captum
#10 85.97   Downloading captum-0.3.0-py3-none-any.whl (5.7 MB)
#10 86.70 Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from -r requirements/developer.txt (line 8)) (1.19.4)
#10 87.10 Collecting future
#10 87.12   Downloading future-0.18.2.tar.gz (829 kB)
#10 87.72 Collecting grpcio
#10 87.74   Downloading grpcio-1.34.0-cp36-cp36m-manylinux2014_x86_64.whl (4.0 MB)
#10 88.40 Requirement already satisfied: six>=1.5.2 in /usr/local/lib/python3.6/dist-packages (from grpcio->-r requirements/developer.txt (line 9)) (1.15.0)
#10 88.40 Collecting grpcio-tools
#10 88.42   Downloading grpcio_tools-1.34.0-cp36-cp36m-manylinux2014_x86_64.whl (2.5 MB)
#10 88.67 Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from grpcio-tools->-r requirements/developer.txt (line 11)) (51.0.0)
#10 88.68 Requirement already satisfied: protobuf in /usr/local/lib/python3.6/dist-packages (from -r requirements/developer.txt (line 10)) (3.14.0)
#10 88.74 Collecting mock
#10 88.76   Downloading mock-4.0.3-py3-none-any.whl (28 kB)
#10 88.78 Collecting nvgpu
#10 88.79   Downloading nvgpu-0.9.0-py2.py3-none-any.whl (9.4 kB)
#10 89.57 Requirement already satisfied: six>=1.5.2 in /usr/local/lib/python3.6/dist-packages (from grpcio->-r requirements/developer.txt (line 9)) (1.15.0)
#10 89.64 Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from -r requirements/common.txt (line 4)) (2.25.0)
#10 89.82 Collecting packaging
#10 89.84   Downloading packaging-20.7-py2.py3-none-any.whl (35 kB)
#10 90.03 Collecting pillow
#10 90.05   Downloading Pillow-8.0.1-cp36-cp36m-manylinux1_x86_64.whl (2.2 MB)
#10 90.24 Requirement already satisfied: six>=1.5.2 in /usr/local/lib/python3.6/dist-packages (from grpcio->-r requirements/developer.txt (line 9)) (1.15.0)
#10 90.24 Collecting psutil
#10 90.26   Downloading psutil-5.7.3.tar.gz (465 kB)
#10 90.93 Collecting pygit2
#10 90.96   Downloading pygit2-1.4.0-cp36-cp36m-manylinux2010_x86_64.whl (3.0 MB)
#10 91.29 Requirement already satisfied: cffi>=1.4.0 in /usr/local/lib/python3.6/dist-packages (from pygit2->-r requirements/developer.txt (line 13)) (1.14.4)
#10 91.30 Collecting pylint==2.6.0
#10 91.31   Downloading pylint-2.6.0-py3-none-any.whl (325 kB)
#10 91.85 Collecting pytest
#10 91.86   Downloading pytest-6.1.2-py3-none-any.whl (272 kB)
#10 92.36 Collecting pytest-cov
#10 92.38   Downloading pytest_cov-2.10.1-py2.py3-none-any.whl (19 kB)
#10 93.17 Collecting pytest-mock
#10 93.19   Downloading pytest_mock-3.3.1-py3-none-any.whl (11 kB)
#10 93.27 Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->-r requirements/common.txt (line 4)) (3.0.4)
#10 93.27 Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->-r requirements/common.txt (line 4)) (2.10)
#10 93.28 Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->-r requirements/common.txt (line 4)) (1.26.2)
#10 93.28 Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->-r requirements/common.txt (line 4)) (2020.12.5)
#10 93.28 Collecting sentencepiece
#10 93.32   Downloading sentencepiece-0.1.94-cp36-cp36m-manylinux2014_x86_64.whl (1.1 MB)
#10 93.44 Collecting transformers==2.10.0
#10 93.46   Downloading transformers-2.10.0-py3-none-any.whl (660 kB)
#10 95.05 Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from -r requirements/common.txt (line 4)) (2.25.0)
#10 95.33 Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from -r requirements/developer.txt (line 8)) (1.19.4)
#10 95.35 Collecting ansi2html
#10 95.36   Downloading ansi2html-1.6.0-py3-none-any.whl (14 kB)
#10 95.37 Collecting arrow
#10 95.39   Downloading arrow-0.17.0-py2.py3-none-any.whl (50 kB)
#10 95.40 Requirement already satisfied: python-dateutil>=2.7.0 in /usr/local/lib/python3.6/dist-packages (from arrow->nvgpu->-r requirements/developer.txt (line 7)) (2.8.1)
#10 95.41 Collecting astroid<=2.5,>=2.4.0
#10 95.42   Downloading astroid-2.4.2-py3-none-any.whl (213 kB)
#10 95.45 Requirement already satisfied: six>=1.5.2 in /usr/local/lib/python3.6/dist-packages (from grpcio->-r requirements/developer.txt (line 9)) (1.15.0)
#10 95.74 Collecting attrs>=17.4.0
#10 95.75   Downloading attrs-20.3.0-py2.py3-none-any.whl (49 kB)
#10 95.79 Collecting cached-property
#10 95.81   Downloading cached_property-1.5.2-py2.py3-none-any.whl (7.6 kB)
#10 95.82 Requirement already satisfied: pycparser in /usr/local/lib/python3.6/dist-packages (from cffi>=1.4.0->pygit2->-r requirements/developer.txt (line 13)) (2.20)
#10 95.83 Collecting coverage>=4.4
#10 95.85   Downloading coverage-5.3-cp36-cp36m-manylinux1_x86_64.whl (229 kB)
#10 95.88 Collecting dataclasses
#10 95.90   Downloading dataclasses-0.8-py3-none-any.whl (19 kB)
#10 95.91 Collecting filelock
#10 95.92   Downloading filelock-3.0.12-py3-none-any.whl (7.6 kB)
#10 95.93 Collecting flask
#10 95.95   Downloading Flask-1.1.2-py2.py3-none-any.whl (94 kB)
#10 96.27 Collecting click>=5.1
#10 96.28   Downloading click-7.1.2-py2.py3-none-any.whl (82 kB)
#10 96.30 Collecting flask-restful
#10 96.32   Downloading Flask_RESTful-0.3.8-py2.py3-none-any.whl (25 kB)
#10 96.43 Requirement already satisfied: six>=1.5.2 in /usr/local/lib/python3.6/dist-packages (from grpcio->-r requirements/developer.txt (line 9)) (1.15.0)
#10 96.43 Requirement already satisfied: pytz in /usr/local/lib/python3.6/dist-packages (from flask-restful->nvgpu->-r requirements/developer.txt (line 7)) (2020.4)
#10 96.43 Collecting aniso8601>=0.82
#10 96.45   Downloading aniso8601-8.1.0-py2.py3-none-any.whl (44 kB)
#10 96.46 Collecting importlib-metadata>=0.12
#10 96.48   Downloading importlib_metadata-3.1.1-py3-none-any.whl (9.6 kB)
#10 96.65 Collecting iniconfig
#10 96.66   Downloading iniconfig-1.1.1-py2.py3-none-any.whl (5.0 kB)
#10 96.68 Collecting isort<6,>=4.2.5
#10 96.69   Downloading isort-5.6.4-py3-none-any.whl (98 kB)
#10 96.72 Collecting itsdangerous>=0.24
#10 96.73   Downloading itsdangerous-1.1.0-py2.py3-none-any.whl (16 kB)
#10 96.75 Collecting Jinja2>=2.10.1
#10 96.76   Downloading Jinja2-2.11.2-py2.py3-none-any.whl (125 kB)
#10 96.85 Collecting lazy-object-proxy==1.4.*
#10 96.87   Downloading lazy_object_proxy-1.4.3-cp36-cp36m-manylinux1_x86_64.whl (55 kB)
#10 96.90 Collecting MarkupSafe>=0.23
#10 96.91   Downloading MarkupSafe-1.1.1-cp36-cp36m-manylinux1_x86_64.whl (27 kB)
#10 96.94 Collecting matplotlib
#10 96.95   Downloading matplotlib-3.3.3-cp36-cp36m-manylinux1_x86_64.whl (11.6 MB)
#10 98.11 Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from -r requirements/developer.txt (line 8)) (1.19.4)
#10 98.12 Requirement already satisfied: python-dateutil>=2.7.0 in /usr/local/lib/python3.6/dist-packages (from arrow->nvgpu->-r requirements/developer.txt (line 7)) (2.8.1)
#10 98.22 Collecting cycler>=0.10
#10 98.24   Downloading cycler-0.10.0-py2.py3-none-any.whl (6.5 kB)
#10 98.25 Requirement already satisfied: six>=1.5.2 in /usr/local/lib/python3.6/dist-packages (from grpcio->-r requirements/developer.txt (line 9)) (1.15.0)
#10 98.25 Collecting kiwisolver>=1.0.1
#10 98.27   Downloading kiwisolver-1.3.1-cp36-cp36m-manylinux1_x86_64.whl (1.1 MB)
#10 98.42 Collecting mccabe<0.7,>=0.6
#10 98.43   Downloading mccabe-0.6.1-py2.py3-none-any.whl (8.6 kB)
#10 98.44 Collecting pandas
#10 98.46   Downloading pandas-1.1.5-cp36-cp36m-manylinux1_x86_64.whl (9.5 MB)
#10 99.12 Requirement already satisfied: python-dateutil>=2.7.0 in /usr/local/lib/python3.6/dist-packages (from arrow->nvgpu->-r requirements/developer.txt (line 7)) (2.8.1)
#10 99.12 Requirement already satisfied: pytz in /usr/local/lib/python3.6/dist-packages (from flask-restful->nvgpu->-r requirements/developer.txt (line 7)) (2020.4)
#10 99.13 Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from -r requirements/developer.txt (line 8)) (1.19.4)
#10 99.16 Collecting pluggy<1.0,>=0.12
#10 99.18   Downloading pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
#10 99.20 Collecting py>=1.8.2
#10 99.22   Downloading py-1.9.0-py2.py3-none-any.whl (99 kB)
#10 99.26 Collecting pynvml
#10 99.28   Downloading pynvml-8.0.4-py3-none-any.whl (36 kB)
#10 99.30 Collecting pyparsing>=2.0.2
#10 99.32   Downloading pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
#10 99.35 Requirement already satisfied: six>=1.5.2 in /usr/local/lib/python3.6/dist-packages (from grpcio->-r requirements/developer.txt (line 9)) (1.15.0)
#10 99.37 Collecting regex!=2019.12.17
#10 99.39   Downloading regex-2020.11.13-cp36-cp36m-manylinux2014_x86_64.whl (723 kB)
#10 99.47 Collecting sacremoses
#10 99.49   Downloading sacremoses-0.0.43.tar.gz (883 kB)
#10 99.84 Requirement already satisfied: six>=1.5.2 in /usr/local/lib/python3.6/dist-packages (from grpcio->-r requirements/developer.txt (line 9)) (1.15.0)
#10 99.96 Collecting joblib
#10 99.98   Downloading joblib-0.17.0-py3-none-any.whl (301 kB)
#10 100.0 Collecting tabulate
#10 100.1   Downloading tabulate-0.8.7-py3-none-any.whl (24 kB)
#10 100.1 Collecting termcolor
#10 100.1   Downloading termcolor-1.1.0.tar.gz (3.9 kB)
#10 100.3 Collecting tokenizers==0.7.0
#10 100.4   Downloading tokenizers-0.7.0-cp36-cp36m-manylinux1_x86_64.whl (3.8 MB)
#10 100.6 Collecting toml>=0.7.1
#10 100.7   Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB)
#10 100.7 Collecting torch>=1.2
#10 100.7   Downloading torch-1.7.1-cp36-cp36m-manylinux1_x86_64.whl (776.8 MB)
#10 135.2 Killed
#10 135.3 sh: 1: sudo: not found
#10 135.3 sh: 1: sudo: not found
#10 135.3 sh: 1: sudo: not found
#10 135.3 sh: 1: sudo: not found
#10 135.3 sh: 1: sudo: not found
#10 135.3 sh: 1: sudo: not found
#10 135.6 Traceback (most recent call last):
#10 135.6   File "ts_scripts/install_from_src.py", line 62, in <module>
#10 135.6     from pygit2 import Repository
#10 135.6 ModuleNotFoundError: No module named 'pygit2'
------
executor failed running [/bin/sh -c if [ "$MACHINE_TYPE" = "gpu" ]; then export USE_CUDA=1; fi     && git clone -b ${BRANCH_NAME_KF} https://github.com/kubeflow/kfserving.git     && pip install -e ./kfserving/python/kfserving     && git clone https://github.com/pytorch/serve.git     && cd serve     && git checkout ${BRANCH_NAME}     && python ts_scripts/install_dependencies.py --environment=dev --cuda $CUDA_VERSION     && python ts_scripts/install_from_src.py     && pip install captum     && pip install transformers    && pip install .     && useradd -m model-server     && mkdir -p /home/model-server/tmp     && cp kubernetes/kfserving/kf_predictor_docker/dockerd-entrypoint_kf.sh /usr/local/bin/dockerd-entrypoint_kf.sh     && chmod +x /usr/local/bin/dockerd-entrypoint_kf.sh     && chown -R model-server /home/model-server     && cp -R kubernetes/kfserving/kfserving_wrapper /home/model-server/kfserving_wrapper     && cp kubernetes/kfserving/kf_predictor_docker/config_kf.properties /home/model-server/config.properties     && mkdir /home/model-server/model-store && chown -R model-server /home/model-server/model-store]: exit code: 1

[maaquib]

Was able to reproduce the issue on CPU (DLAMI 38, c5.4xl), on GPU (DLAMI 38, CU10.2, g4dn.xl).

[maaquib]

@fbbradheintz Adding sudo to the list of apt installs in the Dockerfile seems to resolve this. Will send out a PR

[fbbradheintz]

@maaquib Do you mean here?

RUN --mount=type=cache,id=apt-dev,target=/var/cache/apt \
    apt-get update && \
    DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y \
    fakeroot \
    ...

When I put a sudo in front of that apt-get install, I get a different error:

#7 5.370 Reading package lists...
#7 6.314 /bin/sh: 1: sudo: not found
------
executor failed running [/bin/sh -c apt-get update &&     DEBIAN_FRONTEND=noninteractive sudo apt-get install --no-install-recommends -y     fakeroot     ca-certificates     dpkg-dev     g++     git     python3-dev     build-essential     openjdk-11-jdk     curl     vim     && rm -rf /var/lib/apt/lists/*     && cd /tmp     && curl -O https://bootstrap.pypa.io/get-pip.py     && python3 get-pip.py]: exit code: 127

Any progress on that PR?

[maaquib]

@fbbradheintz I meant installing sudo here

Similar to this one

[chauhang]

@maaquib I was able to successfully build on Mac, Ubuntu 18.04 GPU and CPU machines just now and no errors during docker creation. Do we have some intermitted timeout problem somewhere which is causing different behaviour on different machines?

[fbbradheintz]

@maaquib: I added the line; it does not resolve the problem. What I'm seeing:

• Weirdly long times to clone repos from GitHub. I'm not sure whether this is my network env or something - but when I clone the same repos locally, they take seconds. I'll try building with --network=host, but this is slowing me down.
• I'm seeing "Killed" after the Dockerfile attempts to install PyTorch with pip. (See logs above.) I thought this was a network timeout, but I've since observed various timings for this, and I'm not sure what's going on.
• I'm getting that "Killed" message twice, because for some reason it's trying to install PyTorch twice - once 1.6.0, once 1.7.1. I'm guessing that whatever causes the first attempt (with 1.6.0 specified) to fail causes some later installation to attempt to install latest PyTorch as a dependency. No idea why they're failing consistently, but they are.
• Still getting the "pygit2" error, even though I see pygit being downloaded by pip. It must not be getting installed.

It's not some sort of bogus cache on Docker's part - I'm purging it before each build.

My next move is to try to run all these steps manually from inside a docker container.

[fbbradheintz]

It turned out not to be a timeout - that was just related to how long it took certain packages to download.

The actual error was an OOME that happened while pip was processing the >700MB PyTorch packages. I'm going to put up a PR with a note indicating that Docker instances should have at least 4GB of memory allocated.

I'm still getting sporadically long download times for the KFServing and TorchServe git repos, but that appears to be unrelated, and isn't keeping me from testing.

[chauhang]

@maaquib We still need to investigate why PT 1.6 and 1.7.1 installs are getting called twice in the docker setup

[maaquib]

@maaquib We still need to investigate why PT 1.6 and 1.7.1 installs are getting called twice in the docker setup

I have not been able to reproduce this.

[fbbradheintz]

@maaquib We still need to investigate why PT 1.6 and 1.7.1 installs are getting called twice in the docker setup

The 1.6.0 installation failed on an OOME; later, when installing Captum, there is another attempt (also failed due to OOME) to install PyTorch as a dependency, and it takes latest (1.7.1). This is not a separate bug, just a consequence of the out-of-memory failure.