Conda pip install permission denied in OpenShift

[tsailiming]

Describe the bug

The deployment will fail with a permission problem when doing a conda pip install.

To reproduce

Using the following deployment file

apiVersion: machinelearning.seldon.io/v1alpha2
kind: SeldonDeployment
metadata:
  name: mlflow
  namespace: pipeline
spec:
  name: wines
  predictors:
    - graph:
        children: []
        implementation: MLFLOW_SERVER
        modelUri: s3://mlflow/0/4c8686e6026c4b51bb01b4654fe4cc7a/artifacts/torch-cnn-model #$MODEL_URI
        name: classifier
        envSecretRefName: seldon-init-container-secret
      name: default
      replicas: 1
      componentSpecs:
        - spec:
            containers:
              - name: classifier
                readinessProbe:
                  failureThreshold: 10
                  initialDelaySeconds: 120
                  periodSeconds: 30
                  successThreshold: 1
                  tcpSocket:
                    port: 9000
                  timeoutSeconds: 3
                livenessProbe:
                  failureThreshold: 10
                  initialDelaySeconds: 120
                  periodSeconds: 30
                  successThreshold: 1
                  tcpSocket:
                    port: 9000
                  timeoutSeconds: 3

Expected behaviour

Classifer should install correctly.

Environment

Seldon Operator 1.7.0 on OpenShift 4.8

• Kubernetes Cluster Version [Output of kubectl version]

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v0.21.0-beta.1", GitCommit:"0d10c3f72592addce965b9bb34992eb6fc283a3b", GitTreeState:"clean", BuildDate:"2021-08-31T22:38:26Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1+a620f50", GitCommit:"a620f506e95653a3ee8672445ed2e1133c6470db", GitTreeState:"clean", BuildDate:"2021-09-18T02:34:57Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (0.21) and server (1.21) exceeds the supported minor version skew of +/-1

• Deployed Seldon System Images: [Output of kubectl get --namespace seldon-system deploy seldon-controller-manager -o yaml | grep seldonio]

$ kubectl get --namespace openshift-operators deploy seldon-controller-manager -o yaml  | grep seldonio
                              "image": "seldonio/mock_classifier:1.6.0",
        containerImage: registry.connect.redhat.com/seldonio/seldon-core-operator@sha256:d50e4585f1f212b3fd5aa4200ccb2e2260b44cf17ed4d0fc89bfa61cf1969b9b
          value: registry.connect.redhat.com/seldonio/seldon-core-executor@sha256:2831ca7ae5a1451df2b8ebe26784a9b8e2ef80834350a76512b9f6790c6a0e55
          value: registry.connect.redhat.com/seldonio/seldon-engine@sha256:eac00afa854e5e03f367c5b052b631b20246f0b068f92669f04facc1d6d88ee0
          value: registry.connect.redhat.com/seldonio/storage-initializer@sha256:3bd2c461a522b8de3a350cac259b6188c092fd032f0b38ad446fc5be4a3d2d34
          value: registry.connect.redhat.com/seldonio/sklearnserver@sha256:88d126455b150291cbb3772f67b4f35a88bb54b15ff7c879022f77fb051615ad
          value: registry.connect.redhat.com/seldonio/xgboostserver@sha256:85a572a03e2c0ce1a7fb10c622f73c65d717a6e05c1ee00f5148b25774394f0b
          value: registry.connect.redhat.com/seldonio/mlflowserver@sha256:05cc0ecb43052aedb27954f478c059c08a8f133df71f96a39ab7eeabd0329dcd
          value: registry.connect.redhat.com/seldonio/tfproxy@sha256:e6ed51c45aaf1ca07ae4aaa7c2174795823dd92d7cc1073757ce671fb12abcbb
          value: registry.connect.redhat.com/seldonio/tensorflow-serving@sha256:1dc53edc5c8d1b151782f8386c0770d0358c6397eaf42ec55ebd18f331c5ef9e
          value: registry.connect.redhat.com/seldonio/alibiexplainer@sha256:a8f17861182f9611d50af0db2c93b9ddb6077b8b1919867ca92afe368886ffcd
          value: registry.connect.redhat.com/seldonio/mock-classifier@sha256:35952ee4778151cd5fc69929b94e82e2dcb0ad111ffb76f7b406a2e8cc461700
          value: docker.io/seldonio/engine:1.7.0
          value: seldonio/seldon-core-executor:1.7.0
        image: registry.connect.redhat.com/seldonio/seldon-core-operator@sha256:d50e4585f1f212b3fd5aa4200ccb2e2260b44cf17ed4d0fc89bfa61cf1969b9b

Model Details

• Logs of your model: [You can get the logs of your model by running kubectl logs -n <yourmodelnamespace> <seldonpodname> <container>]

$ oc logs -f mlflow-default-0-classifier-6f68f65787-tdgxn -c classifier
Executing before-run script
---> Creating environment with Conda...
INFO:root:Copying contents of /mnt/models to local
INFO:root:Reading MLmodel file
INFO:root:Creating Conda environment 'mlflow' from conda.yaml
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done

Downloading and Extracting Packages
_libgcc_mutex-0.1    | 3 KB      | ########## | 100% 
xz-5.2.5             | 343 KB    | ########## | 100% 
wheel-0.37.0         | 31 KB     | ########## | 100% 
libzlib-1.2.11       | 59 KB     | ########## | 100% 
python_abi-3.9       | 4 KB      | ########## | 100% 
tzdata-2021e         | 121 KB    | ########## | 100% 
pip-21.3.1           | 1.2 MB    | ########## | 100% 
zlib-1.2.11          | 86 KB     | ########## | 100% 
tk-8.6.11            | 3.3 MB    | ########## | 100% 
python-3.9.6         | 27.5 MB   | ########## | 100% 
ncurses-6.2          | 985 KB    | ########## | 100% 
readline-8.1         | 295 KB    | ########## | 100% 
libgomp-11.2.0       | 427 KB    | ########## | 100% 
openssl-1.1.1l       | 2.1 MB    | ########## | 100% 
libffi-3.3           | 51 KB     | ########## | 100% 
sqlite-3.36.0        | 1.4 MB    | ########## | 100% 
_openmp_mutex-4.5    | 22 KB     | ########## | 100% 
setuptools-58.3.0    | 1011 KB   | ########## | 100% 
libstdcxx-ng-11.2.0  | 4.2 MB    | ########## | 100% 
ld_impl_linux-64-2.3 | 667 KB    | ########## | 100% 
ca-certificates-2021 | 139 KB    | ########## | 100% 
libgcc-ng-11.2.0     | 887 KB    | ########## | 100% 
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Installing pip dependencies: ...working... Ran pip subprocess with arguments:
['/opt/conda/envs/mlflow/bin/python', '-m', 'pip', 'install', '-U', '-r', '/tmp/tmpozxwcy_v/condaenv.1j4lkq9j.requirements.txt']
Pip subprocess output:
Collecting mlflow
  Downloading mlflow-1.21.0-py3-none-any.whl (16.9 MB)
Collecting cloudpickle==2.0.0
  Downloading cloudpickle-2.0.0-py3-none-any.whl (25 kB)
Collecting ipython==7.25.0
  Downloading ipython-7.25.0-py3-none-any.whl (786 kB)
Collecting torch==1.10.0
  Downloading torch-1.10.0-cp39-cp39-manylinux1_x86_64.whl (881.9 MB)
Collecting torchvision==0.11.1
  Downloading torchvision-0.11.1-cp39-cp39-manylinux1_x86_64.whl (23.2 MB)
Collecting tqdm==4.61.2
  Downloading tqdm-4.61.2-py2.py3-none-any.whl (76 kB)
Requirement already satisfied: setuptools>=18.5 in /opt/conda/envs/mlflow/lib/python3.9/site-packages (from ipython==7.25.0->-r /tmp/tmpozxwcy_v/condaenv.1j4lkq9j.requirements.txt (line 3)) (58.3.0)
Collecting backcall
  Downloading backcall-0.2.0-py2.py3-none-any.whl (11 kB)
Collecting pexpect>4.3
  Downloading pexpect-4.8.0-py2.py3-none-any.whl (59 kB)
Collecting pickleshare
  Downloading pickleshare-0.7.5-py2.py3-none-any.whl (6.9 kB)
Collecting prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0
  Downloading prompt_toolkit-3.0.21-py3-none-any.whl (374 kB)
Collecting pygments
  Downloading Pygments-2.10.0-py3-none-any.whl (1.0 MB)
Collecting jedi>=0.16
  Downloading jedi-0.18.0-py2.py3-none-any.whl (1.4 MB)
Collecting traitlets>=4.2
  Downloading traitlets-5.1.1-py3-none-any.whl (102 kB)
Collecting matplotlib-inline
  Downloading matplotlib_inline-0.1.3-py3-none-any.whl (8.2 kB)
Collecting decorator
  Downloading decorator-5.1.0-py3-none-any.whl (9.1 kB)
Collecting typing-extensions
  Downloading typing_extensions-3.10.0.2-py3-none-any.whl (26 kB)
Collecting pillow!=8.3.0,>=5.3.0
  Downloading Pillow-8.4.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB)
Collecting numpy
  Downloading numpy-1.21.3-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB)
Collecting entrypoints
  Downloading entrypoints-0.3-py2.py3-none-any.whl (11 kB)
Collecting pytz
  Downloading pytz-2021.3-py2.py3-none-any.whl (503 kB)
Collecting querystring-parser
  Downloading querystring_parser-1.2.4-py2.py3-none-any.whl (7.9 kB)
Collecting gitpython>=2.1.0
  Downloading GitPython-3.1.24-py3-none-any.whl (180 kB)
Collecting click>=7.0
  Downloading click-8.0.3-py3-none-any.whl (97 kB)
Collecting protobuf>=3.7.0
  Downloading protobuf-3.19.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)
Collecting prometheus-flask-exporter
  Downloading prometheus_flask_exporter-0.18.4-py3-none-any.whl (17 kB)
Collecting importlib-metadata!=4.7.0,>=3.7.0
  Downloading importlib_metadata-4.8.1-py3-none-any.whl (17 kB)
Collecting alembic<=1.4.1
  Downloading alembic-1.4.1.tar.gz (1.1 MB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting pandas
  Downloading pandas-1.3.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.5 MB)
Collecting sqlalchemy
  Downloading SQLAlchemy-1.4.26-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.6 MB)
Collecting packaging
  Downloading packaging-21.2-py3-none-any.whl (40 kB)
Collecting gunicorn
  Downloading gunicorn-20.1.0-py3-none-any.whl (79 kB)
Collecting Flask
  Downloading Flask-2.0.2-py3-none-any.whl (95 kB)
Collecting pyyaml>=5.1
  Downloading PyYAML-6.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (661 kB)
Collecting docker>=4.0.0
  Downloading docker-5.0.3-py2.py3-none-any.whl (146 kB)
Collecting requests>=2.17.3
  Downloading requests-2.26.0-py2.py3-none-any.whl (62 kB)
Collecting databricks-cli>=0.8.7
  Downloading databricks-cli-0.16.2.tar.gz (58 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting sqlparse>=0.3.1
  Downloading sqlparse-0.4.2-py3-none-any.whl (42 kB)
Collecting Mako
  Downloading Mako-1.1.5-py2.py3-none-any.whl (75 kB)
Collecting python-editor>=0.3
  Downloading python_editor-1.0.4-py3-none-any.whl (4.9 kB)
Collecting python-dateutil
  Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting tabulate>=0.7.7
  Downloading tabulate-0.8.9-py3-none-any.whl (25 kB)
Collecting six>=1.10.0
  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting websocket-client>=0.32.0
  Downloading websocket_client-1.2.1-py2.py3-none-any.whl (52 kB)
Collecting gitdb<5,>=4.0.1
  Downloading gitdb-4.0.9-py3-none-any.whl (63 kB)
Collecting zipp>=0.5
  Downloading zipp-3.6.0-py3-none-any.whl (5.3 kB)
Collecting parso<0.9.0,>=0.8.0
  Downloading parso-0.8.2-py2.py3-none-any.whl (94 kB)
Collecting ptyprocess>=0.5
  Downloading ptyprocess-0.7.0-py2.py3-none-any.whl (13 kB)
Collecting wcwidth
  Downloading wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting idna<4,>=2.5
  Downloading idna-3.3-py3-none-any.whl (61 kB)
Collecting charset-normalizer~=2.0.0
  Downloading charset_normalizer-2.0.7-py3-none-any.whl (38 kB)
Collecting urllib3<1.27,>=1.21.1
  Downloading urllib3-1.26.7-py2.py3-none-any.whl (138 kB)
Collecting certifi>=2017.4.17
  Downloading certifi-2021.10.8-py2.py3-none-any.whl (149 kB)
Collecting greenlet!=0.4.17
  Downloading greenlet-1.1.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (153 kB)
Collecting Werkzeug>=2.0
  Downloading Werkzeug-2.0.2-py3-none-any.whl (288 kB)
Collecting Jinja2>=3.0
  Downloading Jinja2-3.0.2-py3-none-any.whl (133 kB)
Collecting itsdangerous>=2.0
  Downloading itsdangerous-2.0.1-py3-none-any.whl (18 kB)
Collecting pyparsing<3,>=2.0.2
  Downloading pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting prometheus-client
  Downloading prometheus_client-0.12.0-py2.py3-none-any.whl (57 kB)
Collecting smmap<6,>=3.0.1
  Downloading smmap-5.0.0-py3-none-any.whl (24 kB)
Collecting MarkupSafe>=2.0
  Downloading MarkupSafe-2.0.1-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (30 kB)
Building wheels for collected packages: alembic, databricks-cli
  Building wheel for alembic (setup.py): started
  Building wheel for alembic (setup.py): finished with status 'done'
  Created wheel for alembic: filename=alembic-1.4.1-py2.py3-none-any.whl size=158172 sha256=a9f354695406ccb6d4355df7ea78c3907b395689be11af5363c0ac6e02090fd0
  Stored in directory: /tmp/pip-ephem-wheel-cache-tgyikjat/wheels/05/12/70/e1473f1aab32af60e9b87f14bbbb745b7c2e86d7f3e5b5742b
  Building wheel for databricks-cli (setup.py): started
  Building wheel for databricks-cli (setup.py): finished with status 'done'
  Created wheel for databricks-cli: filename=databricks_cli-0.16.2-py3-none-any.whl size=106811 sha256=f1f4c47daa5945dcadc57a8c50293f7c91684229c67b9f443a24b46515ec8c9f
  Stored in directory: /tmp/pip-ephem-wheel-cache-tgyikjat/wheels/b1/e1/ad/c7f025c76afdaceefb7a32c8a380f80606ec557b152d7709ba
Successfully built alembic databricks-cli
Installing collected packages: MarkupSafe, Werkzeug, urllib3, smmap, six, Jinja2, itsdangerous, idna, greenlet, click, charset-normalizer, certifi, zipp, websocket-client, wcwidth, typing-extensions, traitlets, tabulate, sqlalchemy, requests, pytz, python-editor, python-dateutil, pyparsing, ptyprocess, prometheus-client, parso, numpy, Mako, gitdb, Flask, torch, sqlparse, querystring-parser, pyyaml, pygments, protobuf, prompt-toolkit, prometheus-flask-exporter, pillow, pickleshare, pexpect, pandas, packaging, matplotlib-inline, jedi, importlib-metadata, gunicorn, gitpython, entrypoints, docker, decorator, databricks-cli, cloudpickle, backcall, alembic, tqdm, torchvision, mlflow, ipython
Successfully installed Flask-2.0.2 Jinja2-3.0.2 Mako-1.1.5 MarkupSafe-2.0.1 Werkzeug-2.0.2 alembic-1.4.1 backcall-0.2.0 certifi-2021.10.8 charset-normalizer-2.0.7 click-8.0.3 cloudpickle-2.0.0 databricks-cli-0.16.2 decorator-5.1.0 docker-5.0.3 entrypoints-0.3 gitdb-4.0.9 gitpython-3.1.24 greenlet-1.1.2 gunicorn-20.1.0 idna-3.3 importlib-metadata-4.8.1 ipython-7.25.0 itsdangerous-2.0.1 jedi-0.18.0 matplotlib-inline-0.1.3 mlflow-1.21.0 numpy-1.21.3 packaging-21.2 pandas-1.3.4 parso-0.8.2 pexpect-4.8.0 pickleshare-0.7.5 pillow-8.4.0 prometheus-client-0.12.0 prometheus-flask-exporter-0.18.4 prompt-toolkit-3.0.21 protobuf-3.19.1 ptyprocess-0.7.0 pygments-2.10.0 pyparsing-2.4.7 python-dateutil-2.8.2 python-editor-1.0.4 pytz-2021.3 pyyaml-6.0 querystring-parser-1.2.4 requests-2.26.0 six-1.16.0 smmap-5.0.0 sqlalchemy-1.4.26 sqlparse-0.4.2 tabulate-0.8.9 torch-1.10.0 torchvision-0.11.1 tqdm-4.61.2 traitlets-5.1.1 typing-extensions-3.10.0.2 urllib3-1.26.7 wcwidth-0.2.5 websocket-client-1.2.1 zipp-3.6.0

done
#
# To activate this environment, use
#
#     $ conda activate mlflow
#
# To deactivate an active environment, use
#
#     $ conda deactivate



==> WARNING: A newer version of conda exists. <==
  current version: 4.9.2
  latest version: 4.10.3

Please update conda by running

    $ conda update -n base -c defaults conda


INFO:root:Install additional package from requirements.txt
ERROR conda.cli.main_run:execute(34): Subprocess for 'conda run ['pip', 'install', '-r', '/microservice/requirements.txt']' command failed.  (See above for error)
Processing ./python
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'error'

WARNING: The directory '/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
  ERROR: Command errored out with exit status 1:
   command: /opt/conda/envs/mlflow/bin/python3.9 /opt/conda/envs/mlflow/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpa5q4vpi7
       cwd: /microservice/python
  Complete output (2 lines):
  running egg_info
  error: [Errno 13] Permission denied
  ----------------------------------------
WARNING: Discarding file:///microservice/python. Command errored out with exit status 1: /opt/conda/envs/mlflow/bin/python3.9 /opt/conda/envs/mlflow/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpa5q4vpi7 Check the logs for full command output.
ERROR: Command errored out with exit status 1: /opt/conda/envs/mlflow/bin/python3.9 /opt/conda/envs/mlflow/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpa5q4vpi7 Check the logs for full command output.

Traceback (most recent call last):
  File "./conda_env_create.py", line 151, in <module>
    main(args)
  File "./conda_env_create.py", line 146, in main
    setup_env(model_folder)
  File "./conda_env_create.py", line 55, in setup_env
    install_base_reqs()
  File "./conda_env_create.py", line 136, in install_base_reqs
    run(cmd, shell=True, check=True)
  File "/opt/conda/lib/python3.7/subprocess.py", line 487, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'conda run -n mlflow pip install -r /microservice/requirements.txt' returned non-zero exit status 1.

[Aiden-Jeon]

I think this is the permission issue to pod.
Add securityContext to let pod has root permission.
Will you try below spec?

apiVersion: machinelearning.seldon.io/v1alpha2
kind: SeldonDeployment
metadata:
  name: mlflow
  namespace: pipeline
spec:
  name: wines
  predictors:
    - graph:
        children: []
        implementation: MLFLOW_SERVER
        modelUri: s3://mlflow/0/4c8686e6026c4b51bb01b4654fe4cc7a/artifacts/torch-cnn-model #$MODEL_URI
        name: classifier
        envSecretRefName: seldon-init-container-secret
      name: default
      replicas: 1
      componentSpecs:
        - spec:
            containers:
              - name: classifier
                readinessProbe:
                  failureThreshold: 10
                  initialDelaySeconds: 120
                  periodSeconds: 30
                  successThreshold: 1
                  tcpSocket:
                    port: 9000
                  timeoutSeconds: 3
                livenessProbe:
                  failureThreshold: 10
                  initialDelaySeconds: 120
                  periodSeconds: 30
                  successThreshold: 1
                  tcpSocket:
                    port: 9000
                  timeoutSeconds: 3
                securityContext:
                  privileged: true
                  runAsGroup: 0
                  runAsUser: 0

[agrski]

Requiring (and enabling) root is generally a poor practice from a security standpoints as presents a lot of risk.
The real problem is that conda is trying to install to a directory it doesn't have permissions for - persuading it to use a user-owned dir or getting the permissions fixed would be a better solution.

I think something about that came up recently, but would need to check.

Edit: this issue has exactly the same symptoms and error log.
Could you see if the steps mentioned in that resolve your issues?

[axsaucedo]

Closing - reopen if still an issue