Sagemaker Local-Mode permission errors

[david-waterworth]

Describe the bug
I keep running into issues trying to run bring your own container (BYOC) images, they either work in local model, or they work in "cloud mode" but not both. The issue is almost certainly docker permission problems but I cannot see a way of resolving.

To reproduce
I have a docker image

FROM continuumio/miniconda3

WORKDIR /opt/ml/code/
COPY src/ /opt/ml/code/

ENTRYPOINT ["bash", "/opt/ml/code/processor.sh"]

The processor.sh creates a conda venv, installs packages into it then runs conda pack with the output set to /opt/ml/processing/output/pyenv.tar.gz

In local mode this step fails on this line

sagemaker-python-sdk/src/sagemaker/local/utils.py

Line 84 in 0e3769e

sagemaker_session.upload_data(source, bucket, path)

Because the container user (root) writes the file with permissions that are too restrictive. Note there's a "fix" on line 99 for a similar issue but it doesn't help.

I tried to work around this by adding a non-root (gid/pid 1000) user to the container. This "fixed" the local issue - but now the contain fails when I run it normally. It appears to be the opposite issue, the container user doesn't have access to the volume /opt/ml/processing/output/ so the script fails.

Expected behavior
Need to be able to run pipelines in local and sagemaker mode without permission errors. It seems that in local mode, scripts need to run with the same pid as the host user, but in sagemaker mode they need to run as root? Is there some way they can work in both?

[deto]

I think I just ran into a similar issue running a BYOC locally.

Error: [('/tmp/tmpqogmrvb3/model/checkpoint.ckpt', '/tmp/tmpqogmrvb3/artifacts/model/checkpoint.ckpt', "[Errno 13] 
Permission denied: '/tmp/tmpqogmrvb3/model/checkpoint.ckpt'")]

Traceback to line 113 in that same utils.py file

Inspecting that temporary directory, all the subfolders in /tmp/tmpqogmrvb3 are owned by the ubuntu user except the model/ subdirectory (owned by root).

The docker-compose.yaml specifies these mounts:

    volumes:
    - /tmp/tmpqogmrvb3/algo-1-7ykbr/output/data:/opt/ml/output/data
    - /tmp/tmpqogmrvb3/algo-1-7ykbr/input:/opt/ml/input
    - /tmp/tmpqogmrvb3/algo-1-7ykbr/output:/opt/ml/output
    - /tmp/tmpqogmrvb3/model:/opt/ml/model
    - /tmp/tmpjs1vhlf5:/opt/ml/input/data/training

However, there is also a model/ subdirectory inside the algo-1-7ykbr directory and that subdirectory is not owned by root.

What I suspect happened was that these directories get pre-created so that they aren't owned by root, but the docker-compose has an error where it incorrectly mounts <tmpdir>/model instead of <tmpdir>/algo-1-7ykbr/model causing the former to get created automatically by the docker daemon.