SELinux prevents volume mount on Fedora

[mbacchi]

Description:

Rather than reopening #676, I'm opening a new issue for the most recent version of aws-sam-cli and docker. Issue #676 was prematurely closed in my opinion with the explanation that the user wasn't using the "current version of sam cli".

On Fedora 32, sam local invoke fails to mount a volume due to the fact that selinux prevents access to the host directory from within the docker container unless the :z or :Z mount option is added to add the selinux label to the directory.

If I run the following sam local invoke command using SAM to locally execute the Lambda function in my test repository here which uses a volume mount I get the error [ERROR] Runtime.ImportModuleError: Unable to import module 'app': No module named 'app':

sam local invoke LambdaEnvVarsFunction --event events/event.json
Invoking app.lambda_handler (python3.7)
Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-python3.7:rapid-1.7.0.

Mounting /home/user/lambda-env-vars/.aws-sam/build/LambdaEnvVarsFunction as /var/task:ro,delegated inside runtime container
START RequestId: afdda989-cb02-1c82-2916-f3e9927bce7d Version: $LATEST
[ERROR] Runtime.ImportModuleError: Unable to import module 'app': No module named 'app'
END RequestId: afdda989-cb02-1c82-2916-f3e9927bce7d
REPORT RequestId: afdda989-cb02-1c82-2916-f3e9927bce7d	Init Duration: 78.38 ms	Duration: 2.04 ms	Billed Duration: 100 ms	Memory Size: 128 MB	Max Memory Used: 23 MB	

{"errorType":"Runtime.ImportModuleError","errorMessage":"Unable to import module 'app': No module named 'app'"}

As discussed in #676, if I try to run Docker without the :z mount option I cannot read the contents of the directory:

docker run --rm -it -v /home/user/lambda-env-vars/.aws-sam/build/LambdaEnvVarsFunction:/var/task amazon/aws-sam-cli-emulation-image-python3.7 /bin/bash
bash-4.2# ls
ls: cannot open directory .: Permission denied
bash-4.2# exit

But it can be verified that the mount works by running Docker with the :z option. If I run the following command I am able to see the contents of the mounted volume within the docker container:

docker run --rm -it -v /home/user/lambda-env-vars/.aws-sam/build/LambdaEnvVarsFunction:/var/task:z amazon/aws-sam-cli-emulation-image-python3.7 /bin/bash
bash-4.2# ls
__init__.py  app.py  certifi  certifi-2020.6.20.dist-info  chardet  chardet-3.0.4.dist-info  idna  idna-2.10.dist-info	requests  requests-2.24.0.dist-info  requirements.txt  urllib3	urllib3-1.25.11.dist-info
bash-4.2# pwd 
/var/task
bash-4.2# exit

Finally, if I use the command also discussed in #676, to change the label on the directory, it allows me to run the sam local invoke command:

chcon -Rt svirt_sandbox_file_t .aws-sam/

sam local invoke LambdaEnvVarsFunction --event events/event.json
Invoking app.lambda_handler (python3.8)
Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-python3.8:rapid-1.7.0.

Mounting /home/user/lambda-env-vars/.aws-sam/build/LambdaEnvVarsFunction as /var/task:ro,delegated inside runtime container
START RequestId: a7f74c73-f625-1c23-9fd3-6ffa7ac22051 Version: $LATEST
END RequestId: a7f74c73-f625-1c23-9fd3-6ffa7ac22051
REPORT RequestId: a7f74c73-f625-1c23-9fd3-6ffa7ac22051	Init Duration: 82.44 ms	Duration: 1.98 ms	Billed Duration: 100 ms	Memory Size: 128 MB	Max Memory Used: 24 MB	

{"statusCode":200,"body":"{\"_HANDLER\": \"app.lambda_handler\", \"AWS_REGION\": \"us-east-1\", \"AWS_EXECUTION_ENV\": \"AWS_Lambda_python3.8\", \"AWS_LAMBDA_FUNCTION_NAME\": \"test\", \"AWS_LAMBDA_FUNCTION_MEMORY_SIZE\": \"128\", \"AWS_LAMBDA_FUNCTION_VERSION\": \"$LATEST\", \"AWS_LAMBDA_LOG_GROUP_NAME\": \"/aws/lambda/test\", \"AWS_LAMBDA_LOG_STREAM_NAME\": \"2020/11/07/[$LATEST]aa421238fa611d2b3e8d42595b803ea9\", \"LANG\": null, \"TZ\": null, \"LAMBDA_TASK_ROOT\": \"/var/task\", \"LAMBDA_RUNTIME_DIR\": \"/var/runtime\", \"PATH\": \"/var/lang/bin:/usr/local/bin:/usr/bin/:/bin:/opt/bin\", \"LD_LIBRARY_PATH\": \"/var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib:/opt/lib\", \"PYTHONPATH\": \"/var/runtime\", \"AWS_LAMBDA_RUNTIME_API\": null}"}

This appears to still be an open issue with aws-sam-cli related to the most recent version. It should be addressed by mounting with the :z or :Z mount option on Fedora or other Linuxes using SELinux.

Steps to reproduce:

1. Clone repo https://gitlab.com/mbacchi/lambda-env-vars ( or use any other SAM template with a volume mount) on Fedora 32.
2. run sam build
3. run sam local invoke LambdaEnvVarsFunction --event events/event.json

Observed result:

The volume mount is prevented by SELinux and the error [ERROR] Runtime.ImportModuleError: Unable to import module 'app': No module named 'app' is received.

Expected result:

The volume is mounted in the docker container and the sam local invoke command succeeds.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Fedora 32
2. sam --version: SAM CLI, version 1.7.0
3. aws cli version: aws-cli/1.18.118 Python/3.8.6 Linux/5.8.16-200.fc32.x86_64 botocore/1.17.63
4. docker version: Docker version 19.03.11, build 42e35e6

Add --debug flag to command you are running

sam --debug local invoke LambdaEnvVarsFunction --event events/event.json
Invoking app.lambda_handler (python3.8)
Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-python3.8:rapid-1.7.0.

Mounting /home/user/lambda-env-vars/.aws-sam/build/LambdaEnvVarsFunction as /var/task:ro,delegated inside runtime container
START RequestId: 00ba8c1a-ee02-1628-1902-e33d6f0e0736 Version: $LATEST
[ERROR] Runtime.ImportModuleError: Unable to import module 'app': No module named 'app'
END RequestId: 00ba8c1a-ee02-1628-1902-e33d6f0e0736
REPORT RequestId: 00ba8c1a-ee02-1628-1902-e33d6f0e0736	Init Duration: 82.01 ms	Duration: 2.25 ms	Billed Duration: 100 ms	Memory Size: 128 MB	Max Memory Used: 24 MB	

{"errorType":"Runtime.ImportModuleError","errorMessage":"Unable to import module 'app': No module named 'app'"}

[mbacchi]

I tried out the below change to container.py and this works by adding the :z SELinux volume mount to the docker command:

git diff
diff --git a/samcli/local/docker/container.py b/samcli/local/docker/container.py
index 676c8b2..9c2706b 100644
--- a/samcli/local/docker/container.py
+++ b/samcli/local/docker/container.py
@@ -7,6 +7,7 @@ import tarfile
 import tempfile
 import threading
 import socket
+import sys
 
 import docker
 import requests
@@ -114,6 +115,11 @@ class Container:
 
         LOG.info("Mounting %s as %s:ro,delegated inside runtime container", self._host_dir, self._working_dir)
 
+        # Define the volume mount mode here
+        mode = "ro,delegated"
+        if sys.platform == "linux":
+            mode = "z," + mode
+
         kwargs = {
             "command": self._cmd,
             "working_dir": self._working_dir,
@@ -123,7 +129,7 @@ class Container:
                     # https://docs.docker.com/storage/bind-mounts
                     # Mount the host directory as "read only" inside container
                     "bind": self._working_dir,
-                    "mode": "ro,delegated",
+                    "mode": mode,
                 }
             },
             # We are not running an interactive shell here.

I tested this out and it works to mount the volume when SELinux is enabled.

I realize this may not be the full and complete fix, and that it might require some verification to see if SELinux is indeed enabled on the system before the :z argument should be added to the command, but at least this is a proof of concept to show it can work.

If there are any suggestions from maintainers as to whether you'd prefer to test SELinux is enabled and whether this volume mount argument is in the right location, I can provide a PR.

[rmoskal]

I am dead in the water until this gets resolved.

[ebr]

Still broken on Fedora 33.

[meeuw]

I think it's best to use :Z instead of :z, according to the documentation:
https://docs.docker.com/storage/bind-mounts/#configure-the-selinux-label

:Z is more strict, not shared between containers.

There's more at play in Fedora because it upgraded to cgroups v2 which is incompatible with Docker/Moby.

To reproduce this issue:

• install Fedora (on EC2 or another VM, you'll need a kernel with SE Linux enabled which is the default for new installs)
• Install podman, dnf install -y podman (docker/moby is hard to install in Fedora)
• Follow the steps I've posted in issue Support local development on machines running Podman instead of Docker #1668
• Install aws-sam-cli
• Run sam init, use AWS Quick Start Templates, use python3.8 runtime
• Run sam build --use-container

[SingingBush]

This is still a problem with sam cli version 1.52.0

Running sam build -u --template-file template.yaml will fail due to SELinux on the host OS (Fedora 36 in my case) preventing the mounted volume from reading the files on the host. The solution has already been provided numerous times in pull requests and the comments above, please will someone from AWS do the right thing and put a Z into the relevant line in container.py:

aws-sam-cli/samcli/local/docker/container.py

Line 146 in 06d31fc

"mode": "ro,delegated",

[dmarinuswoodwing]

I gave up, installed moby-engine (instead of podman) and disabled selinux for docker in /etc/sysconfig/docker (remove --selinux-enabled).