Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SELinux prevents volume mount on Fedora #2360

Open
mbacchi opened this issue Nov 7, 2020 · 6 comments
Open

SELinux prevents volume mount on Fedora #2360

mbacchi opened this issue Nov 7, 2020 · 6 comments
Labels

Comments

@mbacchi
Copy link

mbacchi commented Nov 7, 2020

Description:

Rather than reopening #676, I'm opening a new issue for the most recent version of aws-sam-cli and docker. Issue #676 was prematurely closed in my opinion with the explanation that the user wasn't using the "current version of sam cli".

On Fedora 32, sam local invoke fails to mount a volume due to the fact that selinux prevents access to the host directory from within the docker container unless the :z or :Z mount option is added to add the selinux label to the directory.

If I run the following sam local invoke command using SAM to locally execute the Lambda function in my test repository here which uses a volume mount I get the error [ERROR] Runtime.ImportModuleError: Unable to import module 'app': No module named 'app':

sam local invoke LambdaEnvVarsFunction --event events/event.json
Invoking app.lambda_handler (python3.7)
Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-python3.7:rapid-1.7.0.

Mounting /home/user/lambda-env-vars/.aws-sam/build/LambdaEnvVarsFunction as /var/task:ro,delegated inside runtime container
START RequestId: afdda989-cb02-1c82-2916-f3e9927bce7d Version: $LATEST
[ERROR] Runtime.ImportModuleError: Unable to import module 'app': No module named 'app'
END RequestId: afdda989-cb02-1c82-2916-f3e9927bce7d
REPORT RequestId: afdda989-cb02-1c82-2916-f3e9927bce7d	Init Duration: 78.38 ms	Duration: 2.04 ms	Billed Duration: 100 ms	Memory Size: 128 MB	Max Memory Used: 23 MB	

{"errorType":"Runtime.ImportModuleError","errorMessage":"Unable to import module 'app': No module named 'app'"}

As discussed in #676, if I try to run Docker without the :z mount option I cannot read the contents of the directory:

docker run --rm -it -v /home/user/lambda-env-vars/.aws-sam/build/LambdaEnvVarsFunction:/var/task amazon/aws-sam-cli-emulation-image-python3.7 /bin/bash
bash-4.2# ls
ls: cannot open directory .: Permission denied
bash-4.2# exit

But it can be verified that the mount works by running Docker with the :z option. If I run the following command I am able to see the contents of the mounted volume within the docker container:

docker run --rm -it -v /home/user/lambda-env-vars/.aws-sam/build/LambdaEnvVarsFunction:/var/task:z amazon/aws-sam-cli-emulation-image-python3.7 /bin/bash
bash-4.2# ls
__init__.py  app.py  certifi  certifi-2020.6.20.dist-info  chardet  chardet-3.0.4.dist-info  idna  idna-2.10.dist-info	requests  requests-2.24.0.dist-info  requirements.txt  urllib3	urllib3-1.25.11.dist-info
bash-4.2# pwd 
/var/task
bash-4.2# exit

Finally, if I use the command also discussed in #676, to change the label on the directory, it allows me to run the sam local invoke command:

chcon -Rt svirt_sandbox_file_t .aws-sam/

sam local invoke LambdaEnvVarsFunction --event events/event.json
Invoking app.lambda_handler (python3.8)
Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-python3.8:rapid-1.7.0.

Mounting /home/user/lambda-env-vars/.aws-sam/build/LambdaEnvVarsFunction as /var/task:ro,delegated inside runtime container
START RequestId: a7f74c73-f625-1c23-9fd3-6ffa7ac22051 Version: $LATEST
END RequestId: a7f74c73-f625-1c23-9fd3-6ffa7ac22051
REPORT RequestId: a7f74c73-f625-1c23-9fd3-6ffa7ac22051	Init Duration: 82.44 ms	Duration: 1.98 ms	Billed Duration: 100 ms	Memory Size: 128 MB	Max Memory Used: 24 MB	

{"statusCode":200,"body":"{\"_HANDLER\": \"app.lambda_handler\", \"AWS_REGION\": \"us-east-1\", \"AWS_EXECUTION_ENV\": \"AWS_Lambda_python3.8\", \"AWS_LAMBDA_FUNCTION_NAME\": \"test\", \"AWS_LAMBDA_FUNCTION_MEMORY_SIZE\": \"128\", \"AWS_LAMBDA_FUNCTION_VERSION\": \"$LATEST\", \"AWS_LAMBDA_LOG_GROUP_NAME\": \"/aws/lambda/test\", \"AWS_LAMBDA_LOG_STREAM_NAME\": \"2020/11/07/[$LATEST]aa421238fa611d2b3e8d42595b803ea9\", \"LANG\": null, \"TZ\": null, \"LAMBDA_TASK_ROOT\": \"/var/task\", \"LAMBDA_RUNTIME_DIR\": \"/var/runtime\", \"PATH\": \"/var/lang/bin:/usr/local/bin:/usr/bin/:/bin:/opt/bin\", \"LD_LIBRARY_PATH\": \"/var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib:/opt/lib\", \"PYTHONPATH\": \"/var/runtime\", \"AWS_LAMBDA_RUNTIME_API\": null}"}

This appears to still be an open issue with aws-sam-cli related to the most recent version. It should be addressed by mounting with the :z or :Z mount option on Fedora or other Linuxes using SELinux.

Steps to reproduce:

  1. Clone repo https://gitlab.com/mbacchi/lambda-env-vars ( or use any other SAM template with a volume mount) on Fedora 32.
  2. run sam build
  3. run sam local invoke LambdaEnvVarsFunction --event events/event.json

Observed result:

The volume mount is prevented by SELinux and the error [ERROR] Runtime.ImportModuleError: Unable to import module 'app': No module named 'app' is received.

Expected result:

The volume is mounted in the docker container and the sam local invoke command succeeds.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

  1. OS: Fedora 32
  2. sam --version: SAM CLI, version 1.7.0
  3. aws cli version: aws-cli/1.18.118 Python/3.8.6 Linux/5.8.16-200.fc32.x86_64 botocore/1.17.63
  4. docker version: Docker version 19.03.11, build 42e35e6

Add --debug flag to command you are running

sam --debug local invoke LambdaEnvVarsFunction --event events/event.json
Invoking app.lambda_handler (python3.8)
Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-python3.8:rapid-1.7.0.

Mounting /home/user/lambda-env-vars/.aws-sam/build/LambdaEnvVarsFunction as /var/task:ro,delegated inside runtime container
START RequestId: 00ba8c1a-ee02-1628-1902-e33d6f0e0736 Version: $LATEST
[ERROR] Runtime.ImportModuleError: Unable to import module 'app': No module named 'app'
END RequestId: 00ba8c1a-ee02-1628-1902-e33d6f0e0736
REPORT RequestId: 00ba8c1a-ee02-1628-1902-e33d6f0e0736	Init Duration: 82.01 ms	Duration: 2.25 ms	Billed Duration: 100 ms	Memory Size: 128 MB	Max Memory Used: 24 MB	

{"errorType":"Runtime.ImportModuleError","errorMessage":"Unable to import module 'app': No module named 'app'"}
@mbacchi
Copy link
Author

mbacchi commented Nov 10, 2020

I tried out the below change to container.py and this works by adding the :z SELinux volume mount to the docker command:

git diff
diff --git a/samcli/local/docker/container.py b/samcli/local/docker/container.py
index 676c8b2..9c2706b 100644
--- a/samcli/local/docker/container.py
+++ b/samcli/local/docker/container.py
@@ -7,6 +7,7 @@ import tarfile
 import tempfile
 import threading
 import socket
+import sys
 
 import docker
 import requests
@@ -114,6 +115,11 @@ class Container:
 
         LOG.info("Mounting %s as %s:ro,delegated inside runtime container", self._host_dir, self._working_dir)
 
+        # Define the volume mount mode here
+        mode = "ro,delegated"
+        if sys.platform == "linux":
+            mode = "z," + mode
+
         kwargs = {
             "command": self._cmd,
             "working_dir": self._working_dir,
@@ -123,7 +129,7 @@ class Container:
                     # https://docs.docker.com/storage/bind-mounts
                     # Mount the host directory as "read only" inside container
                     "bind": self._working_dir,
-                    "mode": "ro,delegated",
+                    "mode": mode,
                 }
             },
             # We are not running an interactive shell here.

I tested this out and it works to mount the volume when SELinux is enabled.

I realize this may not be the full and complete fix, and that it might require some verification to see if SELinux is indeed enabled on the system before the :z argument should be added to the command, but at least this is a proof of concept to show it can work.

If there are any suggestions from maintainers as to whether you'd prefer to test SELinux is enabled and whether this volume mount argument is in the right location, I can provide a PR.

@qingchm qingchm added the stage/bug-repro The issue/bug needs to be reproduced label Nov 17, 2020
@rmoskal
Copy link

rmoskal commented Nov 18, 2020

I am dead in the water until this gets resolved.

@ebr
Copy link

ebr commented Apr 27, 2021

Still broken on Fedora 33.

@mbacchi mbacchi changed the title SELinux prevents volume mount on Fedora 32 SELinux prevents volume mount on Fedora Jun 18, 2021
@meeuw
Copy link

meeuw commented Jun 26, 2021

I think it's best to use :Z instead of :z, according to the documentation:
https://docs.docker.com/storage/bind-mounts/#configure-the-selinux-label

:Z is more strict, not shared between containers.

There's more at play in Fedora because it upgraded to cgroups v2 which is incompatible with Docker/Moby.

To reproduce this issue:

@SingingBush
Copy link

This is still a problem with sam cli version 1.52.0

Running sam build -u --template-file template.yaml will fail due to SELinux on the host OS (Fedora 36 in my case) preventing the mounted volume from reading the files on the host. The solution has already been provided numerous times in pull requests and the comments above, please will someone from AWS do the right thing and put a Z into the relevant line in container.py:

"mode": "ro,delegated",

@dmarinuswoodwing
Copy link
Contributor

dmarinuswoodwing commented Jun 16, 2022

I gave up, installed moby-engine (instead of podman) and disabled selinux for docker in /etc/sysconfig/docker (remove --selinux-enabled).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

9 participants