sam build with --use container doesn't work if sam is running inside a container

[atreyd]

I am trying to build my lambda package inside a container with latest aws-sam-cli installed. Consider a scenario with a Jenkins or any CI server running inside containers.

Here are the reproduction steps:

• Run a container in interactive mode with aws-sam-cli installed.
  docker run -v /var/run/docker.sock:/var/run/docker.sock -ti --entrypoint bash samcli:latest
• Run sam Build command sam build -u -t template.yml
• Expected: Sam build should be able to executed with --use-container option
• Actual: Getting an error with Requirements file not found: /tmp/samcli/source/requirements.txt

root@d2da5d926823:/# sam build -u -t aws.cfn.app.yml
Starting Build inside a container
Building codeuri: / runtime: python3.8 metadata: {} functions: ['KafkaLagLambdaFn']
Fetching public.ecr.aws/sam/build-python3.8:latest Docker container image......
Mounting / as /tmp/samcli/source:ro,delegated inside runtime container
Build Failed
Running PythonPipBuilder:ResolveDependencies
Error: PythonPipBuilder:ResolveDependencies - Requirements file not found: /tmp/samcli/source/requirements.txt

[aahung]

Do you have a "requirements.txt" file in your function's "CodeUri" directory?

[atreyd]

Do you have a "requirements.txt" file in your function's "CodeUri" directory?

Yes I do have the requirements.txt in the CodeUri directory. This is working when building from local setup. But not working only when running inside ba container

[aahung]

Your CodeUri seems off from the log. It shouldn't be /?

Building codeuri: / runtime: python3.8 metadata: {} functions: ['KafkaLagLambdaFn']

Can you post the snippet of your template file responsible for the function "KafkaLagLambdaFn?"

[aahung]

Update: I managed to reproduce this error running sam in the docker. It has something to do with how docker mount volumes: https://stackoverflow.com/questions/31381322/docker-in-docker-cannot-mount-volume

[sd65]

Hi all,

@aahung I got the same error with a nodejs Lambda :

$ sam build --use-container
Starting Build inside a container
Building codeuri: /builds/xxx/app runtime: nodejs14.x metadata: {} functions: ['yyy']
Fetching public.ecr.aws/sam/build-nodejs14.x:latest Docker container image......
Mounting /builds/xxx/app as /tmp/samcli/source:ro,delegated inside runtime container
Build Failed
Running NodejsNpmBuilder:NpmPack
Error: NodejsNpmBuilder:NpmPack - NPM Failed: npm ERR! code ENOLOCAL
npm ERR! Could not install from "../source" as it does not contain a package.json file.
npm ERR! A complete log of this run can be found in:
npm ERR! /root/.npm/_logs/2021-05-25T11_23_12_715Z-debug.log

As you said sam is not really to blame here, it's a docker-in-docker catch : the sam container is trying to bind mount the working dir. But the mount path has to be correct from the host perspective, and chances are that's not possible without doing nasty things (like recreating the docker volume driver path/filesystem in the container).

One nice thing would be able to add additional docker volumes to a sam container (should we add a feature request ?). That's almost already implemented but not available in the SAM build CLI interface. The nice thing about named docker volumes are that they don't suffer from the host path perspective "problem".

Finally, what I recommend to fix this issue if you are running sam in a CI/CD container runner : don't use your usual Docker image for your CI/CD container runner docker executor and directly use AWS SAM lambda container images (the images ran if you specify --use-container). For GitLab, that's the image keyword.
They already contains sam and other tools to build and manage dependency in your lambda runtime choice. Find those images here.
That way, if your function depends on packages have natively compiled dependencies, they will be built in AWS Lambda-like Docker container, as --use-container does. That way, you can remove this flag from your CI/CD script and now it works !

[paul-michalik]

@sd65 I ran into this issue and created #3062 since this one did not pop up in my search... You are right, the mounted volume must match the path from the host. Is there a way to inject the host path into "sam build" command via command line options or environment variables?

[mew1033]

This is also a problem when building functions that use container images (https://docs.aws.amazon.com/lambda/latest/dg/images-create.html).