sam build --use-container fails hello_world when behind proxy.

[DevOpsChris]

Description:

The hello_world example bombs everytime when following the written instructions when behind a proxy. Need to create a proxy flag or introduce passing proxy envs with the build command. Better yet, honor the .docker/config.json and service settings in your container code.

Also, really need to make logging or stderr MUCH better in https://github.com/awslabs/aws-lambda-builders, because it just says

{"jsonrpc": "2.0", "id": 1, "error": {"code": 400, "message": "PythonPipBuilder:ResolveDependencies - Could not satisfy the requirement: requests==2.20.0"}}

instead of

Collecting foo
  Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x10c04e780>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known')': /simple/foo/
  Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x10c04e588>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known')': /simple/foo/
......
  Could not find a version that satisfies the requirement foo (from versions: )
No matching distribution found for foo

It really misleads people when debugging.

Dirtiest Fix, put in proxy envs in env dictionary https://github.com/DevOpsChris/aws-sam-cli/blob/develop/samcli/local/docker/lambda_build_container.py#L77-L81

Best fix: Honor the environment settings the docker service provides.

Steps to reproduce the issue:

1. Follow the 0.10.0 hello world instructions
2. sam build --use-container

Observed result:

aws_lambda_builders.exceptions.WorkflowFailedError: PythonPipBuilder:ResolveDependencies - Could not satisfy the requirement: requests==2.20.0
2019-01-04 13:37:35 Build inside container returned response {"jsonrpc": "2.0", "id": 1, "error": {"code": 400, "message": "PythonPipBuilder:ResolveDependencies - Could not satisfy the requirement: requests==2.20.0"}}
Build Failed
Error: PythonPipBuilder:ResolveDependencies - Could not satisfy the requirement: requests==2.20.0

Expected result:

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Package: sam package --s3-bucket <yourbucket>

Running PythonPipBuilder:ResolveDependencies
Running PythonPipBuilder:CopySource

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

• Mac OSX: 10.13.6
• Docker: Version 2.0.0.0-mac81 (29211)

Output of sam --version: SAM CLI, version 0.10.0

[DevOpsChris]

Updated the issue.

[DevOpsChris]

Ugh. I'll put the dirty solution in the morning. I deleted my repo on purpose forgetting that it was already forked.

Planning to create a few fixes in the morning.

[metaskills]

FWIW, I got this error too but my issue was related to using Bundler v2.0.1 locally. Using this command below:

docker run -v $(pwd):/var/task \
  lambci/lambda:build-ruby2.5 \
  bash -c "bundle install --without development test --verbose"

I was able to find that my core issue was You must use Bundler 2 or greater with this lockfile. so in my tooling I just forced use of Bundler 1.17.3 and that fixed my issue. Hope that helps others.

[DevOpsChris]

Here's a fix in progress. https://github.com/DevOpsChris/aws-sam-cli/tree/fix_container_build_proxy

[DevOpsChris]

@metaskills That's definitely a decent dirty fix like mine was in the original post. Doesn't help solve the initial problem of starting the docker environment with the proxy env vars.

I've coded up a way to pass in the vars with this new solution.

[DevOpsChris]

What I've got left to do is write up the tests and this should be in. (fingers crossed)

[DevOpsChris]

I'm not finding the problem anymore on my system. I recommend closing this. I believe my problem was not adding http_proxy and https_proxy into the environment variables in the sam template

[cooper3330]

I'm running into the same issue and unfortunately adding http_proxy and https_proxy to the template.yml global environment variables is not resolving my issue. They don't seem to be set in the container when the pip install on my manifest file (requirements.txt) is called.

Are you able to share an example of your SAM template that resolved the issue after adding proxy env variables? I'm wondering if mine are not placed in the right location of the YAML

[DevOpsChris]

Have you set your proxy settings in the docker config? https://docs.docker.com/network/proxy/#configure-the-docker-client

[cooper3330]

Yep, proxy settings are set in the docker config.

After setting and restarting Docker to make sure it picks up the changes, I find when manually starting a Docker container, the proxy settings are getting set correctly and I have no issues hitting PIP:

Manual Docker Proxy Test

$ docker run -it --name test-proxy lambci/lambda:build-python3.6 /bin/bash
bash-4.2# env
...
NO_PROXY=localhost,*.<corporate-path>.com
...
http_proxy=http://<corporate-proxy-path>.proxy.<corporate-path>.com:80
https_proxy=http://<corporate-proxy-path>.proxy.<corporate-path>.com:80
HTTPS_PROXY=http://<corporate-proxy-path>.proxy.<corporate-path>.com:80
...
no_proxy=localhost,*.<corporate-path>.com
HTTP_PROXY=http://<corporate-proxy-path>.proxy.<corporate-path>.com:80
...
bash-4.2# pip install pandas
Collecting pandas
  Downloading
...
Successfully installed numpy-1.16.3 pandas-0.24.2
bash-4.2# exit

However, when then running a sam build, I'm still not able to hit PIP, presumably due to missing proxy settings since I replicated the same error message by removing proxy settings from docker config:

Sam Build Proxy Issue Remains

$ sam build -m dist/requirements.txt --use-container --debug
2019-05-23 10:25:05 Using SAM Template at /<lambda-path>/template.yml
...
2019-05-23 10:25:05 'build' command is called
2019-05-23 10:25:05 Starting Build inside a container
...
2019-05-23 10:25:05 Trying paths: ['<home-dir>/.docker/config.json', '<home-dir>/.dockercfg']
**2019-05-23 10:25:05 Found file at path: <home-dir>/.docker/config.json**
2019-05-23 10:25:05 Found 'auths' section
...
2019-05-23 10:25:05 Found 'credsStore' section
2019-05-23 10:25:05 Building resource '<lambda-func>'
...
Fetching lambci/lambda:build-python3.6 Docker container image......
...
Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Running workflow 'PythonPipBuilder'
Running PythonPipBuilder:ResolveDependencies
calling pip download -r /tmp/samcli/manifest/requirements.txt --dest /tmp/samcli/scratch
PythonPipBuilder:ResolveDependencies failed
Traceback (most recent call last):
  File "/var/lang/lib/python3.6/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 39, in execute
    self.scratch_dir
  File "/var/lang/lib/python3.6/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 143, in build_dependencies
    requirements_path, artifacts_dir_path, scratch_dir_path)
  File "/var/lang/lib/python3.6/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 209, in build_site_packages
    scratch_directory, requirements_filepath)
  File "/var/lang/lib/python3.6/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 234, in _download_dependencies
    requirements_filename, directory)
  File "/var/lang/lib/python3.6/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 330, in _download_all_dependencies
    self._pip.download_all_dependencies(requirements_filename, directory)
  File "/var/lang/lib/python3.6/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 631, in download_all_dependencies
    raise NoSuchPackageError(str(package_name))
aws_lambda_builders.workflows.python_pip.packager.NoSuchPackageError: Could not satisfy the requirement: dill==0.2.8.2
...
aws_lambda_builders.exceptions.WorkflowFailedError: PythonPipBuilder:ResolveDependencies - Could not satisfy the requirement: dill==0.2.8.2
2019-05-23 10:25:14 Build inside container returned response {"jsonrpc": "2.0", "id": 1, "error": {"code": 400, "message": "PythonPipBuilder:ResolveDependencies - Could not satisfy the requirement: dill==0.2.8.2"}}
...
Build Failed
Error: PythonPipBuilder:ResolveDependencies - Could not satisfy the requirement: dill==0.2.8.2

Still trying to come up with ways to debug this, but I'm running out of ideas at this point. Not sure why the SAM build doesn't seem to be picking up proxy settings from the docker config that it is reading from (based on logs highlighted above).

Edit: Formatting

[cooper3330]

Additional piece of confirmation. SAM build deletes the container, whether the SAM build command completes or you kill the SAM build command mid-execution. So, I started the SAM build in one terminal, and while it was running I executed and "env" command in that container from another terminal. I was able to confirm the proxy settings configured in my Docker Config are not getting picked up:

SAM Build Container Environment Variables - Missing Proxy Variables

$ docker container exec $(docker ps -a -l --format "{{.ID}}") env
PATH=/var/lang/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin
HOSTNAME=7e1408a84fcd
LAMBDA_BUILDERS_LOG_LEVEL=10
LD_LIBRARY_PATH=/var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib:/opt/lib
LANG=en_US.UTF-8
TZ=:UTC
LAMBDA_TASK_ROOT=/var/task
LAMBDA_RUNTIME_DIR=/var/runtime
_LAMBDA_CONTROL_SOCKET=14
_LAMBDA_SHARED_MEM_FD=11
_LAMBDA_LOG_FD=9
_LAMBDA_SB_ID=7
_LAMBDA_CONSOLE_SOCKET=16
_LAMBDA_RUNTIME_LOAD_TIME=1530232235231
_AWS_XRAY_DAEMON_ADDRESS=...
_AWS_XRAY_DAEMON_PORT=2000
AWS_XRAY_DAEMON_ADDRESS=...
AWS_XRAY_CONTEXT_MISSING=LOG_ERROR
_X_AMZN_TRACE_ID=Parent=11560be54abce8ed
AWS_EXECUTION_ENV=AWS_Lambda_python3.6
PYTHONPATH=/var/runtime
PKG_CONFIG_PATH=/var/lang/lib/pkgconfig:/usr/lib64/pkgconfig:/usr/share/pkgconfig
HOME=/root

[DevOpsChris]

here's my env settings in the yaml file.

  somethingLambda:
    Properties:
      ......
      DeploymentPreference:
        ...
      Environment:
        Variables:
          ....
          http_proxy: http://proxyurl:proxyport
          https_proxy: http://proxyurl:proxyport