Bug: "Timed out while attempting to establish a connection to the container" error after fresh install of Docker and AWS SAM

[michaelnicol]

I just installed AWS SAM and Docker and am attempting to work with a basic Lambda Function. The issue is that the Lambda fails to mount on the Docker container.

Set up

I just installed the latest version of docker:

Docker version 24.0.5, build 24.0.5-0ubuntu1~20.04.1

I also installed the latest version of SAM:

SAM CLI, version 1.116.0

I then created a very basic lambda function:

exports.handler = async (event) => {
  console.log("Lambda | handler | Called");
  return {
      statusCode: 200,
      body: JSON.stringify({ message: "Hello from Lambda!" }),
  };
};

And a YAML:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Basic SAM App

Resources:
  BasicFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: .
      Handler: app.handler
      Runtime: nodejs20.x
      Events:
        BasicApi:
          Type: Api
          Properties:
            Path: /basic
            Method: get

Outputs:
  BasicApiUrl:
    Description: "API Gateway endpoint URL for the Basic function"
    Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/basic/"
  BasicFunctionArn:
    Description: "Basic Lambda Function ARN"
    Value: !GetAtt BasicFunction.Arn

Usage & Errors:

The app builds fine via sam build:

Building codeuri: /home/mnicol/Documents/samtest runtime: nodejs20.x metadata: {} architecture: x86_64 functions: BasicFunction                                                            
 Running NodejsNpmBuilder:NpmPack                                                                                                                                                          
 Running NodejsNpmBuilder:CopyNpmrcAndLockfile                                                                                                                                             
 Running NodejsNpmBuilder:CopySource                                                                                                                                                       
 Running NodejsNpmBuilder:NpmInstall                                                                                                                                                       
 Running NodejsNpmBuilder:CleanUpNpmrc                                                                                                                                                     
 Running NodejsNpmBuilder:LockfileCleanUp                                                                                                                                                  

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Validate SAM template: sam validate
[*] Invoke Function: sam local invoke
[*] Test Function in the Cloud: sam sync --stack-name {{stack-name}} --watch
[*] Deploy: sam deploy --guided

The issue is that when I run sam local invoke, the program hangs on the "Mounting" stage and times out.

(base) mnicol@COECE-057000D:~/Documents/samtest$ sam local invoke --debug
2024-05-22 12:12:03,915 | Config file location: /home/mnicol/Documents/samtest/samconfig.toml                                                                                              
2024-05-22 12:12:03,922 | Loading configuration values from [default.['local', 'invoke'].parameters] (env.command_name.section) in config file at                                          
'/home/mnicol/Documents/samtest/samconfig.toml'...                                                                                                                                         
2024-05-22 12:12:03,924 | Configuration values successfully loaded.                                                                                                                        
2024-05-22 12:12:03,926 | Configuration values are: {}                                                                                                                                     
2024-05-22 12:12:03,932 | Using SAM Template at /home/mnicol/Documents/samtest/.aws-sam/build/template.yaml                                                                                
2024-05-22 12:12:03,979 | Using config file: samconfig.toml, config environment: default                                                                                                   
2024-05-22 12:12:03,980 | Expand command line arguments to:                                                                                                                                
2024-05-22 12:12:03,982 | --template_file=/home/mnicol/Documents/samtest/.aws-sam/build/template.yaml --no_event --layer_cache_basedir=/home/mnicol/.aws-sam/layers-pkg                    
--container_host=localhost --container_host_interface=127.0.0.1                                                                                                                            
2024-05-22 12:12:03,984 | local invoke command is called                                                                                                                                   
2024-05-22 12:12:03,990 | No Parameters detected in the template                                                                                                                           
2024-05-22 12:12:04,082 | Sam customer defined id is more priority than other IDs. Customer defined id for resource BasicFunction is BasicFunction                                         
2024-05-22 12:12:04,084 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use the resource logical id as the resource id                    
2024-05-22 12:12:04,087 | 0 stacks found in the template                                                                                                                                   
2024-05-22 12:12:04,089 | No Parameters detected in the template                                                                                                                           
2024-05-22 12:12:04,133 | Sam customer defined id is more priority than other IDs. Customer defined id for resource BasicFunction is BasicFunction                                         
2024-05-22 12:12:04,135 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use the resource logical id as the resource id                    
2024-05-22 12:12:04,137 | 2 resources found in the stack                                                                                                                                   
2024-05-22 12:12:04,138 | Found Serverless function with name='BasicFunction' and CodeUri='BasicFunction'                                                                                  
2024-05-22 12:12:04,140 | --base-dir is not presented, adjusting uri BasicFunction relative to /home/mnicol/Documents/samtest/.aws-sam/build/template.yaml                                 
2024-05-22 12:12:04,150 | Found one Lambda function with name 'BasicFunction'                                                                                                              
2024-05-22 12:12:04,152 | Invoking app.handler (nodejs20.x)                                                                                                                                
2024-05-22 12:12:04,154 | No environment variables found for function 'BasicFunction'                                                                                                      
2024-05-22 12:12:04,155 | Loading AWS credentials from session with profile 'None'                                                                                                         
2024-05-22 12:12:04,177 | Resolving code path. Cwd=/home/mnicol/Documents/samtest/.aws-sam/build, CodeUri=/home/mnicol/Documents/samtest/.aws-sam/build/BasicFunction                      
2024-05-22 12:12:04,179 | Resolved absolute path to code is /home/mnicol/Documents/samtest/.aws-sam/build/BasicFunction                                                                    
2024-05-22 12:12:04,180 | Resolving code path. Cwd=/home/mnicol/Documents/samtest/.aws-sam/build, CodeUri=/home/mnicol/Documents/samtest/.aws-sam/build/BasicFunction                      
2024-05-22 12:12:04,182 | Resolved real code path to /home/mnicol/Documents/samtest/.aws-sam/build/BasicFunction                                                                           
2024-05-22 12:12:04,183 | Code /home/mnicol/Documents/samtest/.aws-sam/build/BasicFunction is not a zip/jar file                                                                           
2024-05-22 12:12:05,276 | Local image is up-to-date                                                                                                                                        
2024-05-22 12:12:05,290 | Checking free port on 127.0.0.1:5776                                                                                                                             
2024-05-22 12:12:05,296 | Using local image: public.ecr.aws/lambda/nodejs:20-rapid-x86_64.                                                                                                 
                                                                                                                                                                                           
2024-05-22 12:12:05,299 | Mounting /home/mnicol/Documents/samtest/.aws-sam/build/BasicFunction as /var/task:ro,delegated, inside runtime container                                         
2024-05-22 12:14:15,314 | [Container state] OOMKilled False                                                                                                                                
2024-05-22 12:14:15,876 | Cleaning all decompressed code dirs                                                                                                                              
2024-05-22 12:14:15,879 | Timed out while attempting to establish a connection to the container. You can increase this timeout by setting the SAM_CLI_CONTAINER_CONNECTION_TIMEOUT         
environment variable. The current timeout is 20.0 (seconds).                                                                                                                               
2024-05-22 12:14:15,882 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics                                                       
2024-05-22 12:14:15,901 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics                                                       
2024-05-22 12:14:15,903 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': '7d6d182b-57cc-4891-9327-584c9a66b9bb', 'installationId': '95421c31-2aea-40e7-9e7d-9f2cbe723bfe',   
'sessionId': 'a36e3464-e364-4d52-a876-924656dc1a41', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.11.8', 'samcliVersion': '1.116.0', 'awsProfileProvided': False,           
'debugFlagProvided': True, 'region': '', 'commandName': 'sam local invoke', 'metricSpecificAttributes': {'projectType': 'CFN', 'gitOrigin': None, 'projectName':                           
'9077a2ede8e4875966ec89aec30503e4979f9a0bcf4cdfdce0b4c7a179127402', 'initialCommit': None}, 'duration': 131903, 'exitReason': 'success', 'exitCode': 0}}]}                                 
2024-05-22 12:14:15,903 | Unable to find Click Context for getting session_id.                                                                                                             
2024-05-22 12:14:15,916 | Sending Telemetry: {'metrics': [{'events': {'requestId': 'b563867b-0767-499b-be91-03c24c62970c', 'installationId': '95421c31-2aea-40e7-9e7d-9f2cbe723bfe',       
'sessionId': 'a36e3464-e364-4d52-a876-924656dc1a41', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.11.8', 'samcliVersion': '1.116.0', 'commandName': 'sam local invoke',     
'metricSpecificAttributes': {'events': [{'event_name': 'SamConfigFileExtension', 'event_value': '.toml', 'thread_id': '2eb82ff0705c4440826af8694d89a149', 'time_stamp': '2024-05-22        
16:12:03.915', 'exception_name': None}, {'event_name': 'SamConfigFileExtension', 'event_value': '.toml', 'thread_id': '8fdf72c122554d4d8c78b360badc3e01', 'time_stamp': '2024-05-22        
16:12:03.979', 'exception_name': None}]}}}]}                                                                                                                                               
2024-05-22 12:14:16,269 | Telemetry response: 200                                                                                                                                          
2024-05-22 12:14:16,271 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)                                 
(base) mnicol@COECE-057000D:~/Documents/samtest$ 

[lucashuy]

Thanks for reporting this, can you paste the contents of sam --info into a code block here?

I see that you mention it gets stuck when mounting, if you run sam local invoke again, and while it is waiting to mount could you do a quick docker ps to see if any node 20 containers were created before the command exists?

[michaelnicol]

While it was mounting, I ran sudo docker ps and got:

(base) mnicol@COECE-057000D:~/Documents/samtest$ sudo docker ps
[sudo] password for mnicol: 
CONTAINER ID   IMAGE                                          COMMAND                  CREATED          STATUS          PORTS                                                  NAMES
558462bdc785   public.ecr.aws/lambda/nodejs:20-rapid-x86_64   "/var/rapid/aws-lamb…"   52 seconds ago   Up 50 seconds   127.0.0.1:6816->8080/tcp                               cool_panini
0b5f96d489e2   mysql:latest                                   "docker-entrypoint.s…"   33 minutes ago   Up 33 minutes   0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060/tcp   mysqlkeycloak

After it timed out, I checked again, and the container was no longer there.

Output from sam --info

(base) mnicol@COECE-057000D:~/Documents/samtest$ sam --info
{
  "version": "1.116.0",
  "system": {
    "python": "3.11.8",
    "os": "Linux-5.4.0-182-generic-x86_64-with-glibc2.31"
  },
  "additional_dependencies": {
    "docker_engine": "24.0.5",
    "aws_cdk": "Not available",
    "terraform": "Not available"
  },
  "available_beta_feature_env_vars": [
    "SAM_CLI_BETA_FEATURES",
    "SAM_CLI_BETA_BUILD_PERFORMANCE",
    "SAM_CLI_BETA_TERRAFORM_SUPPORT",
    "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
  ]
}

[lucashuy]

Thanks for getting back with the additional info, we'll have to investigate this further, it looks like its failing to create a raw socket connection to the port the container spins up as part of the validation we do before invoking. In the meantime, can you try bumping the timeout before it fails using SAM_CLI_CONTAINER_CONNECTION_TIMEOUT? By default it is set to 20 seconds, running something like

SAM_CLI_CONTAINER_CONNECTION_TIMEOUT=60 sam local invoke --debug

will force the logic to wait a minute instead.

[0xsudo]

Facing the same issue, thought it was awscliv1. So, I upgraded to v2, did a fresh installation of both aws and sam and made sure docker is enabled.

uname -srvmpio : Linux 6.5.0-35-generic #35~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue May 7 09:00:52 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

docker --version : Docker version 26.1.3, build b72abbb

sam --version : SAM CLI, version 1.117.0

aws --version : aws-cli/2.15.58 Python/3.11.8 Linux/6.5.0-35-generic exe/x86_64.ubuntu.22

Even after trying @lucashuy's suggestion, still getting an immediate timeout:
$ SAM_CLI_CONTAINER_CONNECTION_TIMEOUT=60 sam local invoke --debug
2024-05-28 19:30:49,684 | Config file location: /home/kaoka/Documents/ht/project/project-source/source/samconfig.toml
2024-05-28 19:30:49,695 | Loading configuration values from [default.['local', 'invoke'].parameters] (env.command_name.section) in config file at '/home/kaoka/Documents/ht/project/project-source/source/samconfig.toml'...
2024-05-28 19:30:49,699 | Configuration values successfully loaded.
2024-05-28 19:30:49,701 | Configuration values are: {'stack_name': 'source', 'region': 'us-west-2', 'profile': 'access'}
2024-05-28 19:30:49,727 | Using SAM Template at /home/kaoka/Documents/ht/project/project-source/source/.aws-sam/build/template.yaml
2024-05-28 19:30:49,859 | Using config file: samconfig.toml, config environment: default
2024-05-28 19:30:49,861 | Expand command line arguments to:
2024-05-28 19:30:49,864 | --template_file=/home/kaoka/Documents/ht/project/project-source/source/.aws-sam/build/template.yaml --no_event --layer_cache_basedir=/home/kaoka/.aws-sam/layers-pkg --container_host=localhost --container_host_interface=127.0.0.1
2024-05-28 19:30:49,868 | local invoke command is called
2024-05-28 19:30:49,894 | No Parameters detected in the template
2024-05-28 19:30:49,974 | Sam customer defined id is more priority than other IDs. Customer defined id for resource projectsourceConnFunction is projectsourceConnFunction
2024-05-28

19:30:49,977 | There is no customer defined id or cdk path defined for resource projectsourceConnFunctionsourceTriggerPermissions, so we will use the resource logical id as the resource id
2024-05-28 19:30:49,980 | There is no customer defined id or cdk path defined for resource projectProcessBackupCompletionNotificationRoleForsource, so we will use the resource logical id as the resource id
2024-05-28 19:30:49,986 | 0 stacks found in the template
2024-05-28 19:30:49,988 | No Parameters detected in the template
2024-05-28 19:30:50,064 | Sam customer defined id is more priority than other IDs. Customer defined id for resource projectsourceConnFunction is projectsourceConnFunction
2024-05-28 19:30:50,067 | There is no customer defined id or cdk path defined for resource projectsourceConnFunctionsourceTriggerPermissions, so we will use the resource logical id as the resource id
2024-05-28 19:30:50,069 | There is no customer defined id or cdk path defined for resource projectProcessBackupCompletionNotificationRoleForsource, so we will use the resource logical id as the resource id
2024-05-28 19:30:50,074 | 3 resources found in the stack
2024-05-28 19:30:50,077 | Found Serverless function with name='projectsourceConnFunction' and ImageUri='projectsourceconnfunction:v1.0.0'
2024-05-28 19:30:50,080 | --base-dir is not presented, adjusting uri /home/kaoka/Documents/ht/project/project-source/source/source relative to /home/kaoka/Documents/ht/project/project-source/source/.aws-sam/build/template.yaml
2024-05-28 19:30:50,083 | --base-dir is not presented, adjusting uri . relative to /home/kaoka/Documents/ht/project/project-source/source/.aws-sam/build/template.yaml
2024-05-28 19:30:50,102 | Found one Lambda function with name 'projectsourceConnFunction'
2024-05-28 19:30:50,105 | Invoking Container created from projectsourceconnfunction:v1.0.0
2024-05-28 19:30:50,108 | Loading AWS credentials from session with profile 'access'
2024-05-28 19:30:54,599 | Code None is not a zip/jar file
2024-05-28 19:30:54,607 | Local image was not found.
2024-05-28 19:30:54,609 | Removing rapid images for repo projectsourceconnfunction
Building image...............
2024-05-28 19:30:55,875 | Failed to build Docker Image
NoneType: None
2024-05-28 19:30:55,881 | Cleaning all decompressed code dirs
2024-05-28 19:30:55,884 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2024-05-28 19:30:55,934 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2024-05-28 19:30:55,937 | Unable to find Click Context for getting session_id.
2024-05-28 19:30:55,943 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': '91a73f08-cfe9-42b8-a6a7-626486445662', 'installationId': 'd664b28c-9a6b-4c07-b9d6-3d8e72a49673', 'sessionId': 'cc023f96-f7cb-4389-910e-85c864028c42', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.11.8', 'samcliVersion': '1.117.0', 'awsProfileProvided': True, 'debugFlagProvided': True, 'region': 'us-west-2', 'commandName': 'sam local invoke', 'metricSpecificAttributes': {'projectType': 'CFN', 'gitOrigin': None, 'projectName': '6ffe9bce28cdff262b30c342183ba88015e3cd094cb35d105067c518430534cb','initialCommit': None}, 'duration': 6024, 'exitReason': 'ImageBuildException', 'exitCode': 1}}]} 2024-05-28 19:30:55,951 | Sending Telemetry: {'metrics': [{'events': {'requestId': '1d8e2aba-f074-461f-9e7c-c4435e0e5079', 'installationId': 'd664b28c-9a6b-4c07-b9d6-3d8e72a49673', 'sessionId': 'cc023f96-f7cb-4389-910e-85c864028c42', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.11.8', 'samcliVersion': '1.117.0', 'commandName': 'sam local invoke', 'metricSpecificAttributes': {'events': [{'event_name': 'SamConfigFileExtension', 'event_value': '.toml', 'thread_id': 'ba0066cbc4c642aaa0f7220abc9661ee', 'time_stamp': '2024-05-28 16:30:49.683', 'exception_name': None}, {'event_name': 'SamConfigFileExtension', 'event_value': '.toml', 'thread_id': '422868ebff7e47578a98b17140b389d3', 'time_stamp': '2024-05-28 16:30:49.858', 'exception_name': None}]}}}]}
2024-05-28 19:30:57,113 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1) Error: Error building docker image: The command '/bin/sh -c mv /var/rapid/aws-lambda-rie-x86_64 /var/rapid/aws-lambda-rie && chmod +x /var/rapid/aws-lambda-rie' returned a non-zero code: 1
2024-05-28 19:30:57,118 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)

Forgive the indent lol

Btw tried:
$ export SAM_CLI_CONTAINER_CONNECTION_TIMEOUT=60 && sam local invoke --debug
And still getting same outcome

[michaelnicol]

Any updates? Because of this, I still can't use AWS SAM on my Linux computer.

How can I downgrade my version of AWS SAM to a working version?