Recoverable error occurred (Inaccessible host: `engine-localstack' at port `undefined'... only when running from within dockerfile script

[mishabruml]

serverless.yml

  localstack:
    stages:
      - dev
    host: http://engine-localstack
    edgePort: 4566
    autostart: false

docker-compose.yml

localstack:
    image: localstack/localstack:1.3.1
    hostname: engine-localstack
    container_name: engine-localstack
    restart: on-failure
    environment:
      - SERVICES=s3,sqs,lambda
      - DEFAULT_REGION=eu-west-1
      - DOCKER_HOST=unix:///var/run/docker.sock
      - DEBUG=true
      - LAMBDA_DOCKER_NETWORK=localstack-network
      - PROVIDER_OVERRIDE_LAMBDA=asf # https://github.com/localstack/localstack/issues/7209
      - HOSTNAME=engine-localstack
    ports:
      - '4566:4566'
      - '4571:4571'
    volumes:
      - '${TMPDIR:-/tmp/localstack}:/var/lib/localstack'
      - '/var/run/docker.sock:/var/run/docker.sock'

  localstack-serverless-deployer:
    hostname: engine-localstack-serverless-deployer
    container_name: engine-localstack-serverless-deployer
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    build:
      dockerfile: localstack/Dockerfile.localstack-serverless-deployer
    depends_on:
      - localstack

Dockerfile.localstack-serverless-deployer

# Container to deploy serverless stack to localhost
FROM node:18 AS docker-build
WORKDIR /engine
COPY package*.json ./
RUN npm install
COPY tsconfig.json ./
COPY src ./src
COPY serverless.yml serverless.yml
RUN npx sls deploy --verbose 

From a clean docker desktop env, run: docker-compose up -d localstack
And then docker compose build localstack-serverless-deployer --progress plain

This yeilds logs: #12 9.870 Recoverable error occurred (Inaccessible host: 'engine-localstack' at port 'undefined'. This service may not be available in the 'eu-west-1' region.), sleeping for ~6 seconds. Try 1 of 4

Full logs

#1 [internal] load build definition from Dockerfile.localstack-serverless-deployer
#1 transferring dockerfile: 63B done
#1 DONE 0.1s

#2 [internal] load .dockerignore
#2 transferring context: 35B 0.0s done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/library/node:18
#3 DONE 1.3s

#4 [1/8] FROM docker.io/library/node:18@sha256:d871edd5b68105ebcbfcde3fe8c79d24cbdbb30430d9bd6251c57c56c7bd7646
#4 DONE 0.0s

#5 [internal] load build context
#5 transferring context: 25.39kB 0.2s done
#5 DONE 0.2s

#6 [3/8] COPY package*.json ./
#6 CACHED

#7 [4/8] RUN npm install
#7 CACHED

#8 [5/8] COPY tsconfig.json ./
#8 CACHED

#9 [6/8] COPY src ./src
#9 CACHED

#10 [2/8] WORKDIR /engine
#10 CACHED

#11 [7/8] COPY serverless.yml serverless.yml
#11 CACHED

#12 [8/8] RUN npx sls deploy --verbose
#12 5.908 Using serverless-localstack
#12 9.436 
#12 9.436 Deploying engine to stage dev (eu-west-1)
#12 9.437 
#12 9.479 Compiling to node18 bundle with esbuild...
#12 9.483 Compiling with concurrency: Infinity
#12 9.870 Recoverable error occurred (Inaccessible host: `engine-localstack' at port `undefined'. This service may not be available in the `eu-west-1' region.), sleeping for ~6 seconds. Try 1 of 4
#12 13.83 Compiling completed.
#12 13.84 Warning: No external modules needed
#12 16.40 Recoverable error occurred (Inaccessible host: `engine-localstack' at port `undefined'. This service may not be available in the `eu-west-1' region.), sleeping for ~5 seconds. Try 2 of 4
#12 22.96 Recoverable error occurred (Inaccessible host: `engine-localstack' at port `undefined'. This service may not be available in the `eu-west-1' region.), sleeping for ~5 seconds. Try 3 of 4
#12 23.61 Zip function: run-db-migrations - 11.19 MB [1766 ms]
#12 23.62 Zip function: test - 11.19 MB [4322 ms]
#12 24.18 Recoverable error occurred (Inaccessible host: `engine-localstack' at port `undefined'. This service may not be available in the `eu-west-1' region.), sleeping for ~6 seconds. Try 1 of 4
#12 28.48 Recoverable error occurred (Inaccessible host: `engine-localstack' at port `undefined'. This service may not be available in the `eu-west-1' region.), sleeping for ~4 seconds. Try 4 of 4
#12 30.14 Recoverable error occurred (Inaccessible host: `engine-localstack' at port `undefined'. This service may not be available in the `eu-west-1' region.), sleeping for ~5 seconds. Try 2 of 4
#12 35.64 Recoverable error occurred (Inaccessible host: `engine-localstack' at port `undefined'. This service may not be available in the `eu-west-1' region.), sleeping for ~6 seconds. Try 3 of 4
#12 42.63 Recoverable error occurred (Inaccessible host: `engine-localstack' at port `undefined'. This service may not be available in the `eu-west-1' region.), sleeping for ~4 seconds. Try 4 of 4
#12 47.23 
#12 47.23 × Stack engine-dev failed to deploy (37s)
#12 47.23 Environment: linux, node 18.13.0, framework 3.27.0 (local), plugin 6.2.3, SDK 4.3.2
#12 47.23 Credentials: Local, environment variables
#12 47.23 Docs:        docs.serverless.com
#12 47.23 Support:     forum.serverless.com
#12 47.23 Bugs:        github.com/serverless/serverless/issues
#12 47.23 
#12 47.23 Error:
#12 47.23 Inaccessible host: `engine-localstack' at port `undefined'. This service may not be available in the `eu-west-1' region.
#12 ERROR: executor failed running [/bin/sh -c npx sls deploy --verbose]: exit code: 1
------
 > [8/8] RUN npx sls deploy --verbose:
#12 47.23 
#12 47.23 × Stack engine-dev failed to deploy (37s)
#12 47.23 Environment: linux, node 18.13.0, framework 3.27.0 (local), plugin 6.2.3, SDK 4.3.2
#12 47.23 Credentials: Local, environment variables
#12 47.23 Docs:        docs.serverless.com
#12 47.23 Support:     forum.serverless.com
#12 47.23 Bugs:        github.com/serverless/serverless/issues
#12 47.23 
#12 47.23 Error:
#12 47.23 Inaccessible host: `engine-localstack' at port `undefined'. This service may not be available in the `eu-west-1' region.
------
failed to solve: executor failed running [/bin/sh -c npx sls deploy --verbose]: exit code: 1

What is super interesting/annoying about this issue is that with the exact same setup, running
docker compose run localstack-serverless-deployer npx sls deploy --verbose works!

Full logs

$ docker compose run localstack-serverless-deployer npx sls deploy --verbose
[+] Running 1/0
 ⠿ Container engine-localstack  Running 0.0s
Using serverless-localstack

Deploying engine to stage dev (eu-west-1)

Compiling to node18 bundle with esbuild...
Compiling with concurrency: Infinity
Compiling completed.
Warning: No external modules needed
Zip function: run-db-migrations - 11.19 MB [1307 ms]
Zip function: test - 11.19 MB [3454 ms]
Creating new change set
Waiting for new change set to be created
Executing created change set
  CREATE_IN_PROGRESS - AWS::CloudFormation::Stack - engine-dev
  CREATE_COMPLETE - AWS::CloudFormation::Stack - ServerlessDeploymentBucket
  CREATE_COMPLETE - AWS::CloudFormation::Stack - ServerlessDeploymentBucketPolicy
  CREATE_COMPLETE - AWS::CloudFormation::Stack - engine-dev
Uploading CloudFormation file to S3
Uploading State file to S3
Uploading service test.zip file to S3 (11.73 MB)
Uploading service run-db-migrations.zip file to S3 (11.74 MB)
Skipping template validation: Unsupported in Localstack
Creating new change set
Waiting for new change set to be created
Executing created change set
  UPDATE_IN_PROGRESS - AWS::CloudFormation::Stack - ServerlessDeploymentBucketPolicy
  CREATE_COMPLETE - AWS::CloudFormation::Stack - TestLogGroup
  CREATE_COMPLETE - AWS::CloudFormation::Stack - RunDashdbDashmigrationsLogGroup
  CREATE_COMPLETE - AWS::CloudFormation::Stack - IamRoleLambdaExecution
  CREATE_COMPLETE - AWS::CloudFormation::Stack - TestLambdaFunction
  CREATE_COMPLETE - AWS::CloudFormation::Stack - RunDashdbDashmigrationsLambdaFunction
  CREATE_COMPLETE - AWS::CloudFormation::Stack - TestLambdaVersionWY5FGxphepHypRgXaj3G8Qwr3LyYTwGwS0Tvgpn0
  CREATE_COMPLETE - AWS::CloudFormation::Stack - RunDashdbDashmigrationsLambdaVersionoh86AQf7Yu64pCOHL0pgFYErkvztrfDIhVccbDND3ww
  UPDATE_COMPLETE - AWS::CloudFormation::Stack - ServerlessDeploymentBucket
  UPDATE_COMPLETE - AWS::CloudFormation::Stack - ServerlessDeploymentBucketPolicy
  UPDATE_COMPLETE - AWS::CloudFormation::Stack - engine-dev
Removing old service artifacts from S3

✔ Service deployed to stack engine-dev (30s)

Need a better logging experience than CloudWatch? Try our Dev Mode in console: run "serverless --console"

[mishabruml]

Update:

changing the localstack host config in serverless.yml to host: http://host.docker.internal allows the npx sls deploy command to succeed in the context of a docker build script.

This solution isn't desired though as host.docker.internal can't be resolved on all systems, and isn't designed for use outside of docker desktop. We are looking to run these deployments in CI as part of test suite

[whummer]

Thanks for reporting @mishabruml . Have you been able to verify if there is connectivity from the localstack-serverless-deployer container to the engine-localstack host on port 4566? You could exec into the container and give it a try, e.g., by running a simple curl engine-localstack:4566 command. Hope that helps - please keep us posted on how it goes.. Thanks!

[steffyP]

Hi @mishabruml,

could you provide an update on your issue? Were you able to verify the connectivity to the container?

[steffyP]

This issue seems to be stale. As we didn't get any response in two months, we are closing this issue now. Please feel free to reopen if it still persists, and provide updates logs and details.