question: Running lambda in CI using REMOTE_DOCKER=true

[peterabbott]

Is there an existing issue for this?

• I have searched the existing issues and read the documentation

Question

I have stared at this problem too long, I was wanting to see if anybody else has this scenario.

We have a setup that uses lambdas along with a bunch of other things. Our CI pipeline runs some integration tests which rely on those lambdas to processes some SQS messages.

Initially we were running in local mode where we can map the code into the container that executes the lambda. Useful for development albeit really slow. The local mode does not appear to work when using docker-reuse for the lambda execution.

Locally I can get the setup working using docker-reuse executor mode, where the packages are pulled from S3. Starting up localstack in CircleCI there seems to be an issue with configuring the lambda functions.

Any call to awslocal lambda, for example awslocal lambda list-functions causes a 500 error. All the other commands work.

The error from localstack is that it is getting a connection refused from localstack:4566. If I connect to the CircleCI over ssh and poke around, localstack is running and I can execute curl http://localstack:4566/2015-03-31/functions/ and get a response. But running awslocal lambda list-functions returns a 500.

I have tried configuring all sorts of environment variables mentioned in the docs that relate to hostnames and networks

This is the stack from the localstack logs:

2021-10-28T02:26:26:WARNING:localstack.utils.server.http2_server: Error in proxy handler for request GET http://localhost:4566/2015-03-31/functions/: MyHTTPConnectionPool(host='localhost', port=49247): Max retries exceeded with url: /2015-03-31/functions/ (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fcd9c61cf10>: Failed to establish a new connection: [Errno 111] Connection refused')) Traceback (most recent call last):
  File "/opt/code/localstack/.venv/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/opt/code/localstack/.venv/lib/python3.7/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/opt/code/localstack/.venv/lib/python3.7/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/code/localstack/.venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/opt/code/localstack/.venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/opt/code/localstack/.venv/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/lib/python3.7/http/client.py", line 1277, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1323, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1272, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1032, in _send_output
    self.send(msg)
  File "/usr/lib/python3.7/http/client.py", line 972, in send
    self.connect()
  File "/opt/code/localstack/.venv/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/opt/code/localstack/.venv/lib/python3.7/site-packages/urllib3/connection.py", line 187, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fcd9c61cf10>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/code/localstack/.venv/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/opt/code/localstack/.venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 756, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/opt/code/localstack/.venv/lib/python3.7/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: MyHTTPConnectionPool(host='localhost', port=49247): Max retries exceeded with url: /2015-03-31/functions/ (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fcd9c61cf10>: Failed to establish a new connection: [Errno 111] Connection refused'))

Anything else?

No response

[peterabbott]

An example of it running in practice.

https://app.circleci.com/pipelines/github/peterabbott/compose-localstack/4/workflows/ed42f9de-6cf6-4a8a-8b8f-6cb4663906a3/jobs/4?invite=true#step-106-661

Setup:
https://github.com/peterabbott/compose-localstack

Will be happy if it is simply user error in my config anybody can spot. Tried all sorts things in the compose setup

[peterabbott]

Turns out the issue was I had assumed that the remote mode would require DOCKER_HOST to be supplied. Taking it out fixed the issue since the docker socket is already mapped in to the localstack container.

After days of trying various things, Rubber Duck debugging wins again!