bug: localstack/localstack:lastest fails to startup

[robo-corg]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

I have been using localstack in a github workflow for CI and it seems to fail to come up now:

  services:
      localstack:
        image: localstack/localstack
        env:
          SERVICES: s3
          DEFAULT_REGION: us-east-1
          DATA_DIR: /tmp/localstack/data
        ports:
        - 4566:4566

Seems to fail from connecting to postgresql

Service container localstack failed.
 /usr/bin/docker logs --details 8dc603486e76bc37f7762a235feb82e480926166c0b4cbebc904bca9e971b5ef
  
  LocalStack version: 3.4.1.dev
  LocalStack Docker container name: localstack-main
  Failed to inspect docker container: Docker not available Traceback (most recent call last):
    File "/opt/code/localstack/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 793, in urlopen
      response = self._make_request(
                 ^^^^^^^^^^^^^^^^^^^
    File "/opt/code/localstack/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 496, in _make_request
      conn.request(
    File "/opt/code/localstack/.venv/lib/python3.11/site-packages/urllib3/connection.py", line 400, in request
      self.endheaders()
    File "/usr/local/lib/python3.11/http/client.py", line 1298, in endheaders
      self._send_output(message_body, encode_chunked=encode_chunked)
    File "/usr/local/lib/python3.11/http/client.py", line 1058, in _send_output
      self.send(msg)
    File "/usr/local/lib/python3.11/http/client.py", line 996, in send
      self.connect()
    File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/transport/unixconn.py", line 26, in connect
      sock.connect(self.unix_socket)
  FileNotFoundError: [Errno 2] No such file or directory

Expected Behavior

localstack service should be ready and not timeout typically it is ready in <10s

How are you starting LocalStack?

Custom (please describe below)

Steps To Reproduce

Found this using localstack as a github action as described above but I suspect a simple docker run might trigger the bug

How are you starting localstack (e.g., bin/localstack command, arguments, or docker-compose.yml)

docker run localstack/localstack

Environment

- OS:
- LocalStack:
  LocalStack version: Not sure, using the latest tag
  LocalStack Docker image sha:sha256:e4aa48d5f60ec4047e06b0c8a6078384ccfff0c0301097723474a226dabbf19f
  LocalStack build date: ??
  LocalStack build git hash: ??

Anything else?

No response

[localstack-bot]

Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our Slack Community for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our contributing guidelines and our contributing guide.

[robo-corg]

Using the tag localstack/localstack:3.4.0 seems to fix things

[robo-corg]

Confirmed that docker run localstack/localstack seems to break in a similar way

[cloutierMat]

Thank you for reporting this issue.

There was a pretty big code restructure that just merged in today. While the image seems to be working and I can start the latest image locally, it is possible that you are running into some caching issues.

Can you try deleting your cache before re-triggering and let us know if that fixed the issue.

Also, docker run localstack/localstack is expected to show this error as you would need to mount the docker socket for docker to be available. See documentation

[GREsau]

The current latest causes the container's health checks to fail, although I think the container itself "works" correctly otherwise, i.e. it will response to AWS API requests as expected.

With this docker-compose.yml:

---
version: '3.8'

services:
  localstack:
    image: localstack/localstack:latest
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"
    environment:
      - DEBUG=1

...running docker compose up --wait fails because the container never becomes healthy. There's nothing useful in the container logs, but when I start a shell in the container and manually invoke the healthcheck command (taken from here) directly I get an error, presumably related to the big code restructure you mentioned:

# ./bin/localstack status services --format=json
Traceback (most recent call last):
  File "/opt/code/localstack/./bin/localstack", line 23, in <module>
    main()
  File "/opt/code/localstack/./bin/localstack", line 17, in main
    from localstack.cli import main
ModuleNotFoundError: No module named 'localstack'

[thrau]

I can confirm there's a very loud exception about the docker socket not being mounted, but localstack still starts up fine.

are you having issues actually using localstack?

I don't think this has anything to do with the code restructuring. my guess would be that it has to do with the recent docker sdk upgrade from 6 to 7 https://github.com/localstack/localstack/pull/10853/files, but TBC

[thrau]

Thanks @GREsau, great catch. should be fixed with #10894

[thrau]

The issue reported by @robo-corg is actually related to #9761 and is expected behavior according to this change. Again I believe it's a non-issue since it's not affecting localstack in any way, but it's misleading because it suggests the startup failed. I think we should be providing better output. Something like:

LocalStack Docker information: Failed to inspect the LocalStack docker container. This is likely
because the docker socket was not mounted into the container. Without access to the docker socket,
LocalStack will not function properly. Please consult the LocalStack documentation on how to correctly
start up LocalStack, or run localstack with DEBUG=1 to find out why this error occurred.

cc @dominikschubert

[robo-corg]

Issue I encountered is that github considers the container to have not started up successfully. I didn't try actually using it since the tests never ran.

[thrau]

thanks for the clarification. in which case i think the startup issue you really had is fixed with #10894. the stack traces you reported are definitely misleading and are addressed in #10896

[thrau]

i'm closing this issue since the underlying issue (the healthcheck) has been fixed.