motoserver/moto:latest (sha256:ee260b) unable to connect with localhost with boto3

[mroeschke]

Using this GHA workflow with this moto image: https://hub.docker.com/layers/motoserver/moto/latest/images/sha256-ee260b673d67a470572ee914be006d23a82cf85010b3efa0372f96dfecb1c88c?context=explore

name: Moto bug

on: push

permissions:
  contents: read

jobs:
  moto_test:
    runs-on: ubuntu-22.04
    env:
      AWS_ACCESS_KEY_ID: foobar_key
      AWS_SECRET_ACCESS_KEY: foobar_secret
    defaults:
      run:
        shell: bash -el {0}

    services:
      moto:
        image: motoserver/moto
        env:
          AWS_ACCESS_KEY_ID: foobar_key
          AWS_SECRET_ACCESS_KEY: foobar_secret
        ports:
          - 5000:5000
    steps:
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.10'

      - name: Install dependencies
        run: python -m pip install boto3==1.24.59

      - name: Test S3 ls
        run: python -c "import boto3; cli = boto3.client('s3', endpoint_url='http://localhost:5000'); cli.list_buckets()"

It is unable to connect to the service: https://github.com/mroeschke/pandas/actions/runs/4388687842/jobs/7685460881

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/urllib3/util/connection.py", line 95, in create_connection
    raise err
  File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/botocore/httpsession.py", line 455, in send
    urllib_response = conn.urlopen(
  File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/urllib3/util/retry.py", line 525, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/urllib3/packages/six.py", line 770, in reraise
    raise value
  File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/urllib3/connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/http/client.py", line 1282, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/botocore/awsrequest.py", line 94, in _send_request
    rval = super()._send_request(
  File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/http/client.py", line 1328, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/http/client.py", line 1277, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/botocore/awsrequest.py", line 123, in _send_output
    self.send(msg)
  File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/botocore/awsrequest.py", line 218, in send
    return super().send(str)
  File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/http/client.py", line 975, in send
    self.connect()
  File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPConnection object at 0x7f51acbeaec0>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/botocore/client.py", line 514, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/botocore/client.py", line 921, in _make_api_call
    http, parsed_response = self._make_request(
  File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/botocore/client.py", line 944, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/botocore/endpoint.py", line 119, in make_request
    return self._send_request(request_dict, operation_model)
  File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/botocore/endpoint.py", line 202, in _send_request
    while self._needs_retry(
  File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/botocore/endpoint.py", line 354, in _needs_retry
    responses = self._event_emitter.emit(
  File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/botocore/hooks.py", line 412, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/botocore/hooks.py", line 256, in emit
    return self._emit(event_name, kwargs)
  File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/botocore/hooks.py", line 239, in _emit
    response = handler(**kwargs)
  File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/botocore/retryhandler.py", line 207, in __call__
    if self._checker(**checker_kwargs):
  File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/botocore/retryhandler.py", line 284, in __call__
    should_retry = self._should_retry(
  File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/botocore/retryhandler.py", line 320, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/botocore/retryhandler.py", line 363, in __call__
    checker_response = checker(
  File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/botocore/retryhandler.py", line 247, in __call__
    return self._check_caught_exception(
  File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/botocore/retryhandler.py", line 416, in _check_caught_exception
    raise caught_exception
  File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/botocore/endpoint.py", line 281, in _do_get_response
    http_response = self._send(request)
  File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/botocore/endpoint.py", line 377, in _send
    return self.http_session.send(request)
  File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/botocore/httpsession.py", line 484, in send
    raise EndpointConnectionError(endpoint_url=request.url, error=e)
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://localhost:5000/"

It works with the 4.1.4 tag: https://github.com/mroeschke/pandas/actions/runs/4388697680/jobs/7685484714

[bblommers]

The Stop Containers step shows the root cause:

Traceback (most recent call last):
   File "/usr/local/lib/python3.7/site-packages/responses/__init__.py", line 31, in <module>
     import tomli as _toml
 ModuleNotFoundError: No module named 'tomli'

This was a bug in the responses-module, and has been fixed with their next release: getsentry/responses#630

So as far as I can tell, this will be fixed the moment we create a new release (that pulls in the fixed responses-module)

[mroeschke]

Ah sorry, I should have looked at the logs more closely. Great to know it will be fixed with the next release

[bblommers]

No worries - thanks for raising this anyway. Our CI builds were failing as well, now I understand why!

[beliaev-maksim]

sorry for that!

new version of responses is released with the fix

[bblommers]

@beliaev-maksim Well, in the end this is our fault for not pinning the dependencies that we test :)

@mroeschke We have released latest that should contain the fixed version of responses - can you verify it now works for you?

[mroeschke]

Yup, looks to work now so closing