ConnectError: [Errno -2] Name or service not known when URL contains IPv6 #1311

filwie · 2020-09-23T09:45:17Z

Describe the bug

When trying to connect to HTTP or HTTP/2 server using URL containing IPv6 address
httpx.ConnectError: [Errno -2] Name or service not known exception is raised.

To reproduce

Run some http server.

In my case it is a Docker container with IPv6 enabled, so for example:

docker run --name=serv --rm hashicorp/http-echo -text=hello

To get IPs of the container:

docker inspect serv --format '{{ .NetworkSettings.GlobalIPv6Address  }}    {{ .NetworkSettings.IPAddress }}'

Run below code (after adjusting URLs to match IPsextracted from docker inspect

import httpx
import requests

url = 'http://172.17.0.2:5678'
url6 = 'http://[2001:db8:1::242:ac11:2]:5678'

print(requests.get(url))
print(requests.get(url6))

print(httpx.get(url))
print(httpx.get(url6))

Expected behavior

Response is returned

<Response [200 OK]>

Actual behavior

httpx.ConnectError: [Errno -2] Name or service not known exception is raised

Debugging material

Output from above snippet:

<Response [200]>
<Response [200]>
<Response [200 OK]>
Traceback (most recent call last):
  File "/home/user/repos/lb-test/.venv/lib/python3.7/site-packages/httpx/_exceptions.py", line 339, in map_exceptions
    yield
  File "/home/user/repos/lb-test/.venv/lib/python3.7/site-packages/httpx/_client.py", line 858, in _send_single_request
    ext={"timeout": timeout.as_dict()},
  File "/home/user/repos/lb-test/.venv/lib/python3.7/site-packages/httpcore/_sync/connection_pool.py", line 195, in request
    method, url, headers=headers, stream=stream, ext=ext
  File "/home/user/repos/lb-test/.venv/lib/python3.7/site-packages/httpcore/_sync/connection.py", line 87, in request
    self.socket = self._open_socket(timeout)
  File "/home/user/repos/lb-test/.venv/lib/python3.7/site-packages/httpcore/_sync/connection.py", line 113, in _open_socket
    local_address=self.local_address,
  File "/home/user/repos/lb-test/.venv/lib/python3.7/site-packages/httpcore/_backends/sync.py", line 144, in open_tcp_stream
    return SyncSocketStream(sock=sock)
  File "/home/user/.local/share/pyenv/versions/3.7.6/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/user/repos/lb-test/.venv/lib/python3.7/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
    raise to_exc(exc) from None
httpcore.ConnectError: [Errno -2] Name or service not known

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "kek.py", line 11, in <module>
    print(httpx.get(url6))
  File "/home/user/repos/lb-test/.venv/lib/python3.7/site-packages/httpx/_api.py", line 193, in get
    trust_env=trust_env,
  File "/home/user/repos/lb-test/.venv/lib/python3.7/site-packages/httpx/_api.py", line 104, in request
    allow_redirects=allow_redirects,
  File "/home/user/repos/lb-test/.venv/lib/python3.7/site-packages/httpx/_client.py", line 722, in request
    request, auth=auth, allow_redirects=allow_redirects, timeout=timeout
  File "/home/user/repos/lb-test/.venv/lib/python3.7/site-packages/httpx/_client.py", line 758, in send
    history=[],
  File "/home/user/repos/lb-test/.venv/lib/python3.7/site-packages/httpx/_client.py", line 795, in _send_handling_auth
    history=history,
  File "/home/user/repos/lb-test/.venv/lib/python3.7/site-packages/httpx/_client.py", line 823, in _send_handling_redirects
    response = self._send_single_request(request, timeout)
  File "/home/user/repos/lb-test/.venv/lib/python3.7/site-packages/httpx/_client.py", line 858, in _send_single_request
    ext={"timeout": timeout.as_dict()},
  File "/home/user/.local/share/pyenv/versions/3.7.6/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/user/repos/lb-test/.venv/lib/python3.7/site-packages/httpx/_exceptions.py", line 356, in map_exceptions
    raise mapped_exc(message, **kwargs) from exc  # type: ignore
httpx.ConnectError: [Errno -2] Name or service not known

Environment

OS: Linux
Python version: 3.7.2, 3.8.5
HTTPX version: 0.15.0
Async environment: not used
HTTP proxy: no
Custom certificates: no

Additional context

The text was updated successfully, but these errors were encountered:

filwie · 2020-09-28T10:26:10Z

To my eye it looks like the problem is with the brackets in the URL - they are never stripped URL.host returns [2001:db8:1::242:ac11:2] which is then passed to httpcore that does not now what to do with it.

Quick fix that I am using at the moment - modification of _models.py (fork commit):

    @property
    def host(self) -> str:
        """
        The URL host as a string.
        Always normlized to lowercase, IDNA encoded
        with square brackets stripped from IPv6 host.

        Examples:

        url = httpx.URL("http://www.EXAMPLE.org")
        assert url.host == "www.example.org"

        url = httpx.URL("http://中国.icom.museum")
        assert url.host == "xn--fiqs8s.icom.museum"

        url = httpx.URL("http://[2001:db8:1::242:ac11:2]:5000")
        assert url.host = "2001:db8:1::242:ac11:2"
        """
        return (self._uri_reference.host or "").lstrip('[').rstrip(']')

I do not know however if it is the best place to strip those brackets - let me know what you think.
I am also not sure if the brackets should be stripped for netloc if port is None.

tomchristie · 2020-10-06T12:08:32Z

Interesting, yup.

I guess our options here are to either strip at the httpx level, or to strip at the httpcore level.

Of those I'm minded to strip them at httpx, because url.host semantically looks like it ought to handle the escaping really.

So, what do we think of the following?...

Strip the [, ] from literal IP's at the httpx level.
.host should return the stripped version.
.netloc will need to include them. (Eg see https://bugzilla.mozilla.org/show_bug.cgi?id=45891)
Possible that .copy_with(host=...) should automatically add [, ] for hosts that match an IPv6 regex? (Or more simply just any host name that includes a ":" character.)

bdieng-sudo · 2021-04-13T09:49:44Z

I have the same issue and received the following message:
shaman-install configuration.yaml
2021-04-13 08:31:59.315 | DEBUG | bb_wrapper.tunable_component.install_component:install_component:36 - Sending component data {'components': {'shaman_pbo_msr_profiling': {'plugin': '', 'header': 'shaman_pbo_profiling', 'command': "rsh pboadm@hana02 'unpbo_shaman.sh'", 'ld_preload': None, 'parameters': {'nbthreads': {'type': 'int', 'default': 8, 'optional': False, 'env_var': True, 'description': 'Number of threads setting {1..10}', 'cmd_var': None, 'cli_var': None, 'flag': None, 'suffix': None}, 'msr_1a4': {'type': 'int', 'default': 0, 'optional': False, 'env_var': True, 'description': 'MSR 0x1a4 setting {0..15}', 'cmd_var': None, 'cli_var': None, 'flag': None, 'suffix': None}, 'msr_64': {'type': 'int', 'default': 0, 'optional': False, 'env_var': True, 'description': 'MSR 0x64 setting {0..15}', 'cmd_var': None, 'cli_var': None, 'flag': None, 'suffix': None}}, 'custom_target': 'parse_execution_time.parse_slurm_times'}}}to endpointhttp: // api: 5000components
Traceback (most recent call last):
File "/root/.cache/pypoetry/virtualenvs/shaman-project-jdZZMt2c-py3.6/lib/python3.6/site-packages/httpx/_exceptions.py", line 326, in map_exceptions
yield
File "/root/.cache/pypoetry/virtualenvs/shaman-project-jdZZMt2c-py3.6/lib/python3.6/site-packages/httpx/_client.py", line 861, in _send_single_request
ext={"timeout": timeout.as_dict()},
File "/root/.cache/pypoetry/virtualenvs/shaman-project-jdZZMt2c-py3.6/lib/python3.6/site-packages/httpx/_transports/default.py", line 102, in request
return self._pool.request(method, url, headers=headers, stream=stream, ext=ext)
File "/root/.cache/pypoetry/virtualenvs/shaman-project-jdZZMt2c-py3.6/lib/python3.6/site-packages/httpcore/_sync/connection_pool.py", line 219, in request
method, url, headers=headers, stream=stream, ext=ext
File "/root/.cache/pypoetry/virtualenvs/shaman-project-jdZZMt2c-py3.6/lib/python3.6/site-packages/httpcore/_sync/connection.py", line 93, in request
self.socket = self._open_socket(timeout)
File "/root/.cache/pypoetry/virtualenvs/shaman-project-jdZZMt2c-py3.6/lib/python3.6/site-packages/httpcore/_sync/connection.py", line 124, in _open_socket
local_address=self.local_address,
File "/root/.cache/pypoetry/virtualenvs/shaman-project-jdZZMt2c-py3.6/lib/python3.6/site-packages/httpcore/_backends/sync.py", line 143, in open_tcp_stream
return SyncSocketStream(sock=sock)
File "/usr/lib64/python3.6/contextlib.py", line 99, in exit
self.gen.throw(type, value, traceback)
File "/root/.cache/pypoetry/virtualenvs/shaman-project-jdZZMt2c-py3.6/lib/python3.6/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
raise to_exc(exc) from None
httpcore.ConnectError: [Errno -2] Name or service not known

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/root/.cache/pypoetry/virtualenvs/shaman-project-jdZZMt2c-py3.6/bin/shaman-install", line 5, in
cli()
File "/root/.cache/pypoetry/virtualenvs/shaman-project-jdZZMt2c-py3.6/lib/python3.6/site-packages/typer/main.py", line 214, in call
return get_command(self)(*args, **kwargs)
File "/root/.cache/pypoetry/virtualenvs/shaman-project-jdZZMt2c-py3.6/lib/python3.6/site-packages/click/core.py", line 829, in call
return self.main(*args, **kwargs)
File "/root/.cache/pypoetry/virtualenvs/shaman-project-jdZZMt2c-py3.6/lib/python3.6/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/root/.cache/pypoetry/virtualenvs/shaman-project-jdZZMt2c-py3.6/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/root/.cache/pypoetry/virtualenvs/shaman-project-jdZZMt2c-py3.6/lib/python3.6/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/root/.cache/pypoetry/virtualenvs/shaman-project-jdZZMt2c-py3.6/lib/python3.6/site-packages/typer/main.py", line 497, in wrapper
return callback(**use_params) # type: ignore
File "/root/shaman-next/shaman_project/bb_wrapper/tunable_component/install_component.py", line 42, in install_component
api_settings.component_endpoint, json=component.dict())
File "/root/.cache/pypoetry/virtualenvs/shaman-project-jdZZMt2c-py3.6/lib/python3.6/site-packages/httpx/_client.py", line 999, in post
timeout=timeout,
File "/root/.cache/pypoetry/virtualenvs/shaman-project-jdZZMt2c-py3.6/lib/python3.6/site-packages/httpx/_client.py", line 729, in request
request, auth=auth, allow_redirects=allow_redirects, timeout=timeout
File "/root/.cache/pypoetry/virtualenvs/shaman-project-jdZZMt2c-py3.6/lib/python3.6/site-packages/httpx/_client.py", line 767, in send
history=[],
File "/root/.cache/pypoetry/virtualenvs/shaman-project-jdZZMt2c-py3.6/lib/python3.6/site-packages/httpx/_client.py", line 804, in _send_handling_auth
history=history,
File "/root/.cache/pypoetry/virtualenvs/shaman-project-jdZZMt2c-py3.6/lib/python3.6/site-packages/httpx/_client.py", line 832, in _send_handling_redirects
response = self._send_single_request(request, timeout)
File "/root/.cache/pypoetry/virtualenvs/shaman-project-jdZZMt2c-py3.6/lib/python3.6/site-packages/httpx/_client.py", line 861, in _send_single_request
ext={"timeout": timeout.as_dict()},
File "/usr/lib64/python3.6/contextlib.py", line 99, in exit
self.gen.throw(type, value, traceback)
File "/root/.cache/pypoetry/virtualenvs/shaman-project-jdZZMt2c-py3.6/lib/python3.6/site-packages/httpx/_exceptions.py", line 343, in map_exceptions
raise mapped_exc(message, **kwargs) from exc # type: ignore
httpx.ConnectError: [Errno -2] Name or service not known
hana03:~/shaman-next #

tomchristie added the bug Something isn't working label Oct 6, 2020

cdeler mentioned this issue Oct 7, 2020

Correctly handle ipv6 addresses as a part of URL #1349

Merged

cdeler closed this as completed in #1349 Oct 8, 2020

the-rich-piana mentioned this issue Dec 4, 2023

httpx.ConnectError: [Errno -2] Name or service not known using Docker Supabase Image #2977

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ConnectError: [Errno -2] Name or service not known when URL contains IPv6 #1311

ConnectError: [Errno -2] Name or service not known when URL contains IPv6 #1311

filwie commented Sep 23, 2020 •

edited

Loading

filwie commented Sep 28, 2020

tomchristie commented Oct 6, 2020 •

edited

Loading

bdieng-sudo commented Apr 13, 2021

ConnectError: [Errno -2] Name or service not known when URL contains IPv6 #1311

ConnectError: [Errno -2] Name or service not known when URL contains IPv6 #1311

Comments

filwie commented Sep 23, 2020 • edited Loading

Describe the bug

To reproduce

Expected behavior

Actual behavior

Debugging material

Environment

Additional context

filwie commented Sep 28, 2020

tomchristie commented Oct 6, 2020 • edited Loading

bdieng-sudo commented Apr 13, 2021

filwie commented Sep 23, 2020 •

edited

Loading

tomchristie commented Oct 6, 2020 •

edited

Loading