socket.getfqdn and socket.gethostbyname fail on MacOS

[ssbarnea]

BPO	35164
Nosy	@ronaldoussoren, @ned-deily, @ssbarnea, @ghickman

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2018-11-04.16:50:41.667>
labels = ['OS-mac', 'type-bug', '3.8', '3.9', '3.10', '3.7']
title = 'socket.getfqdn and socket.gethostbyname fail on MacOS'
updated_at = <Date 2022-03-10.09:27:18.347>
user = 'https://github.com/ssbarnea'

bugs.python.org fields:

activity = <Date 2022-03-10.09:27:18.347>
actor = 'ssbarnea'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['macOS']
creation = <Date 2018-11-04.16:50:41.667>
creator = 'ssbarnea'
dependencies = []
files = []
hgrepos = []
issue_num = 35164
keywords = []
message_count = 11.0
messages = ['329237', '329238', '329240', '329295', '329605', '331102', '331751', '331753', '359718', '414834', '414835']
nosy_count = 5.0
nosy_names = ['ronaldoussoren', 'ned.deily', 'ssbarnea', 'James Crowther', 'ghickman']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue35164'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5', 'Python 3.6', 'Python 3.7', 'Python 3.8', 'Python 3.9', 'Python 3.10']

[ssbarnea]

It seems that when the MacOS machine does not have a FQDN name manually configured using scutil, python will be fail to resolve domain names and timeout after ~30 seconds.

I mention that the DNS resolution works correctly on the machine and that the DHCP servers provides one ore more domains to be useles for resolution. Tools like nslookup or ping are not affected by this, only python (tried py27,34-36 and got same behavior).

Usually python user encounters errors like:

Traceback (most recent call last):
    ...
    socket.gethostbyname(socket.gethostname())
gaierror: [Errno 8] nodename nor servname provided, or not known

One these machines hostname cli command returns a valid FQDN name like foo.lan but scutil will not return one:

sudo scutil --get HostName
HostName: not set

One temporary workaround is to manually run:

sudo scutil --set HostName `hostname`

This will set the hostname and python sockets functions will start to work (after few minutes as some caching is involved).

Still, we cannot expect users to run this command themselves. Even worse, they would have to re-run this each time the roam to another network that may have a different set of domains.

See: https://stackoverflow.com/questions/52602107

[ned-deily]

Thanks for the report but I don't think the problem you are seeing is that simple. On my 10.14.1 system, I don't see that problem:

$ scutil --get HostName
HostName: not set
$ /usr/local/bin/python3.7
Python 3.7.1 (v3.7.1:260ec2c36a, Oct 20 2018, 03:13:28)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> socket.gethostname()
'harj.local'
>>> socket.gethostbyname(socket.gethostname())
'10.0.1.8'
>>>

So there's likely some other configuration issue here.

[ned-deily]

See also bpo-29705.

[ronaldoussoren]

I agree with Ned. The code fragment works for me as well on 10.14.1 and I've used simular code in the past on different versions of macOS as well.

@ssbarnea:

• What is the hostname as shown in the "Sharing" preference pane? Does the name include special characters (anything but ASCII letters and digits)?
• What version of macOS do you use?
• How did you install Python? (Python.org installer, homebrew, self built, ...)

[JamesCrowther]

Hi guys, I am the OP for the stack overflow issue question that sparked this bug report.

In response to Ned's question about configuration of the system:

MacOS 10.14.1
Installed x-code and command line tools
python versions used, varies, 3.5.3, 3.6 and 3.7.0 (sorry can't be more specific about 3.6 since I've used a lot. These python configurations have been both a bundled python env in blender, and also MacOS's native python 2.7.

So far they've all had the same issue, that being;

socket.gethostname() 

returns a value that seems to be set by the router/switch if

scutil --set hostname somehostname

has not been run. What Sorin mentioned on the stack overflow question that he didn't here (and I feel is important) is that ping and nslookup work with the expected hostname, i.e. the one the user can actually see by going to the system preferences and viewing it under the sharing section. Here it is called "Computer name", analogous to the same used on windows (which I might add has no such issues with hostnames being set to something other than the computer name unless a certain command is run).

So the real issue for us, is how come ping and nslookup will work fine with the macOS "computer name" but the socket module uses something else, seemingly the hostname and computer name are not or loosely connected on macOS perhaps.

Ideally what we need here is a way to ensure that the user can input the computer name of a target machine on another system and that system resolve the ip address for the target machine. We can't currently use socket.gethostname for this on mac since it seems to require that the scutil --set hostname command has been run. Otherwise the hostname that is returned doesn't work with socket.gethostbyname(hostname).

Hope this helps, happy to clarify.

J.

[ronaldoussoren]

This issue does not have enough information to determine what is wrong. It is however fairly clear that it is not related to "scutil --get HostName" mentioning that there is no hostname set as that is the normal state of systems, and I have never run into the problem mentioned here (and have used macOS exclusively from one of the earliest public beta's of macOS 10.0).

My gut feeling is that this is some issue with your system or environment.

Some more questions:

• Is this is a consistent problem, or only on some networks?
• Do you have any custom settings in the Networking preference pane related to name resolution (extra search domains, DNS servers, ...)?
• Do the domains returned by the DHCP server actually exist?
• Likewise for the DNS servers
• Does the result of socket.gethostname() look sane? In particular, does this include a domain name?
• The original message mentions: "and that the DHCP servers provides one ore more domains to be useles for resolution.". What is meant by that? AFAIK the DHCP server can only return 1 domain name, not a full DNS search list (or at least not one that is used by most major platforms).

[JamesCrowther]

Hi Ronald, I've replied to your comments below

My gut feeling is that this is some issue with your system or environment.

I Doubt it, happens with another MacBook air which is literally out of the box, no mods.

Some more questions:

• Is this is a consistent problem, or only on some networks?
  All networks
• Do you have any custom settings in the Networking preference pane related to name resolution (extra search domains, DNS servers, ...)?
  No
• Do the domains returned by the DHCP server actually exist?
  Not sure
• Likewise for the DNS servers
  likewise not sure
• Does the result of socket.gethostname() look sane? In particular, does this include a domain name?
  it is given a name that includes the ip address and the name of the router.
• The original message mentions: "and that the DHCP servers provides one ore more domains to be useles for resolution.". What is meant by that? AFAIK the DHCP server can only return 1 domain name, not a full DNS search list (or at least not one that is used by most major platforms).
  Not sure what was meant by this comment.

[ronaldoussoren]

>My gut feeling is that this is some issue with your system or environment.

I Doubt it, happens with another MacBook air which is literally out of the box, no mods.

But on the other hand most other users (including myself and Ned) don't have this problem.

>- Does the result of socket.gethostname() look sane? In particular, does this include a domain name?
it is given a name that includes the ip address and the name of the router.

Does resolving that name work? If it doesn't that points to a problem with your environment.

What's the result of resolving the IP address of your system? That name gets used by the system when available, and that could explain the issue your seeing.

[ghickman]

I came across this today with the same timeout behaviour on macOS 10.14.6.

After some digging I tracked it down to the Search Domains setting of my local network, this was set to "local", removing that immediately fixed the issue.

[ssbarnea]

I am able to reproduce this bug with Python 3.10 on MacOS 12.2 too, and I happen to have some extra insights about what happens, see tox-dev/tox#2375

Basically the culprit is a reverse-dns query (PTR) that takes 30s to return, causing that very long delay. It appears that mDNSResponder is able to cache the result for some time but not very long, so it will happen again.

In my particular case that PTR query is an .ip6.arpa one and there is no ipv4 query made at all.

I was not able to identify any similar delay while calling scutil --get HostName or hostname, but socket.getfqdn() does reproduce it reliably.

Interestingly, it seems that the delay is at least sometimes 15s, maybe the delay of 30s I observed on tox was caused by two calls.

Anyway, I am more than interested in finding a solution for this issue. Such a delays can be a real dealbreaker even for cli tools.