Fix socket.type on OSes with SOCK_NONBLOCK

[1st1]

BPO	32331
Nosy	@pitrou, @vstinner, @tiran, @njsmith, @asvetlov, @methane, @1st1
PRs	bpo-32331: Fix socket.type on Linux; expose SOCK_TYPE_MASK. #4877

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-12-19.01:08:26.719>
created_at = <Date 2017-12-15.03:42:49.184>
labels = ['3.7', 'type-feature', 'library']
title = 'Fix socket.type on OSes with SOCK_NONBLOCK'
updated_at = <Date 2017-12-19.01:14:14.988>
user = 'https://github.com/1st1'

bugs.python.org fields:

activity = <Date 2017-12-19.01:14:14.988>
actor = 'yselivanov'
assignee = 'none'
closed = True
closed_date = <Date 2017-12-19.01:08:26.719>
closer = 'yselivanov'
components = ['Library (Lib)']
creation = <Date 2017-12-15.03:42:49.184>
creator = 'yselivanov'
dependencies = []
files = []
hgrepos = []
issue_num = 32331
keywords = ['patch']
message_count = 23.0
messages = ['308363', '308370', '308371', '308376', '308409', '308410', '308411', '308432', '308437', '308442', '308444', '308446', '308450', '308451', '308455', '308589', '308590', '308591', '308592', '308603', '308604', '308605', '308606']
nosy_count = 7.0
nosy_names = ['pitrou', 'vstinner', 'christian.heimes', 'njs', 'asvetlov', 'methane', 'yselivanov']
pr_nums = ['4877']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue32331'
versions = ['Python 3.7']

[1st1]

On Linux, socket type is both a socket type and a bit mask (of SOCK_CLOEXEC and SOCK_NONBLOCK). Therefore, anyone who write code like 'if sock.type == SOCK_STREAM' writes non-portable code, that occasionally breaks on Linux.

This caused some hard to spot bugs in asyncio already: https://bugs.python.org/issue27456 -- this one was discovered only 1 year after the actual change.

On Linux, in 'include/linux/net.h' there's a SOCK_TYPE_MASK macro defined to 0xF. The idea is that on Linux you should mask socket type before comparing it to SOCK_STREAM or other socket types.

Using socket.SOCK_NONBLOCK on Python was always error-prone even if one targets only Linux. For instance, socket.type won't be updated to include SOCK_NONBLOCK if the socket was updated via ioctl call directly.

Therefore, I propose to fix socket.type getter to always (on Linux) apply that mask in Python.

[methane]

Should we backport it to 3.6?

[methane]

Document of socket.type says:

The socket type.

https://docs.python.org/3/library/socket.html#socket.socket.type

OK, so, are SOCK_CLOEXEC and SOCK_NONBLOCK socket type?

These two constants, if defined, can be combined with the socket types and allow you to set some flags atomically...

https://docs.python.org/3/library/socket.html#socket.SOCK_CLOEXEC

So they are not socket type. They are constants which can be used for
setting some flags.

Regarding to this document, I think this can be bugfix rather than
behavior change.

[tiran]

Python's handling of socket has multiple issues, see bpo-28134 for more fun. At the moment the values of type, family and protocol are unreliable at best and dangerously wrong at worst. The attributes are only correct if and only if the socket has not been created from a file descriptor and the socket hasn't been manipulated with ioctl or setsockopt().

Now to the issue at hand. I'm -1 on applying a mask to type. You can easily work around your problem by replacing "sock.type == SOCK_STREAM" with "sock.type & SOCK_STREAM == SOCK_STREAM".

[1st1]

You can easily work around your problem by replacing "sock.type == SOCK_STREAM" with "sock.type & SOCK_STREAM == SOCK_STREAM".

Heh :) No, that would be a buggy code. Try this on your Linux box:

(socket.SOCK_SEQPACKET & socket.SOCK_STREAM) == socket.SOCK_STREAM

The _only_ way of checking socket type reliably is to explicitly reset SOCK_NONBLOCK and SOCK_CLOEXEC or to apply 0xF mask on Linux, and don't do that on other platforms.

That's why I think it's important to fix this one way or another.

[1st1]

On GH (#4877), Antoine wrote:

I agree with Victor:

it can't go into 3.6
making the change in 3.7 is contentious
Can there be an other way to solve the issue? Can we for example keep socket.type as it is and add a utility function or a property reporting the "masked" type?

I agree. Here's what I propose:

1. We don't touch socket.type. We don't know what kind of code exists out there and in what ways it will break. The ship has sailed to fix it.

2. We add socket.SOCK_TYPE_MASK (on Linux).

3. We fix socket.__repr__ to correctly render type. Currently:

    >>> s = socket.socket(socket.AF_INET, socket.SOCK_STREAM | socket.SOCK_NONBLOCK)
    >>> s
    <socket.socket fd=3, family=AddressFamily.AF_INET, type=2049, proto=0, laddr=('0.0.0.0', 0)>

What I want to see:

<socket.socket fd=3, family=AddressFamily.AF_INET, type=SOCK_STREAM | SOCK_NONBLOCK, proto=0, laddr=('0.0.0.0', 0)>

4. We add socket.true_type which will apply SOCK_TYPE_MASK and return a socket type that's safe to compare to socket.SOCK_* constants.

Thoughts?

[tiran]

Should socket.true_type use getsockopt(fd, ...) to get the actual type?

While we are talking about socket types, please have a look at bpo-28134 .

[1st1]

Update:

I've rewritten the PR from scratch.

1. SOCK_TYPE_MASK isn't exported on Linux. Therefore we will not export socket.SOCK_TYPE_MASK too.

2. I've added the new socket.truetype property.

3. When a socket is created without an FD, e.g. "socket.socket(family, type, proto)", we trust that the type of the socket is correct. It might contain extra flags on Linux, so we filter them out.

4. When a socket is created from an FD, e.g. with "socket.fromfd" or with "socket.socket(family, type, proto, FD)" we don't trust the type. So socket.truetype will call getsockopt(SOL_SOCKET, SO_TYPE) on the FD to ensure we have the correct socket type at hand.

5. Since Linux doesn't export SOCK_TYPE_MASK I decided to hardcode it in socketmodule.c. My reasoning:

a/ It's highly unlikely that Linux will implement 5 more new socket types anytime soon, so 0xF should last for a *very* long time.

b/ It's more likely than "a/" that they add a new flag like SOCK_CLOEXEC, in which case the "type & ~(SOCK_NONBLOCK | SOCK_CLOEXEC)" approach will fail.

So given a/ and b/ I think it's safe to just use 0xF mask on Linux.

[1st1]

Another name option: socket.realtype

[njsmith]

I think the earliest open bug about this is bpo-21327.

[1st1]

I've been thinking a lot about this problem, and I'm really tempted to fix sock.type property:

1. The problem first appeared in Python 3.2.

2. Python 2.7 doesn't have this problem at all, and doesn't even export socket.SOCK_NONBLOCK. If we fix this in 3.7 it *will* actually help some poor souls with porting their network applications.

3. People use Python when they want a high-level portable language. This annoying Linux quirk makes it super hard to write correct socket code.

4. I can't actually come up with any decent Linux-only example of using this quirk of socket.type. Why would one check if sock.type has SOCK_NONBLOCK? And even if they check it, one call to sock.settimeout() will make sock.type information outdated and simply wrong.

Let's just fix sock.type?