Make threading.get_ident() return an opaque type

[pdox]

BPO	31622
Nosy	@tim-one, @ncoghlan, @pitrou, @benjaminp, @bitdancer, @serhiy-storchaka, @pdox

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-10-13.23:25:16.443>
created_at = <Date 2017-09-29.00:39:32.243>
labels = ['3.7', 'type-feature', 'library']
title = 'Make threading.get_ident() return an opaque type'
updated_at = <Date 2017-10-13.23:25:16.441>
user = 'https://github.com/pdox'

bugs.python.org fields:

activity = <Date 2017-10-13.23:25:16.441>
actor = 'pdox'
assignee = 'none'
closed = True
closed_date = <Date 2017-10-13.23:25:16.443>
closer = 'pdox'
components = ['Library (Lib)']
creation = <Date 2017-09-29.00:39:32.243>
creator = 'pdox'
dependencies = []
files = []
hgrepos = []
issue_num = 31622
keywords = []
message_count = 14.0
messages = ['303291', '303293', '303470', '303480', '303481', '303482', '303483', '303485', '303486', '303487', '303493', '303511', '303575', '304369']
nosy_count = 7.0
nosy_names = ['tim.peters', 'ncoghlan', 'pitrou', 'benjamin.peterson', 'r.david.murray', 'serhiy.storchaka', 'pdox']
pr_nums = []
priority = 'normal'
resolution = 'postponed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue31622'
versions = ['Python 3.7']

[pdox]

Currently, Python exposes "thread identifiers" as unsigned long values across multiple modules:

threading.get_ident() -> Returns an integer thread id
sys._current_frames() -> Dictionary keys are integer thread ids
signal.pthread_kill() -> Accepts an arbitrary integer thread id

In reality, OS level thread identifiers are opaque types. On a system with pthreads, the thread_id that Python provides is merely pthread_t casted to unsigned long. This works today, but is in violation of the standard, and could break on systems with exotic pthread_t.

There is also the ability to introduce undefined behavior, such as sending a signal to an invalid thread id:

>>> signal.pthread_kill(42, signal.SIGHUP)
Segmentation fault (core dumped)

Changing the thread identifiers to an opaque type (which also tracks the validity of the thread id, for blocking use of expired system thread ids) will solve these two issues.

[bitdancer]

Are you aware of PEP-539? (See bpo-25658 for discussion.) I haven't followed that closely, but I imagine this issue is at least related.

[pitrou]

On a system with pthreads, the thread_id that Python provides is merely pthread_t casted to unsigned long. This works today, but is in violation of the standard, and could break on systems with exotic pthread_t.

I don't think that follows. Even if pthread_t is not an integer, it does have a binary representation that can be trivially converted to a arbitrary-sized Python int in Python-facing APIs such as threading.get_ident().

(and similarly, of course, for converting in the other direction e.g. for pthread_kill())

There is also the ability to introduce undefined behavior, such as sending a signal to an invalid thread id:

I wouldn't worry much about pthread_kill(), a little-used feature.

[benjaminp]

You don't think having a function that takes an integer and dereferences it as memory is a bad idea?

[pitrou]

You don't think having a function that takes an integer and dereferences it as memory is a bad idea?

I'm not sure what you're talking about. What function is that?

[benjaminp]

On Sun, Oct 1, 2017, at 12:05, Antoine Pitrou wrote:

Antoine Pitrou <pitrou@free.fr> added the comment:

> You don't think having a function that takes an integer and dereferences it as memory is a bad idea?

I'm not sure what you're talking about. What function is that?

signal.pthread_kill and pdox's proposed time.pthread_getcpuclockid on
#3756.

[pitrou]

Le 01/10/2017 à 21:07, Benjamin Peterson a écrit :

signal.pthread_kill and pdox's proposed time.pthread_getcpuclockid on
#3756

Given how specialized (and of little actual use) those functions are, I
don't think it's much of a problem if they misbehave if you deliberately
pass an invalid thread id.

At least it sounds less of a problem than breaking all code that expects
a thread id to be an integer.

(btw, pthread_getcpuclockid returns ESRCH on Linux if called with an
invalid thread id)

[benjaminp]

On Sun, Oct 1, 2017, at 12:15, Antoine Pitrou wrote:

Antoine Pitrou <pitrou@free.fr> added the comment:

Le 01/10/2017 à 21:07, Benjamin Peterson a écrit :
>
> signal.pthread_kill and pdox's proposed time.pthread_getcpuclockid on
> #3756

Given how specialized (and of little actual use) those functions are, I
don't think it's much of a problem if they misbehave if you deliberately
pass an invalid thread id.

Okay, so you think this PR is reasonable?

At least it sounds less of a problem than breaking all code that expects
a thread id to be an integer.

I suppose, though you can't usefully use it as an integer. I imagine a
comparable and hashable opaque object would be compatible with most
code.

(btw, pthread_getcpuclockid returns ESRCH on Linux if called with an
invalid thread id)

More precisely, glibc has a heuristic check for thread id validity,
which dereferences whatever you pass in. It won't save you from
crashing.

[pitrou]

Le 01/10/2017 à 21:20, Benjamin Peterson a écrit :
> 
>>> signal.pthread_kill and pdox's proposed time.pthread_getcpuclockid on
>>> https://github.com/python/cpython/pull/3756
>>
>> Given how specialized (and of little actual use) those functions are, I
>> don't think it's much of a problem if they misbehave if you deliberately
>> pass an invalid thread id.
> 
> Okay, so you think this PR is reasonable?

On the principle yes. I am not interested enough in this feature to
look at the implementation.

Another possibility would be to have a new separate threading API
returning a "low-level opaque thread handle" that you could pass to
pthread_kill() and pthread_getcpuclockid(). threading.get_ident() could
still be used for logging purposes and others.

A related issue: what if you want to call pthread_kill() on a non-Python
thread?

I suppose, though you can't usefully use it as an integer. I imagine a
comparable and hashable opaque object would be compatible with most
code.

What happens for code that uses e.g. "%x" to format thread ids?

[benjaminp]

On Sun, Oct 1, 2017, at 12:25, Antoine Pitrou wrote:

Another possibility would be to have a new separate threading API
returning a "low-level opaque thread handle" that you could pass to
pthread_kill() and pthread_getcpuclockid(). threading.get_ident() could
still be used for logging purposes and others.

A related issue: what if you want to call pthread_kill() on a non-Python
thread?

C code that provides pthread_t to Python would have to wrap it.

> I suppose, though you can't usefully use it as an integer. I imagine a
> comparable and hashable opaque object would be compatible with most
> code.

What happens for code that uses e.g. "%x" to format thread ids?

It's okay for the opaque object to have a int() conversion. The
important part is that threading apis don't accept ints.

[serhiy-storchaka]

Currently, Python exposes "thread identifiers" as unsigned long values across multiple modules:

It happened not so long ago. In 3.6 and earlier "thread identifiers" are signed integers. See bpo-6532, it was massive change. This issue was deferred for long time due to the afraid of breaking the word. And it still was not tested with third-party code.

> What happens for code that uses e.g. "%x" to format thread ids?
It's okay for the opaque object to have a int() conversion.

What happens for C code that uses "%lu" to format thread ids?

[ncoghlan]

Note that it would be entirely possible to leave the Python level thread IDs alone, but change the way they were used internally to keep the thread ID distinct from the operating system level thread handle.

That would limit the adjustment to signal.pthread_kill() and other APIs that wanted to resolve the integer ID back to an OS level handle.