Add 128-bit integer support to struct

[fil]

BPO	19904
Nosy	@tim-one, @gpshead, @mdickinson, @pitrou, @alex, @meadori, @nanjekyejoannah

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2013-12-06.02:04:23.032>
labels = ['ctypes', 'type-feature']
title = 'Add 128-bit integer support to struct'
updated_at = <Date 2019-05-27.19:23:17.757>
user = 'https://bugs.python.org/fil'

bugs.python.org fields:

activity = <Date 2019-05-27.19:23:17.757>
actor = 'nanjekyejoannah'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['ctypes']
creation = <Date 2013-12-06.02:04:23.032>
creator = 'fil'
dependencies = []
files = []
hgrepos = []
issue_num = 19904
keywords = []
message_count = 15.0
messages = ['205344', '205357', '205399', '205400', '205403', '205431', '205447', '205574', '205626', '205627', '205628', '205629', '205634', '205763', '205765']
nosy_count = 9.0
nosy_names = ['tim.peters', 'gregory.p.smith', 'mark.dickinson', 'pitrou', 'alex', 'meador.inge', 'francismb', 'fil', 'nanjekyejoannah']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue19904'
versions = ['Python 3.6']

[fil]

I've been looking at adding 128-bit support to the struct module. Currently only named integer types are supported, which vary in implementation. These include:

short
int
long
long long

Depending on the platform, none may translate to 128-bit integer (the case with all platforms today?).

One approach would be to make a new type that relates specifically to 128-bit integer, side-stepping the naming approaches to integer in C.

The other, would be to make new types for all integer sizes that relate to specific sizes, instead of relying on C namings. Much bigger implications?

I propose creating new types:

"o": __int128_t
"O": __uint128_t
"t": __int256_t (why not?)
"T": __uint256_t
"v": __int512_t (what, too far?)
"V": __int512_t

What implications are there here in killing the connection between a C named type and a specific size?

[mdickinson]

Do you have a particular application / use-case in mind? Would int.from_bytes and int.to_bytes fill the need?

[fil]

The use case is interacting with C structures that have 128/256/512 bit integers in them. These require correct memory alignment, and can't reliably be hacked with multiple int64's.

These size ints come from the fact that CPU's now have registers of these sizes, and can't be serviced with int.from/to_bytes for performance reasons. The same reason int64 being supported with this approach would be very inconvenient and terrible for performance since they are an intrinsic type in modern hardware, not a software construction.

Two key use cases I can think of are:

• any form of integer SIMD operation (vectors)
• hosting and maintaining hash values which are routinely 128-bit and greater

[fil]

I noticed that python 2.7 has been removed - why would this not be appropriate for this version?

Given the current level of use of this version, I see it's inclusion as very important - without having to be relegated to 3.x only.

[tim-one]

Fil, here's the release schedule for Python 2.8:

http://www.python.org/dev/peps/pep-0404/

In short, 2.8 will never be released (well, never by us), and only bugfixes can be applied to the 2.7 line. That's why 2.7 was removed. Regardless of its merits, this is a new feature request, not a bugfix.

A future Python 3 release is the only possibility for new features.

[pitrou]

any form of integer SIMD operation (vectors)

Wouldn't these be appropriately represented by a tuple of integers (or floats)? For example, a SIMD vector of four 32-bit integers could be represented as four Python ints. Or would that be "terrible for performance"?

Note that the struct module may include support for int128_t, but it certainly won't have native support for every SIMD vector format under the sun (if they have different alignment requirements).

hosting and maintaining hash values which are routinely 128-bit and greater

That sounds like a job for a bytes object (as returned by e.g. hashlib.sha1(...).digest()).

[skrah]

If performance is the reason for the feature: My impression is that
the goal of the struct module is not necessarily top performance.

E.g. in memoryview the custom unpackers for comparisons are 30-60 times
faster than using the struct module.

[francismb]

If performance is the reason for the feature: My impression is that
the goal of the struct module is not necessarily top performance.

I'm not sure if it applies but on bpo-19905 (message 205345 [1])
is said its a dependency for that issue

----
[1] http://bugs.python.org/issue19905#msg205345

[fil]

Stefan, performance is not the principle motivator here: the intention is that it is just sensible to support this integer type, just like supporting int64 since it is an intrinsic type supported by CPU's.

Of course any integer length could be handled by memoryview unpacker, but this would not really make sense for int64. My view is that any intrinsic type (ie. register type) is the key point at which it should be supported by struct, and not a custom unpacker. 128/256/512 bit integers are in this category..

Principally this is so that it can be used by ctypes (see bpo-19905).

[pitrou]

Of course any integer length could be handled by memoryview unpacker,
but this would not really make sense for int64. My view is that any
intrinsic type (ie. register type) is the key point at which it should
be supported by struct, and not a custom unpacker. 128/256/512 bit
integers are in this category..

I don't think "register types" matter here. The struct module provides
interoperability with low-level *data types* (in C and other common
languages), not CPU *registers*.

So IMHO the question is whether there is an use case for 128-bit
integers; *not* for arrays of four 32-bit integers packed in a 128-bit
register.

[fil]

Antoine,

No, the SIMD vector should be treated as a int128 (for eg.), not as 4 python ints. It is the unpacking process that unpacks into 4 python ints. The two shouldn't be confused - you definitely want to be able to treat the vector in both states (packed and unpacked).

I'm not saying that Python should necessarily make use of SIMD instructions (don't think performance is critical there in py), but that it should at least be access to a raw 128/256/512 bit integer within a C structure. Part of this is interacting with SIMD'able structures - at least in my use cases :)

[pitrou]

The two shouldn't be confused - you definitely want to be able to
treat the vector in both states (packed and unpacked).

Why do you need the packed state as a 128-bit int, rather than as a
bytes object? You are not doing arithmetic on it, AFAIU.

[fil]

Antoine,

I don't think "register types" matter here. The struct module provides
interoperability with low-level *data types* (in C and other common
languages), not CPU *registers*.

OK, see where you're coming from. I guess my thinking was such that struct (and ctypes) support is required for data types that are common within C structures. The fact that 128-bit ints are now appearing within these structures is due to the fact that CPU support has dramatically improved, and are supported natively in register types. So it was kind of an indirect connection.

So the question I guess is whether you think these types are "common enough" in C structs, which for me they certainly are for my data processing applications.. (pretty please :)

So IMHO the question is whether there is an use case for 128-bit
integers; *not* for arrays of four 32-bit integers packed in a 128-bit
register.

Agree completely - just stick to thinking about atomic 128-bit integers. A pack/unpack function should do the conversion, and have nothing to do with this struct support.