introduce bytes.hex method (also for bytearray and memoryview) #54160
Comments
|
Following up on these discussions: I'm submitting a patch to add bytes.hex method in accordance to PEP-358. There are additional things to discuss, for example:
|
wiggin15
mannequin
added
interpreter-core
|
fixed to Py_UNICODE |
|
See also: bpo-11756 |
|
Also bpo-3532 |
|
Hi, is there any chance to get this merged? This ticket has been open for almost 3 years... |
|
There are several ways to do this: base64.b16encode, binascii.a2b_hex, hex(int.from_bytes(...)), etc. Why you need yet one? |
|
You can follow the discussion I linked in the ticket description for an answer: |
|
I like to see the feature in 3.4, too. |
|
If it's the reverse of fromhex(), perhaps we should call it tohex()? |
|
Funny thing. I was searching for "tohex" when I found this ticket. |
|
Blasphemous question: why not give bytes a __hex__ method? Then you could use hex() to convert them :) The patch is outdated; it should not use PyUnicode_AS_UNICODE, but PyUnicode_New(..., 127) and then PyUnicode_1BYTE_DATA to get the char array. |
|
would be good if we can specify a optional flag to get all cap hex. currently, I have to do hexlify( some_bytes ).decode( 'UTF-8' ).upper(). would be good to be able to do some_bytes.hex( upper=1 ) |
|
New features cannot be added to Python 2 anymore, only to the current development version which is now Python 3.5. If new methods are added to bytes, they should be added to bytearray too. Maybe we should also consider add them to memoryview? memoryview has already a .bytes() method and can be casted to type "B" (array of integers in range 0..255). The float type has .hex() and .fromhex() methods. We should kepe these names to stay consistent. Which kind of output do you prefer? "0xHH 0xHH ...", "HH HH HH ..." or "HHHHHH..."? Do you want to add parameters to choose the format? Current binascii format: >>> binascii.hexlify('abc')
'616263' |
|
To answer Serhiy, the goal is to have a bytes method that represents bytes as bytes rather than as a mixture of bytes and encoded ascii characters. This would aid people who work with bytes that are not encoded ascii and that do not embed encoded ascii. It should not be necessary to import anything. >>> hex(int.from_bytes(b'abc', 'big'))
'0x616263'
is a bit baroque and produces a hex representation of an int, not of multiple bytes.
I think following the float precedent is a good idea.
>>> float.fromhex(1.5.hex())
1.5
>>> float.fromhex('0x1.8000000000000p+0').hex()
'0x1.8000000000000p+0'The output of bytes.hex should be one that is accepted by bytes.fromhex, which is to say, hex 'digit' pairs. Spaces are optionally allowed between pairs. I would add a 'spaces' parameter, defaulting to False. to output spaces when set to true. (Or possible reverse the default -- what do potential users think?) A possible altermative for the parameter could be form='' (default), form=' ' (add spaces), and form='x' to add '\x' prefixes. I don't know that adding '\x' would be useful. The prefixes are not accepted by .fromhex. |
|
binascii.hexlify('abc') doesn't work in 3.4. I assume this is a new thing for 3.5 >>> import binascii
>>> binascii.hexlify('abc')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'str' does not support the buffer interface
>>>
>>> binascii.hexlify(b'abc')
b'616263'@terry I propose the hex functions for bytes/memoryview/bytearray should be as follow. I prefer to not have the '0x' prefix at all, but I understand all other hex functions adds it. would be good to have the option to not have the prefix. bytes.hex( byte_order = sys.byteorder ) returns a hex string in small letter. ex. c0ffee bytes.HEX( byte_order = sys.byteorder ) returns a hex string in capital letters. ex. DEADBEEF bytes.from_hex( hex_str, byte_order = sys.byteorder ) returns a bytes object. ex. b'\xFE\xFF' another more flexible way is to have hex function accept a format similar to how sscanf works, but this will probably bring lots of trouble for all kinds of variants to support and the required error checks. |
|
Just as a recap of at least some of the *current* ways to do a bytes -> hex conversion: >>> import codecs
>>> codecs.encode(b"abc", "hex")
b'616263'
>>> import binascii
>>> binascii.hexlify(b"abc")
b'616263'
>>> import base64
>>> base64.b16encode(b"abc")
b'616263'
>>> hex(int.from_bytes(b"abc", "big"))
'0x616263'
>>> hex(int.from_bytes(b"abc", "little"))
'0x636261'Thus, the underlying purpose of this proposal is to provide a single "more obvious way to do it". As per the recent discussion on python-ideas, the point where that is most useful is in debugging output. However, rather than a new method on bytes/bytearray/memoryview for this, I instead suggest it would be appropriate to extend the default handling of the "x" and "X" format characters to accept arbitrary bytes-like objects. The processing of these characters would be as follows: "x": display a-f as lowercase digits Output order would match binascii.hexlify() Examples: format(b"xyz", "x") -> '78797a' format(b"xyz", ".1x") -> '78 79 7a' format(b"xyz", ",.1x") -> '78,79,7a' This approach makes it easy to inspect binary data, with the ability to inject regular spaces or commas to improved readability. Those are the basic features needed to support debugging. Anything more complicated than that, and we're starting to want something more like the struct module. |
|
The proposal is to add a .hex method (similar to binascii.hexlify) that is the inverse of .fromhex (similar to binascii.unhexlify), as originally specified in PEP-358. To aid debugging, I would change spaces to be None or a positive int n to insert a space every n bytes. So .hex(8) for an array of 64 bit ints. |
|
natural bytes do not have space between them. I would think adding space is for typesetting situation which should be done by user's post-processing. I agree to not have any prefix to make .hex and from_hex uniform. the \x is the str representation of bytes when you print a bytes object directly in Python. the actual bytes object doesn't have that \x prefix. |
|
Open question: the current patch adds bytes.hex() and bytearray.hex(). Should we also add memoryview.hex(), or split that suggestion out to a separate proposal? |
|
I'd say add memoryview.hex() here as everything seems related. Victor has also mentioned memoryview in msg226692. |
|
int has int.from_bytes and int.to_bytes. Currently, bytes has bytes.fromhex. Would the core developers please consider naming the method "bytes.tohex" instead of "bytes.hex", so there's at least a modicum of consistency in the method names of Python's builtin types? |
On 11.09.2014 01:04, Nick Coghlan wrote:
>
> Nick Coghlan added the comment:
>
> Just as a recap of at least some of the *current* ways to do a bytes -> hex conversion:
>
>>>> import codecs
>>>> codecs.encode(b"abc", "hex")
> b'616263'
>>>> import binascii
>>>> binascii.hexlify(b"abc")
> b'616263'
>>>> import base64
>>>> base64.b16encode(b"abc")
> b'616263'
>>>> hex(int.from_bytes(b"abc", "big"))
> '0x616263'
>>>> hex(int.from_bytes(b"abc", "little"))
> '0x636261'
>
> Thus, the underlying purpose of this proposal is to provide a single "more obvious way to do it". As per the recent discussion on python-ideas, the point where that is most useful is in debugging output.
>
> However, rather than a new method on bytes/bytearray/memoryview for this, I instead suggest it would be appropriate to extend the default handling of the "x" and "X" format characters to accept arbitrary bytes-like objects. The processing of these characters would be as follows:
>
> "x": display a-f as lowercase digits
> "X": display A-F as uppercase digits
> "#": includes 0x prefix
> ".precision": chunks output, placing a space after every <precision> bytes
> ",": uses a comma as the separator, rather than a spaceHmm, but those would then work for str.format() as well, right ? Since "x" and "X" are already used to convert numbers to hex |
|
Updated issue title to indicate proposal also covers bytearray and memoryview. |
ncoghlan
changed the title
|
Arnon is here at the PyCon 2015 sprints, so bringing the current status up to date: = Why *.hex()? =
= Why add it to the builtin types? =
= Why postpone configurability and str.format() integration? =
|
|
I added the implementation for memoryview, updated to use PyUnicode_New etc., and moved the common implementation to its own file for code reuse. |
|
minor updates to stdtypes.rst. I also want to add a line to whatsnew/3.5 but don't know how to put it in words - maybe it's better if someone with better english will add it. |
|
bytes.hex-1.diff looks good, i'll take care of committing this and adding a what's new entry. thanks! |
|
New changeset c9f1630cf2b1 by Gregory P. Smith in branch 'default': |
|
New changeset 955a479b31a8 by Gregory P. Smith in branch 'default': |
|
note quite fixed, looks like some of the buildbots are having fun not compiling with this change: http://buildbot.python.org/all/builders/x86%20Tiger%203.x/builds/9569/steps/compile/logs/stdio investigating... |
|
i missed the hg adds :) |
|
New changeset a7737204c221 by Gregory P. Smith in branch 'default': |
|
New changeset 7f0811452d0f by Gregory P. Smith in branch 'default': |
|
Thank you Arnon, and thank you Greg! |
|
I see some _Py_strhex related link errors on the Windows buildbots: http://buildbot.python.org/all/builders/x86%20Windows7%203.x/builds/9642/steps/compile/logs/stdio |
|
New changeset b46308353ed9 by Gregory P. Smith in branch 'default': |
|
New changeset f3d8bb3ffa98 by Stefan Krah in branch '3.5': |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: