BytesWarning in execnet when deserializing float #596
Comments
|
NOTE: This was most likely a red herring So I think the Python warning stack is just a complete lie. When I use gdb and break in Using the Python gdb macros I was able to find out that this is apparently happening with a comparison to and getting the python stack from gdb tells me something much more plausible: It remains a mystery why Python thinks this is triggered by the Next steps:
|
|
Hi @The-Compiler, Ouch, yeah that's a doozy. I don't know |
|
Hey folks, I was intrigued by this and took some time to debug it, here are my findings. There's an internal cache in >>> import struct
>>> struct.calcsize(b'!d') # cache for '!d' uses bytes
8
>>> struct.calcsize('!d') # so there's a warning when trying to use str
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
BytesWarning: Comparison between bytes and string
>>> struct.calcsize('>d') # cache for '>d' uses str
8
>>> struct.calcsize(b'>d') # so now the warning is inverted, it shows up when trying to use bytes
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
BytesWarning: Comparison between bytes and string
>>> struct.calcsize('>d') # no problem when using str
8In this case, hypothesis is to blame, since it's the first one to populate the cache (via |
|
Can we fix this by wrapping struct calls with a retry that attempts with both types of string? |
|
Some possible ways to fix it:
|
Probably, or, as a workaround, just disable this warning in the wrapped struct calls. |
|
@tadeu Amazing work! I guess my stack trace above was a red herring then, and probably caused by some other BytesWarning which is already ignored somewhere. This indeed seems like a CPython bug. Do you want to report it there, or should I do so? I'm closing this for now as I don't think this is something that should/can be fixed in xdist or execnet. I can ignore the warning via |
|
I'll report there and post a link here 👍 |
|
It seems to be a big problem with
|
|
Ra! @tadeu is amazing! 😁 |
|
The problem is that their hashes are equal: >>> hash('a')
-8194812638379419632
>>> hash(b'a')
-8194812638379419632And since hashes are never a guarantee of equality, a dict internally will use |
|
I've posted it here: https://bugs.python.org/issue41777 |
Yeah, this makes sense, dicts are type-agnostic (reason why |
|
Yup - I guess Python should internally somehow suppress that warning when doing equality comparisons for dicts, or perhaps special-case this case to always assume they're not equal. No idea who to ping in the Python bugtracker about that, so for now I just added some people involved in the |
|
With HypothesisWorks/hypothesis#2620 hypothesis now uses str instead of bytes for |
Behold, this is probably one of the strangest issues I've ever had the joy (?) of debugging. I'd really appreciate some help, as I'm completely stumped here.
I run tests in qutebrowser with
python -bb, which enables byte warnings and turns them into errors.When I run the following in the qutebrowser repository:
tox -e py38-pyqt515 --notest.tox/py38-pyqt515/bin/pip install pytest-xdisttox -e py38-pyqt515 -- -n 2I get crashing gateways with:
This happens when trying to deserialize the test duration (?). When I print
self.stackbefore that line I see e.g.:As you might guess, there's no comparison between bytes and strings involved, this is a completely normal
struct.unpackcall:(Pdb) interact *interactive* >>> FLOAT_FORMAT '!d' >>> binary b'?\xb7\xd8x\x89t\x00\x00'Yet I can reproduce the error when I run the same thing by hand, inside the pdb environment:
No funny things are going on, as far as I can see:
Here's where things are getting weird: Both according to the docs and according to the cpython code, both
>and!mean the same - but using>instead works!And because this wasn't weird enough, I can not reproduce the issue in a Python console with the same Python version, on the same machine:
Any idea what kind of black magic xdist/execnet could be doing to cause this? I haven't been able to find a reproducer outside of qutebrowser so far, but I guess that's not too surprising with how strange this issue is. Did I find a bug in cpython or something?
The text was updated successfully, but these errors were encountered: