New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
struct returns incorrect 4 byte float #48364
Comments
FYI Actual version is 2.5.2 running on Linux RHE4. >>> import struct
>>> fmt ='>f'
>>> v=1.8183e-7
>>> v
1.8183000000000001e-07
>>> s=struct.pack(fmt,v)
>>> struct.unpack(fmt,s)
(1.818300034983622e-07,) Looks to me like the float->double conversion is not being zeroed out |
Why do you use float (32 bits) instead of double (64 bits)? Your If you convert 64 bits float to 32 bits, for sure you will loose some |
That's not my code - it's an example ;) My code reads binary data from a hardware system that is encoding 32 bit There is no 32 bit float type in python that I can allocate. If you >>> v=123456789
>>> struct.unpack(fmt,struct.pack(fmt,v))
(123456792.0,) |
I don't understand your problem/question. It's not a bug of Python, |
I'm receiving a 32 bit floating point number encoded in binary format. After doing some experiments, I think the problem is a basic fact of Since this is a function of the underlying C language, I'll withdraw the |
The problem is not from Python but from your FPU during the conversion #include <stdio.h>
int main()
{
float f;
double d;
d = 1.8183;
printf("d=%.20f\n", d);
f = (float)d;
d = (double)f;
printf("f=%.20f\n", f);
printf("d=%.20f\n", d);
return 0;
} Result: |
I think the complaint is that presumably, when expanding the float to py> Seems to me that the zero-padding works just well. TD22057, why do you say that 1.818300034983622e-07 is not the closest |
Martin is correct. I expected (naively) that struct would zero pad the
digits beyond the significant digits of a float. As to whether it's
exact or not, see my first message:
>>> v=1.8183e-7
>>> v
1.8183000000000001e-07 Since 32 bit floats only have ~7 digits of precision, I expected to get Like I said, you can mark this bug as invalid since Python is just |
"Since 32 bit floats only have ~7 digits of precision, I expected to This problem is a well known problem of conversion from base 2 (IEEE "Python is just reflecting what C is doing": the problem is deeper in |
The problem is indeed deeper, however, I doubt GMP is an answer here: |
@loewis: Yes, the initial problem is about unpack("f", bytes). It's |
I agree that it should be closed; people should read general CS
Interestingly enough, it is possible - using the OPs approach.
I think it's documented somewhere that a Python float is represented |
I have to disagree. It seems entirely reasonable to expect that unpack should return the same value passed to pack. That it doesn't (as of 2.6.5 at least) is completely unexpected and undocumented. And yes I understand the limitations of floating point numbers. I suggest that struct should be fixed so that struct.unpack(fmt,struct.pack(fmt,v)) == v and format is something like '!f'. This can be done in C code (I do it) for IEEE 754 floats. At the very list this unexpected behavior should be documented in struct. |
Robert: Can you please suggest an algorithm that would have given the result you expect (preferably as a C program, using a string literal as its input, for some platform)? Ideally, we would stick to the example given by the OP. |
I don't think this needs to be documented beyond the limitations of floating-point that are already documented in the tutorial. It's the obvious behaviour: double to float (when packing) converts to the nearest float; the float to double conversion is exact. |
[Robert]
Robert: notice that a *Python* float (a *64-bit* C double internally) is here being stored as a *32-bit* float, losing precision. So no, it's not at all reasonable to expect that unpack should return the same value passed to pack---it's mathematically impossible for it to do so. There are (around) 2**64 distinct Python floats, and only 2**32 ways to pack them using '<f'. When packing / unpacking using '<d', it *is* reasonable to expect the value to be recovered exactly, and as far as I know that's always what happens (barring peculiarities like NaN payloads not being reproduced exactly). |
Martin: in C I have the luxury of using 32 bit floats; not an option in Python. Simple code doing the moral equivalent of NTOHL(HTONL()) works in this case for C but wouldn't help for Python. Mark: I understand about the precision truncation issue and how Python does floating point arithmetic. This C code clearly demonstrates what is going on: #include <stdio.h>
int main(int argc, char *argv[])
{
double d1 = 6.21;
float f = 6.21;
double d2 = f;
printf("double: %.15f\n", d1);
printf("float: %.15f\n", f);
printf("double converted from float: %15.15f\n", d2);
} The point here is about the contract of struct, NOT how Python does floating point arithmetic. The contract is: what pack packs, unpack will unpack resulting in the original value. At least, that is what the documentation leads you to believe. For the 'f' format character, this contract is broken because of a basic implementation detail of Python and there is nothing in the documentation for struct that *directly* lets you know this will happen. After all, the mentions in the documentation about 32 bit versus 64 bit talk about C not Python! Even worse, there is no straightforward way (that I'm aware of) to write portable tests for code using the 'f' format character. In my case I'm writing a tool that creates message codecs in multiple languages and the most basic unit test goes something like this: m1 = example.message()
m1.f1 = 6.21
b = m.encode() # uses struct pack
m2 = example.message(b) # uses struct unpack
if m1 != m2: # rich comparison
print('fail') This test will fail when you use the 'f' format code. I suggest two things could be done to improve the situation:
m1 = example.message()
m1.f1 = 6.21.as_32_bit_float() # Does the precision truncation upfront
b = m.encode() # uses struct pack
m2 = example.message(b) # uses struct unpack
if m1 != m2: # rich comparison
print('fail') I'd expect this test to pass. Hope this long-winded note helps. |
If you agree that Python actually behaves correct, I fail to If all you want is a documentation change, can you please propose
If you use numbers that are exactly representable as floats, [...]
So use 6.25 instead. |
I don't agree that Python is behaving correctly as far as the documented contract for struct is concerned. I disagree with the statement in the preceding msg74708 which says:
Aside from being patronizing it doesn't solve the problem in any meaningful way.
A reasonable suggestion, but it is a constrained definition of "portable". Since most (or nearly all?) modern platforms use '754 it is probably not a bad constraint, given that struct explicitly uses '754.
I confess, the "CS introductory material" I read 30 years ago (predating '754) don't give me enough information to know if this is correct. Anyway:
It isn't exactly "all I want", but it is a good start. I note that msg74705 suggests adding documentation to struct about the 'f' format code. First of all, as far as I know, struct is the only place where this issue of 32 bit versus 64 bit floating point numbers shows up in Python because the rest of Python uses only 64 bit numbers. (Is there anywhere else in Python where a 32 bit float is converted to a 64 bit float?) So the documentation probably belongs in struct. I would add to note 4 of 7.3.2.2 (in the 2.7.1 documentation) something like: "Note that 32 bit representations do not generally convert exactly to 64 bit representations (which Python uses internally) so that the results of unpack(fmt,pack(fmt,number)) may not equal number when using the 'f' format character." It would be friendly to add an example at the bottom demonstrating the issue and incorporating your comments about fractions and non-fractional values. >>> x = unpack('!f', pack('!f', 6.24))[0]
>>> x == 6.24
False
>>> x = unpack('!f', pack('!f', 6.25))[0]
>>> x == 6.25
True |
The suggested examples are misleading because they use 6.24 which is not exactly representable in binary floating point. Representation issues are orthogonal to the OP's issue which is really just a simple rounding example: >>> x = float.fromhex('0x0.1234560000001')
>>> unpack('!f', pack('!f', x))[0].hex()
'0x1.2345600000000p-4' Also, if something like the suggested note is adopted, it needs to be worded in a way that doesn't imply that the struct implementation is broken or misdesigned. A better note would focus on the basic (and obvious) fact that downgrading from double precision to single precision |
Agree.
Sort of where I was going, but I'm sure my text could be vastly improved.
I'd quibble with this for two reasons:
One comment about portability I forgot to mention earlier: I don't know how wed Python is to '754 or even binary floating point representations. My personal belief is that it should be possible to write a test so that the unpack(fmt, pack(fmt, precision_truncate(number))) == precision_truncate(number) test works for any legal number on any platform. I don't like the idea that one has to pick specific numbers based on knowledge of the platform's floating point format. I acknowledge that this may not bother others as much as it bothers me though. I'm a portability nut. |
For completeness: msg131234 states that the issue of 64 bit -> 32 bit precision truncation is covered in the floating point tutorial. I believe that is incorrect; at least I can't find it explicitly mentioned. Ref: http://docs.python.org/tutorial/floatingpoint.html. If struct is the only place this (64->32 bit precision truncation) can happen in Python, the lack of discussion in the tutorial makes sense. Otherwise, a sentence about it should be added to the tutorial. As it is, there is no _explicit_ mention of this anywhere in Python documentation. It is all well and good to state that it is "obvious", but it seems that explicit documentation is preferable to implicit documentation, given the rarity of the issue in Python and the meager cost of adding a sentence here or there. Incidentally, it is simple to create the truncation routine I mention earlier: >>> def fptrunc(value):
... return unpack('!f', pack('!f', value))[0]
...
>>> fptrunc(6.24)
6.2399997711181641
>>> fptrunc(6.25)
6.25 But this has the questionable smell of using pack/unpack in a test of pack/unpack. It's sorta OK for _users_ of pack/unpack though. A quick scan of the Python source code shows that only two things try to pack 4 byte floats: struct and ctypes and both of these use the underlying Python float object routines. So a better way of doing the truncation is to use ctypes: >>> def fptrunc(value):
... return c_float(value).value
...
>>> fptrunc(6.24)
6.2399997711181641
>>> fptrunc(6.25)
6.25 Doing this allows you to write tests that work for any number and don't require the use of magic numbers or knowledge of the underlying floating point implementation. Even if nothing gets put into the documentation, people will probably find this discussion by Googling. I can't imagine there is much more that can be said about this, so I'll leave you guys alone now... ;-) |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: