Compilation warnings under x64 Windows #53775
Comments
|
A 64-bit build under Windows produces many compilation warnings, mostly related to lossy conversions between different int sizes. 5>..\Python\pythonrun.c(1210) : warning C4267: '=' : conversion from 'size_t' to 'int', possible loss of data 5>..\Python\Python-ast.c(3403) : warning C4244: 'function' : conversion from 'Py_ssize_t' to 'int', possible loss of data 5>..\Python\dtoa.c(1529) : warning C4244: '=' : conversion from '__int64' to 'int', possible loss of data 5>..\Python\pystrtod.c(902) : warning C4244: 'function' : conversion from 'Py_ssize_t' to 'int', possible loss of data 5>..\Python\peephole.c(84) : warning C4244: '=' : conversion from 'Py_ssize_t' to 'unsigned char', possible loss of data 5>..\Python\getargs.c(390) : warning C4244: 'function' : conversion from 'Py_ssize_t' to 'int', possible loss of data 5>z:\py3k\svn\python\../Objects/stringlib/formatter.h(984) : warning C4244: 'function' : conversion from 'Py_ssize_t' to 'int', possible loss of data 5>..\Python\compile.c(339) : warning C4244: 'function' : conversion from 'Py_ssize_t' to 'long', possible loss of data 5>..\Python\ceval.c(3967) : warning C4244: '=' : conversion from 'Py_ssize_t' to 'int', possible loss of data 5>..\PC\winreg.c(769) : warning C4267: '=' : conversion from 'size_t' to 'DWORD', possible loss of data 5>..\PC\_subprocess.c(345) : warning C4244: '=' : conversion from 'Py_ssize_t' to 'int', possible loss of data 5>..\Parser\tokenizer.c(658) : warning C4267: 'initializing' : conversion from 'size_t' to 'int', possible loss of data 5>..\Parser\parsetok.c(199) : warning C4244: '=' : conversion from '__int64' to 'int', possible loss of data 5>..\Parser\grammar.c(66) : warning C4244: 'return' : conversion from '__int64' to 'int', possible loss of data 5>..\Objects\typeobject.c(6173) : warning C4244: '=' : conversion from 'Py_ssize_t' to 'int', possible loss of data 5>..\Objects\setobject.c(743) : warning C4244: '=' : conversion from 'Py_ssize_t' to 'long', possible loss of data 5>..\Objects\obmalloc.c(904) : warning C4244: '=' : conversion from '__int64' to 'unsigned int', possible loss of data 5>..\Objects\funcobject.c(633) : warning C4244: 'function' : conversion from 'Py_ssize_t' to 'int', possible loss of data 5>..\Objects\frameobject.c(480) : warning C4244: '=' : conversion from 'Py_ssize_t' to 'int', possible loss of data 5>..\Objects\fileobject.c(401) : warning C4244: 'function' : conversion from 'Py_ssize_t' to 'unsigned int', possible loss of data 5>..\Objects\descrobject.c(891) : warning C4244: '=' : conversion from '__int64' to 'int', possible loss of data 5>..\Objects\codeobject.c(495) : warning C4244: 'initializing' : conversion from 'Py_ssize_t' to 'int', possible loss of data 5>..\Objects\bytesobject.c(786) : warning C4244: '=' : conversion from 'Py_ssize_t' to 'int', possible loss of data 5>..\Objects\bytearrayobject.c(1222) : warning C4244: '=' : conversion from 'Py_ssize_t' to 'int', possible loss of data 5>..\Objects\bytes_methods.c(454) : warning C4244: '=' : conversion from 'Py_ssize_t' to 'char', possible loss of data 5>..\Modules\_io\fileio.c(483) : warning C4244: 'function' : conversion from 'Py_ssize_t' to 'unsigned int', possible loss of data 5>..\Modules\zlibmodule.c(127) : warning C4244: '=' : conversion from 'Py_ssize_t' to 'int', possible loss of data 5>..\Modules\sha512module.c(558) : warning C4244: 'function' : conversion from 'Py_ssize_t' to 'int', possible loss of data 5>..\Modules\sha256module.c(492) : warning C4244: 'function' : conversion from 'Py_ssize_t' to 'int', possible loss of data 5>..\Modules\sha1module.c(399) : warning C4244: 'function' : conversion from 'Py_ssize_t' to 'unsigned long', possible loss of data 5>..\Modules\posixmodule.c(394) : warning C4267: '=' : conversion from 'size_t' to 'int', possible loss of data 5>..\Modules\mmapmodule.c(881) : warning C4244: '=' : conversion from 'Py_ssize_t' to 'char', possible loss of data 5>..\Modules\md5module.c(423) : warning C4244: 'function' : conversion from 'Py_ssize_t' to 'unsigned long', possible loss of data 5>..\Modules\binascii.c(667) : warning C4244: '=' : conversion from 'Py_ssize_t' to 'unsigned char', possible loss of data 5>..\Modules\audioop.c(332) : warning C4244: 'function' : conversion from 'Py_ssize_t' to 'int', possible loss of data 5>z:\py3k\svn\modules\_sre.c(1898) : warning C4244: '=' : conversion from 'Py_ssize_t' to 'int', possible loss of data 5>..\Modules\_pickle.c(284) : warning C4244: '=' : conversion from 'Py_ssize_t' to 'int', possible loss of data 5>..\Modules\_csv.c(971) : warning C4244: '=' : conversion from 'Py_ssize_t' to 'int', possible loss of data 5>..\Modules\_collectionsmodule.c(1124) : warning C4244: 'function' : |
pitrou
added
extension-modules
|
This must be a duplicate report - this issue has been known for years, and Kristjan and others (including myself) volunteered to work on it, and did actually work on it. I don't think it's necessary to have a bug report in the bug tracker for it. If you find a place that you would like to fix, just go ahead and fix it. Please keep in mind the following principles:
This leaves us with a number of cases where there are pragmatic Python limitations (such as the data type for line numbers, or the number of values when building a tuple in source) which make it impossible to get overflows in the current implementation. Such limitations should be collected in some place, and then SAFECASTs should be added, along with asserts at startup (e.g. that a certain datatype has no more than 2 bytes). |
|
Created bpo-9611 to report the FileIO 64-bit issue under Windows. |
|
Created bpo-9612 for the mostly harmless warnings in the set implementation. |
|
The zlib warnings are genuine and there's a specific bug for 64-bitness of zlibmodule.c: bpo-8650. |
|
Opened bpo-9613 for 64-bitness of process ids. |
|
Opened bpo-9614 for the _pickle issues (which look quite legitimate). |
|
r87733 fixes some conversions in:
For md5, sha1, sha256 and sha512: a smaller type (int) can be used for the local n variable. I don't know if it impacts performances or not. md5 and sha1 used "unsigned long", sha256 and sha512 used "int". |
|
r87733 in pyexpat.c modified a call to PyErr_Format with a %zi format code. |
|
bpo-10823 has been marked as a duplicate of this issue. |
|
haypo: please avoid introducing explicit casts (such as the one in pyexpat.c). Use SAFECASTs instead. If you are absolutely certain that a cast cannot possibly truncate, add a comment explaining why that is. In the specific case, I think it actually *is* possible that the cast truncates, because the case that read() returned too many bytes is incorrectly handled (it should set len to -1). |
|
amaury> r87733 in pyexpat.c modified a call to PyErr_Format with a %zi Oh, that's surprising. Why %zd is supported but not %zi? PyUnicode_FromFormatV() supports %d, %ld, %lld, %zd, %i, but not %li, %lli or %zi. Before fixing PyUnicode_FromFormatV(), I fixed pyexpat.c: r87747. MvL> If you are absolutely certain that a cast cannot possibly truncate, Ah yes, sorry, I forgot to add a comment: done in r87746. |
I asked dmalcolm on IRC if int or Py_ssize_t should be used: "this is the number of digits to use in the floating point representation, right? it strikes me that if you're overflowing an int, you're asking for a very very very long string" (yes, INT_MAX is usually 2^31) "... looking at usage of precision seems to ultimately land in Python/dtoa.c, which uses int..." |
But the comment is actually wrong: It says len <= buf_size <= INT_MAX (see above) however, len > buf_size may happen, after this code: if (len > buf_size) {
PyErr_Format(PyExc_ValueError,
"read() returned too much data: "
"%i bytes requested, %zd returned",
buf_size, len);
goto finally;
} |
Oh. Not only is the comment is wrong, but the code is also wrong. It |
Right. See msg125367. |
|
pyexpat.patch: new try, does it look better? |
Fixed by r87911. |
|
socketmodule_int.patch: Fix similar to the fix for bpo-9611: clamp length to INT_MAX on Windows for recv(), recvfrom(), send() and sendto(). |
|
ssl_sock_fd_int.patch: add SOCKET_T_AS_INT(fd) macro to cast a socket descriptor (SOCKET_T) to int. I choosed a macro just to not have to repeat a comment twice :-) It looks like it is a safe to cast SOCKET_T to int. My comment: /* On Windows, SOCKET_T is an offset in per-process handle table. |
Reference: http://www.mail-archive.com/openssl-dev@openssl.org/msg23781.html |
|
ssl_write.patch: always clamp SSL_Write() size to INT_MAX. This patch fixes also an overflow if Py_ssize_t is 32 bits (if the size is bigger than INT_MAX). |
|
hash.patch: continue my fix on md5, sha1, sha256, sha512. |
|
_sre.patch: store sre_match() result in Py_ssize_t instead of int. |
|
... and bpo-9566.patch: various fixes. Interesting parts: ---
--- LOCAL(PyObject*)
-expat_parse(XMLParserObject* self, char* data, int data_len, int final)
+expat_parse(XMLParserObject* self, char* data, Py_ssize_t data_len, int final)
{
int ok;
- ok = EXPAT(Parse)(self->parser, data, data_len, final);
+ if (data_len > INT_MAX) {
+ PyErr_SetString(PyExc_OverflowError, "length doesn't fit in an int");
+ return NULL;
+ }
+
+ ok = EXPAT(Parse)(self->parser, data, (int)data_len, final);
---
+ Py_hash_t hash = PyObject_Hash(arg); |
|
New changeset 43fd627cc060 by Victor Stinner in branch 'default': |
|
Attached file is a copy of the remaining warnings on the AMD64 Windows7 SP1 3.x builder. |
|
I forgot to mention this issue in the following changesets. changeset: 87128:f0a7924fac56 changeset: 87129:8adbd8a3a626 changeset: 87130:baab11a466ab changeset: 87131:e086cb1c6e5a changeset: 87132:cf399d13a707 changeset: 87133:309d855ebc3e changeset: 87134:7cd4c3e9e310 changeset: 87135:9e25367095c4 changeset: 87229:ac4272df1af6 INT_MAX states and labels should be enough for everyone changeset: 87230:19e900e3033f Python parser doesn't support lines longer than INT_MAX bytes yet changeset: 87231:103998db4407 Fix compiler warnings on Windows 64-bit changeset: 87232:0f7f1f2121a1 Fix a compiler warning on Windows 64-bit changeset: 87233:855e172bcac4 changeset: 87234:40d25b2b93f0 Fix compiler warnings on Windows 64-bit changeset: 87235:d1dc7888656f changeset: 87236:2a01ca4b0edc Fix a compiler warning on Windows 64-bit |
|
New changeset 68fd86a83ece by Victor Stinner in branch 'default': |
|
New changeset 8d3e85dfa46f by Victor Stinner in branch 'default': |
|
New changeset ee4da7291211 by Victor Stinner in branch 'default': |
|
New changeset 116bd550e309 by Victor Stinner in branch 'default': |
|
Are there any more warnings that need silencing or can this be closed as fixed? How do I check for these warnings as I don't have a 64 bit machine? |
Check buildbots. Example: There are still open issues related to this one: bpo-18295 at least. |
|
I submitted a PR which fixes some more warnings. The ones I haven't fixed are detailed in the PR description. |
|
New changeset e7bc7aa by Steve Dower (Segev Finer) in branch 'master': |
|
The only warnings that remain are:
|
|
There is also: commit cdbcb77 |
It fixed the last known warning. Thanks everyone who was involved in this funny ride. It's now time to close the issue! It just took 9 years to fix all compiler warnings on 64-bit Windows ;-) |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: