_multiprocessing.c warnings under 64-bit Windows

[pitrou]

BPO	9782
Nosy	@loewis, @pitrou, @tjguk, @zware, @serhiy-storchaka, @zooba, @applio

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-09-11.13:10:12.961>
created_at = <Date 2010-09-06.12:43:03.518>
labels = ['extension-modules', 'build', 'OS-windows']
title = '_multiprocessing.c warnings under 64-bit Windows'
updated_at = <Date 2019-09-11.13:10:12.960>
user = 'https://github.com/pitrou'

bugs.python.org fields:

activity = <Date 2019-09-11.13:10:12.960>
actor = 'steve.dower'
assignee = 'none'
closed = True
closed_date = <Date 2019-09-11.13:10:12.961>
closer = 'steve.dower'
components = ['Extension Modules', 'Windows']
creation = <Date 2010-09-06.12:43:03.518>
creator = 'pitrou'
dependencies = []
files = []
hgrepos = []
issue_num = 9782
keywords = []
message_count = 4.0
messages = ['115699', '116173', '222617', '238370']
nosy_count = 9.0
nosy_names = ['loewis', 'pitrou', 'tim.golden', 'jnoller', 'janglin', 'zach.ware', 'serhiy.storchaka', 'steve.dower', 'davin']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue9782'
versions = ['Python 3.4', 'Python 3.5']

[pitrou]

There are various warnings when compiling the _multiprocessing extension in 64-bit mode under Windows. Many seem related to the fact that read() and friends under Windows take "int" size arguments rather than "size_t".

20>------ Build started: Project: _multiprocessing, Configuration: Release x64 ------
20>Compiling...
20>multiprocessing.c
20>win32_functions.c
20>socket_connection.c
20>..\Modules\_multiprocessing\socket_connection.c(32) : warning C4267: 'function' : conversion from 'size_t' to 'int', possible loss of data
20>..\Modules\_multiprocessing\socket_connection.c(54) : warning C4267: 'function' : conversion from 'size_t' to 'int', possible loss of data
20>..\Modules\_multiprocessing\socket_connection.c(126) : warning C4244: '=' : conversion from 'Py_ssize_t' to 'int', possible loss of data
20>..\Modules\_multiprocessing\socket_connection.c(137) : warning C4244: '=' : conversion from 'Py_ssize_t' to 'int', possible loss of data
20>..\Modules\_multiprocessing\socket_connection.c(145) : warning C4244: '=' : conversion from 'Py_ssize_t' to 'int', possible loss of data
20>z:\py3k\svn\modules\_multiprocessing\connection.h(139) : warning C4244: '=' : conversion from 'Py_ssize_t' to 'int', possible loss of data
20>z:\py3k\svn\modules\_multiprocessing\connection.h(183) : warning C4244: 'function' : conversion from 'Py_ssize_t' to 'int', possible loss of data
20>z:\py3k\svn\modules\_multiprocessing\connection.h(237) : warning C4244: 'function' : conversion from 'Py_ssize_t' to 'int', possible loss of data
20>z:\py3k\svn\modules\_multiprocessing\connection.h(285) : warning C4244: '=' : conversion from 'Py_ssize_t' to 'int', possible loss of data
20>z:\py3k\svn\modules\_multiprocessing\connection.h(323) : warning C4244: 'function' : conversion from 'Py_ssize_t' to 'int', possible loss of data
20>semaphore.c
20>pipe_connection.c
20>..\Modules\_multiprocessing\pipe_connection.c(24) : warning C4267: 'function' : conversion from 'size_t' to 'DWORD', possible loss of data
20>..\Modules\_multiprocessing\pipe_connection.c(50) : warning C4267: 'function' : conversion from 'size_t' to 'DWORD', possible loss of data
20>z:\py3k\svn\modules\_multiprocessing\connection.h(139) : warning C4244: '=' : conversion from 'Py_ssize_t' to 'int', possible loss of data
20>z:\py3k\svn\modules\_multiprocessing\connection.h(183) : warning C4244: 'function' : conversion from 'Py_ssize_t' to 'int', possible loss of data
20>z:\py3k\svn\modules\_multiprocessing\connection.h(237) : warning C4244: 'function' : conversion from 'Py_ssize_t' to 'int', possible loss of data
20>z:\py3k\svn\modules\_multiprocessing\connection.h(285) : warning C4244: '=' : conversion from 'Py_ssize_t' to 'int', possible loss of data
20>z:\py3k\svn\modules\_multiprocessing\connection.h(323) : warning C4244: 'function' : conversion from 'Py_ssize_t' to 'int', possible loss of data
20>Linking...
20> Creating library Z:\py3k\svn\PCbuild\\amd64\\_multiprocessing.lib and object Z:\py3k\svn\PCbuild\\amd64\\_multiprocessing.exp
20>Generating code
20>Finished generating code
20>Build log was saved at "file://Z:\py3k\svn\PCbuild\x64-temp-Release\_multiprocessing\BuildLog.htm"
20>_multiprocessing - 0 error(s), 17 warning(s)

[loewis]

It seems that multiprocessing is fairly conservative wrt. 64-bit support. For example, conn_send_string has a string limit of 0x7fffffff. Therefore, several of the warnings are harmless; the respective lengths fit into int just fine.

I recommend that the size_t parameters are converted to int in the places where this is actually known to be correct from control flow (e.g. both cases of conn_send_string), along with an assert() that the value being cast is <= INT_MAX.

IIUC, _conn_recvall can legitimately return int, no need to return Py_ssize_t: it will only return error codes (possibly MP_SUCCESS). Also, it might be sufficient to restrict the length of _conn_recvall to MAX_INT.

The alternate route would be to actually widen socket_connection to support 64 bit transmits (*); this would be Jesse's call. It's probably acceptable to defer this until somebody needs this enough to contribute a patch. Copying 2GiB over a pipe takes 1.5s on my system.

(*) My protocol proposal would be this: a length with the highest bit set (i.e. >= 0x80000000) indicates a huge message, where another four bytes of of length follow, for a total of 63 bits of length (MSB is unavailable as it serves as the length-of-length indicator). This would not waste bytes for small messages, and be backwards compatible.

[BreamoreBoy]

I'd assume that these were cleared years ago.

[BreamoreBoy]

Where do we currently stand with all compiler warnings, I'm still seeing some but I recall that we've other open issues about this problem?