Invalid function cast warnings with gcc 8 for METH_NOARGS #77193
Comments
|
gcc 8 has added a new warning heuristic to detect invalid function casts and a stock python build seems to hit that warning quite often. The most common is the cast of a METH_NOARGS function (that uses just one argument) to a PyCFunction. The fix is pretty simple but needs to be applied widely. I'm slowly knocking them off in my spare time; WIP here, which has a few other types of warnings mixed in that I'll sift out during submission and also create separate bug reports for: https://github.com/siddhesh/cpython/tree/func-cast I'll clean up and post PR(s) once I am done but I figured I should file this report first since it is a pretty big change in terms of number of files touched and wanted to be sure that I'm making changes the way the community prefers. |
|
Argument Clinic generates the following declaration for the second parameter of METH_NOARGS functions: It would be nice to follow the same style. If the first parameter is of type PyObject* too, the explicit cast to PyCFunction can be removed. Skip the curses module. It will be converted to Argument Clinic. |
|
We are getting hit by that quite often on Fedora, with the transition to gcc 8 and it creates unnecessary noise at our build logs. Thanks for working on that. When you sent your PR I can test it within our build system and verify if it works. |
|
I don't have GCC 8 so I cannot verify this bug, but *function pointer casts* are fine - any function pointer can be cast to any other function pointer - it is only that they must *not* be called unless cast back again to be compatible with the function definition. Any fix to the contrary might well *cause* undefined behaviour! Could you provide a sample of the *actual warnings* so that they could be studied? |
Please see the attached PR; METH_NOARGS callbacks are inconsistent in their signature and many have just one argument while they're called with two arguments, the second being NULL. The patch fixes these to consistently take and call with two arguments.
Here's one of a few hundred. Objects/bytesobject.c:3085:25: warning: cast between incompatible function types from ‘PyObject * ()(striterobject *)’ {aka ‘struct _object * ()(struct <anonymous> *)’} to ‘PyObject * ()(PyObject *, PyObject *)’ {aka ‘struct _object * ()(struct _object *, struct _object *)’} [-Wcast-function-type] |
|
Yea, I looked into if (flags & (METH_NOARGS | METH_O)) {
PyCFunction meth = PyCFunction_GET_FUNCTION(func);
PyObject *self = PyCFunction_GET_SELF(func);
if (flags & METH_NOARGS && na == 0) {
C_TRACE(x, (*meth)(self,NULL)); x = _Py_CheckFunctionResult(func, x, NULL);
}The warning in GCC shouldn't probably have been enabled at all in However, the correct way to fix would be to have the METH_NOARGS case cast the function to the right prototype. There exists lots of existing code that *is* going to break too. Perhaps PyCFunction should declare no prototype, i.e. empty parentheses, for backwards compatibility: typedef PyObject *(*PyCFunction)();and deprecate it; start using a new typedef for it - and then add proper casts in every place that call a function. |
The explicit cast is precisely what enables the more nuanced function cast warning where it checks the function for type compatibility. That is, it will check the types of the return and arguments and then warn in case of mismatch.
AFAICT, there is no right prototype for the METH_NOARGS function; there used to be a PyCFunctionWithNoArguments but that seems to have fallen out of favour some time back. The prototype that seems to be commonly in use (and in clinic as well, which is what I based my patches on) is the PyCFunction, which seems like a safe way to do things.
I have a patch in the works that makes it PyObject *(*)(PyObject *, PyObject *, ...) which allows for two compulsory arguments (fits in with most cases, provided the METH_NOARGS patch is accepted) and then the rest depending on the type of the cast function. The rest of the PyMethodDef functions are much simpler fixes this way. It also seems like a more intuitive description of the callbacks. Then there are getter and setter and other function pointers that need similar fixes to METH_NOARGS. |
|
I forgot to clarify that the function cast warning allows for variable argument casts as a wildcard, which is my basis for the PyObject *(*)(PyObject *, PyObject *, ...) fix proposal. |
|
Siddhesh, it looks like your fixes make the C function signatures match the signature expected in the PyMethodDef structure. If so, I suggest to remove the (PyCFunction) casts from those structure definitions as well. For instance, now that we have PyObject *Noddy_name(Noddy *self, PyObject *Py_UNUSED(ignored)) I suggest changing PyMethodDef Noddy_methods[] = { to PyMethodDef Noddy_methods[] = { I suspect the casts were only added to hide compiler warnings related to this bug. If you are proposing to add an ellipsis (...) to the definition of PyCFunction, that seems misguided. I understand this is incompatible under standard C. Are you relying on a GCC extension perhaps? Python is used with other compilers too. |
|
Sorry, I realize there is a problem remaining with the pointer types for "Noddy_name" (Noddy vs PyObject pointers), so you can't remove the cast there. But my suggestion should still apply to other places, for instance the "error_out" method in Doc/howto/cporting.rst. |
|
Fair enough, I'll reduce my scope of changes for this patchset, especially since I'm unable to find enough time to work on the remaining changes I had thought of in the coming weeks. I'll post an updated patch soonish. |
|
The commit 55edd0c introduced a new warning: gcc -pthread -c -Wno-unused-result -Wsign-compare -g -Og -Wall -Wstrict-prototypes -O0 -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -I. -I./Include -fPIC -DPy_BUILD_CORE -o Objects/longobject.o Objects/longobject.c It seems like a conversion to (unaryfunc) is needed when passing long_long as nb_int in long_as_number. |
|
Yeah, there are multiple such uses that need wrappers to actually fix for gcc8, which will be released this week. I think I'll have more time for some more patches in this vein this weekend. |
|
The following PRs fix warnings in casts to PyCFunction for method conventions different from METH_NOARGS, METH_O and |
|
Is it possible/feasible to fix that for the 3.7 and 3.6 branches as well? |
|
See also bpo-33454 about a real bug found thanks to these warnings. |
|
I think it safer to silence this kind of warnings in 3.7 and 3.6. PR 6757 does this. |
|
I propose to enhance the changes made by PR 6748 and PR 6749 so that gcc 8 triggers a warning when the function type of a PyMethodDef element does not match its flags by using a macro (see below) that casts the function first to the function type corresponding to those flags and then to (void *) as done in those PRs to silence the warning when cast to PyCFunction. This would allow detecting bugs such as the one found in bpo-33454. With the following patch (it is just an example) gcc 8 would emit a warning if bisect_right() does not match the type (in the sense defined by -Wcast-function-type) of PyCFunctionWithKeywords: diff --git a/Modules/_bisectmodule.c b/Modules/_bisectmodule.c
index 831e10aa60..85fb0c188e 100644
--- a/Modules/_bisectmodule.c
+++ b/Modules/_bisectmodule.c
@@ -216,15 +216,14 @@ If x is already in a, insert it to the left of the leftmost x.\n\
Optional args lo (default 0) and hi (default len(a)) bound the\n\
slice of a to be searched.\n");
+#define SET_METH_VARARGS_KEYWORDS(func) \
+ (PyCFunction)(void *)(PyCFunctionWithKeywords)(func), METH_VARARGS|METH_KEYWORDS
+
static PyMethodDef bisect_methods[] = {
- {"bisect_right", (PyCFunction)bisect_right,
- METH_VARARGS|METH_KEYWORDS, bisect_right_doc},
- {"insort_right", (PyCFunction)insort_right,
- METH_VARARGS|METH_KEYWORDS, insort_right_doc},
- {"bisect_left", (PyCFunction)bisect_left,
- METH_VARARGS|METH_KEYWORDS, bisect_left_doc},
- {"insort_left", (PyCFunction)insort_left,
- METH_VARARGS|METH_KEYWORDS, insort_left_doc},
+ {"bisect_right", SET_METH_VARARGS_KEYWORDS(bisect_right), bisect_right_doc},
+ {"insort_right", SET_METH_VARARGS_KEYWORDS(insort_right), insort_right_doc},
+ {"bisect_left", SET_METH_VARARGS_KEYWORDS(bisect_left), bisect_left_doc},
+ {"insort_left", SET_METH_VARARGS_KEYWORDS(insort_left), insort_left_doc},
{NULL, NULL} /* sentinel */
}; |
|
Great idea! But the problem is that additional flags can be used, e.g. METH_CLASS. |
Right, https://docs.python.org/3/c-api/structures.html says: "The individual flags indicate either a calling convention or a binding convention". This may be overcome by introducing another macro with 2 arguments, the second argument being used to set the binding convention flag: #define SET_METH_VARARGS_KEYWORDS_FLAG(func, flag) \
(PyCFunction)(void *)(PyCFunctionWithKeywords)func, METH_VARARGS|METH_KEYWORDS|flag
#define SET_METH_VARARGS_KEYWORDS(func) SET_METH_VARARGS_KEYWORDS_FLAG(func, 0x0000)The refactoring would be quite large though. |
|
Well, there's only one problem with casting to void *: while converting the function pointer to another *is* standard-compliant, and GCC is being just hypersensitive here, casting a function pointer to void * isn't, though it is a common extension (http://port70.net/~nsz/c/c11/n1570.html#J.5.7). Pedantically the correct way is to cast to a function pointer with no prototype (empty parentheses) and from that to the target type. See for example. See for example https://godbolt.org/g/FdPdUj |
This is one way that the gcc diagnostics explicitly allow in addition to variable arguments like so: https://godbolt.org/g/Dtb4fv |
|
There are still quite a few similar warnings on git-master. Do they indicate the same problem or should I open new issues? For example: gcc -pthread -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -I. -I./Include -DPy_BUILD_CORE -o Objects/bytearrayobject.o Objects/bytearrayobject.c I'm using GCC 8.1.0 on Arch Linux. |
Thanks for raising this point Antti. This is hinted at by the gcc 8 documentation on -Wcast-function-type: "The function type void (*) (void) is special and matches everything, which can be used to suppress this warning" [1]. One cannot use empty parentheses though as this raises -Wstrict-prototypes on Python builds with gcc and -Wcast-function-type with gcc 8. So (void *) may be replaced with (void (*) (void)) in my previous code snippets and also in PR 6748 and PR 6749 (I just checked my code). [1] https://gcc.gnu.org/onlinedocs/gcc-8.1.0/gcc/Warning-Options.html#Warning-Options |
This is the same issue, the warning line ends with [-Wcast-function-type]. |
|
The problem with invalid function signatures was initially reported in old bpo-1648268. |
|
I marked bpo-34992 as a duplicate of this issue: I use Fedora 28 and gcc (GCC) 8.1.1 20180712 (Red Hat 8.1.1-5) and I get a lot of warnings, with the standard config of Python (./configure) Objects/call.c: In function '_PyMethodDef_RawFastCallDict': .... +- 827 warnings |
I still get something 100 warnings with GCC 8.2.1 on Fedora 29. I wrote PR 10744 to add a macro to cast a function pointer. I chose to add a macro, so it will be easier to spot where we cast function pointers and change the implementation (cast) if needed at a single place. |
|
Only one function cast warning is left, and it is a separate issue: bpo-33015. |
|
I marked bpo-36197 as a duplicate of this issue: """ I am preparing a small PR for this issue. => PR 12179 |
|
Looking at: This is not fixing the underlying issue but hiding it. The right fix would be to use a union for ml_meth providing members for the 3 different function. So the developer could assign them correctly and the compiler would warn if he would do something wrong. Casting to (void *) is just hiding the problem not fixing it! |
|
Yes, and this particular case was fixed in adfffc7. |
|
And how do you deal with METH_VARARGS|METH_KEYWORDS functions which have 3 arguments? PyObject* myfunc(PyObject *py_obj, PyObject *args, PyObject *kwargs) |
|
FYI I modified SystemError("bad call flags") error message to include the method name in bpo-39884. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: