test_ctypes fails on AIX with xlc #71830
Comments
|
I am preparing a system with gcc to see if it is compiler related, i.e., goes away with gcc. On the one hand, fingers crossed - but on the other, having bitfields working regardless of the compiler should be preferred. This issue is similar to a Solaris (C compiler) http://bugs.python.org/issue16275 During the build using clc _ get these messages - which make me hope it is an unexpected compiler issue. "/data/prj/aixtools/python/python-2.7.12.1/Modules/_ctypes/_ctypes_test.c", line 382.5: 1506-159 (E) Bit field type specified for M is not valid. Type unsigned assumed. Note: I modified test_bitfields to do an additional print. Maybe this gives an idea about what the issue is (when considering the compiler messages) root@x064:[/data/prj/aixtools/python/python-2.7.12.1/Lib/ctypes/test]../../../python ./runtests.py Traceback (most recent call last):
File "/data/prj/aixtools/python/python-2.7.12.1/Lib/ctypes/test/test_bitfields.py", line 41, in test_ints
self.assertEqual((name, i, getattr(b, name)), (name, i, func(byref(b), name)))
AssertionError: Tuples differ: ('A', 1, -1) != ('A', 1, 1)First differing element 2:
+ ('A', 1, 1) ====================================================================== Traceback (most recent call last):
File "/data/prj/aixtools/python/python-2.7.12.1/Lib/ctypes/test/test_bitfields.py", line 48, in test_shorts
self.assertEqual((name, i, getattr(b, name)), (name, i, func(byref(b), name)))
AssertionError: Tuples differ: ('M', 1, -1) != ('M', 1, 1)First differing element 2:
+ ('M', 1, 1) ---------------------------------------------------------------------- FAILED (failures=2) |
|
FYI: similar (exact) results when 64-bit mode: Traceback (most recent call last):
File "/data/prj/aixtools/python/python-2.7.12.1/Lib/ctypes/test/test_bitfields.py", line 41, in test_ints
self.assertEqual((name, i, getattr(b, name)), (name, i, func(byref(b), name)))
AssertionError: Tuples differ: ('A', 1, -1) != ('A', 1, 1)First differing element 2:
+ ('A', 1, 1) ====================================================================== Traceback (most recent call last):
File "/data/prj/aixtools/python/python-2.7.12.1/Lib/ctypes/test/test_bitfields.py", line 48, in test_shorts
self.assertEqual((name, i, getattr(b, name)), (name, i, func(byref(b), name)))
AssertionError: Tuples differ: ('M', 1, -1) != ('M', 1, 1)First differing element 2:
+ ('M', 1, 1) ---------------------------------------------------------------------- FAILED (failures=2) |
|
So, it seems to be a compiler issue - when built using gcc (v4.7.4) the tests take a bit longer, but no failures. root@x065:[/data/prj/aixtools/python/python-2.7.12.1/Lib/ctypes/test]../../../python runtests.py OK |
|
The test seems to be comparing ctypes and native C bit-field structures, c_int and c_short versus native int and short. In general in C, if a bit-field has type “int” without a signed or unsigned qualifier, it is up to the implementation which mode is chosen. This is different from normal, where int always means signed int. Since ctypes documents c_int as both meaning “int” and “signed int”, I would say that a ctypes bit-field using c_int should always behave as “signed int”. So perhaps we should just change the test structure to use “signed int” and “signed short”. That should also help avoid the warnings, and may help the Solaris situation (if that is still relevant). I presume this also affects Python 3. |
I presume this as well - however, harder to verify... michael@x071:[/data/prj/aixtools/python/python-3.6.0.162/Lib/ctypes/test]../../../python test_bitfield.py
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Failed to import the site module
Traceback (most recent call last):
File "/data/prj/aixtools/python/python-3.6.0.162/Lib/ctypes/test/../../../Lib/site.py", line 571, in <module>
main()
File "/data/prj/aixtools/python/python-3.6.0.162/Lib/ctypes/test/../../../Lib/site.py", line 557, in main
known_paths = addusersitepackages(known_paths)
File "/data/prj/aixtools/python/python-3.6.0.162/Lib/ctypes/test/../../../Lib/site.py", line 281, in addusersitepackages
user_site = getusersitepackages()
File "/data/prj/aixtools/python/python-3.6.0.162/Lib/ctypes/test/../../../Lib/site.py", line 257, in getusersitepackages
user_base = getuserbase() # this will also set USER_BASE
File "/data/prj/aixtools/python/python-3.6.0.162/Lib/ctypes/test/../../../Lib/site.py", line 247, in getuserbase
USER_BASE = get_config_var('userbase')
File "/data/prj/aixtools/python/python-3.6.0.162/Lib/sysconfig.py", line 587, in get_config_var
return get_config_vars().get(name)
File "/data/prj/aixtools/python/python-3.6.0.162/Lib/sysconfig.py", line 536, in get_config_vars
_init_posix(_CONFIG_VARS)
File "/data/prj/aixtools/python/python-3.6.0.162/Lib/sysconfig.py", line 408, in _init_posix
from _sysconfigdata import build_time_vars
ImportError: No module named '_sysconfigdata'I guess I will have to install it, rather than only work from a test directory.
I believe int means signed on AIX as well - as the comment from the compiler is that type "unsigned" is assumed for a bitfield. Note: the single line of code is: And from your comments I am wondering if the fix is 'simple' - to: +382 unsigned short M: 1, N: 2, O: 3, P: 4, Q: 5, R: 6, S: 7; I can at least try that. Looking at Python-3.6.0.0a2: the line concerned is identical - so that is the verification (for now) that it is also in Python3 |
|
FYI - after a couple of test compiles, it seems that IBM XLC does not report a message when the size specified is int, unsigned or long (i.e., does not like short). Are you interested in the results using a size other than short - or would that break something else I cannot forsee? |
|
Looks like your Python 3 build is messed up. Maybe it doesn’t like running from a different directory. I would try from the main build directory, and note the test_bitfields has an S: ./python -m unittest -v ctypes.test.test_bitfields What I am suggesting as a fix is to change line 381 from plain “int” to “signed int”, and 382 to “signed short”. I can make a patch later if that will help. Hopefully with these changes the C compiler will use signed integer logic, matching what I believe the ctypes library uses for c_int and c_short. Using a type other than short is not right, because the Python test is explicitly trying to test c_short behaviour. If your compiler does not support “signed short” bitfields, maybe we just have to accept that ctypes supports it even though the compiler doesn’t, and skip the test. |
|
On 30-Jul-16 02:51, Martin Panter wrote:
root@x064:[/data/prj/aixtools/python/python-3.5.1.2]./python -m unittest ====================================================================== Traceback (most recent call last):
File
"/data/prj/aixtools/python/python-3.5.1.2/Lib/ctypes/test/test_bitfields.py",
line 41, in test_ints
self.assertEqual(getattr(b, name), func(byref(b),
name.encode('ascii')))
AssertionError: -1 != 1====================================================================== Traceback (most recent call last):
File
"/data/prj/aixtools/python/python-3.5.1.2/Lib/ctypes/test/test_bitfields.py",
line 48, in test_shorts
self.assertEqual(getattr(b, name), func(byref(b),
name.encode('ascii')))
AssertionError: -1 != 1Ran 19 tests in 0.013s FAILED (failures=2)
root@x064:[/data/prj/aixtools/python/python-2.7.12.0]./python -m ====================================================================== Traceback (most recent call last):
File
"/data/prj/aixtools/python/python-2.7.12.0/Lib/ctypes/test/test_bitfields.py",
line 48, in test_shorts
self.assertEqual((name, i, getattr(b, name)), (name, i,
func(byref(b), name)))
AssertionError: Tuples differ: ('M', 1, -1) != ('M', 1, 1)First differing element 2:
+ ('M', 1, 1) ---------------------------------------------------------------------- FAILED (failures=1)
skipping test - only if compiler is not gcc, as when using gcc, test
|
|
Okay, so to be clear, I am assuming XLC supports all of the following fields, and uses unsigned bit fields by default: struct UNSIGNED_BITS {
unsigned int AU: 1;
int A: 1; /* Equivalent to unsigned int */
signed int AS: 1;
unsigned short MU: 1;
short M: 1; /* Equivalent to unsigned short; triggers warning */
};and that it cannot compile the following: struct SIGNED_BITS {
signed short MS: 1; /* Not supported */
};Attached is what I think a patch to resolve this would look like. However it needs a line in Modules/_ctypes/_ctypes_test.c completed to detect the compiler: #ifndef /* Something to identify XLC */Can you figure out a way to test for XLC (but not GCC, which the AIX buildbot uses), and then try my patch out? Hopefully you see no more compiler warnings, test_ints() should now pass, and test_shorts() should be skipped. |
|
Had some flooding (leaking pipes, rather values) issues to fix. Will On 04-Aug-16 10:58, Martin Panter wrote:
|
|
On 8/4/2016 10:58 AM, Martin Panter wrote:
Was not as simple as it sounds. I could not find a specific parameter
What I am thinking of now is #if defined(__GNUC__) || !defined(_AIX)
... #endif As this will pass on AIX when GCC is used, but only be skipped on AIX +++++++++++++++++++++++++++++++++++++++++++++++++++++ More TESTS ++++ Summary - maybe the problem is with the "byref()" routine... ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ My initial test of the patch confused me, so I went back to the original I have three partitions I am testing on - x064 (AIX 5.3 TL7, vac v.11); My working directory is /data/prj/aixtools/python/python-2.7.12.0 which The first builds - no changes - expect to test_bitfields.py: ##for n in "ABCDEFGHIMNOPQRS":
## print n, hex(getattr(BITS, n).size), getattr(BITS, n).offset
for n in "ABCDEFGHIMNOPQRS":
b = BITS()
print n, hex(getattr(BITS, n).size), getattr(BITS, n).offset,
func(byref(b), n)x065: root@x067:/data/prj/aixtools/python/python-2.7.12.0# ./python OK root@x064:[/data/prj/aixtools/python/python-2.7.12.0]./python Traceback (most recent call last):
File "Lib/ctypes/test/test_bitfields.py", line 45, in test_ints
self.assertEqual((name, i, getattr(b, name)), (name, i,
func(byref(b), name)))
AssertionError: Tuples differ: ('A', 1, -1) != ('A', 1, 1)First differing element 2:
+ ('A', 1, 1) ====================================================================== Traceback (most recent call last):
File "Lib/ctypes/test/test_bitfields.py", line 52, in test_shorts
self.assertEqual((name, i, getattr(b, name)), (name, i,
func(byref(b), name)))
AssertionError: Tuples differ: ('M', 1, -1) != ('M', 1, 1)First differing element 2:
+ ('M', 1, 1) ---------------------------------------------------------------------- FAILED (failures=2) Changed the test again to include: root@x064:[/data/prj/aixtools/python/python-2.7.12.0]./python Traceback (most recent call last):
File "Lib/ctypes/test/test_bitfields.py", line 45, in test_ints
self.assertEqual((name, i, getattr(b, name)), (name, i,
func(byref(b), name)))
AssertionError: Tuples differ: ('A', 1, -1) != ('A', 1, 1)First differing element 2:
+ ('A', 1, 1) ====================================================================== Traceback (most recent call last):
File "Lib/ctypes/test/test_bitfields.py", line 52, in test_shorts
self.assertEqual((name, i, getattr(b, name)), (name, i,
func(byref(b), name)))
AssertionError: Tuples differ: ('M', 1, -1) != ('M', 1, 1)First differing element 2:
+ ('M', 1, 1) ---------------------------------------------------------------------- FAILED (failures=2) So, I am wondering if the problem could be better repaired in And changed the test again:
class C_Test(unittest.TestCase):
def test_ints(self):
for i in range(5):
for name in "ABCDEFGHI":
b = BITS()
setattr(b, name, i)
print "name:", name, "range:", i, "getattr():",
getattr(b,name), "byref():", func(byref(b), name)
# self.assertEqual((name, i, getattr(b, name)), (name,
i, func(byref(b), name)))
def test_shorts(self):
for i in range(5):
for name in "MNOPQRS":
b = BITS()
setattr(b, name, i)
print "name:", name, "range:", i, "getattr():",
getattr(b,name), "byref():", func(byref(b), name)
# self.assertEqual((name, i, getattr(b, name)), (name,
i, func(byref(b), name)))And for most tests (remember the range is 5 now) - all is fine root@x064:[/data/prj/aixtools/python/python-2.7.12.0]./python Back to x065... root@x065:[/data/prj/aixtools/python/python-2.7.12.0]./python Lib OK Now looking at the results - it seems the byref() function is where the Hope this helps! |
|
Michael, byref() is just a helper for passing an object’s address to a C function. Calling func(byref(b), ...) in Python is equivalent to the C code unpack_bitfields(&b, ...)I still think the root problem is in unpack_bitfields(). When compiled with XLC, your experiments confirm that it always returns unsigned values, and with GCC, it returns signed values. It looks like you can detect XLC with the __IBMC__ and __xlC__ macros (not sure if there is a practical difference). So perhaps we can fix this with #ifndef __xlC__ |
|
I'll get to this asap. Have to install a new "clean" environment aka a On 18-Aug-16 02:43, Martin Panter wrote:
|
|
I came across this issue while researching where to post my patch (having come across this while building Python 2.7 & 3.x on AIX via xlc).
Unfortunately, the ISO C standard leaves it up to the compiler to decide whether to default to 'signed' or 'unsigned' for non-qualified bit-field declarations. gcc defaults to signed (https://gcc.gnu.org/onlinedocs/gcc-3.3.6/gcc/Non_002dbugs.html); however, xlc defaults to unsigned (https://www.ibm.com/support/knowledgecenter/SSGH2K_13.1.3/com.ibm.xlc1313.aix.doc/compiler_ref/opt_bitfields.html).
However, ctypes_test assumes an unqualified bit-field will be signed. To achieve bit-fields working regardless of the compiler, declarations must explicitly qualifying the sign for bit-fields. This makes the intent of what is expected explicit and avoids implementation-defined behavior that will differ from one compiler to the next. With patch ctypes_test_sign_bitfields.diff provided, I have verified ctypes_test passes on Python 2.7, Python 3.4, and Python 3.5 (on AIX & Solaris). If need be, I'm happy to provide before-&-after output of ctypes_test with & without the patch applied (or verification from others would be greatly appreciated). |
|
On 22-Aug-16 19:43, Eric N. Vander Weele wrote:
|
Oops - this was an oversight when I created the patch. I just uploaded ctypes_test_sign_bitfields_2.diff, which is what I originally intended. |
|
Now I am confused. In <https://bugs.python.org/issue27643#msg271773\> we have [Me] If your compiler does not support “signed short” bitfields, maybe we just have to accept that ctypes supports it even though the compiler doesn’t, and skip the test. [Michael] Looks like this may be the approach: as signed int, signed short give the same message - it seems that xlc (and maybe Sun C) does not accept "short" as a bitfield type. Apparently XLC doesn’t accept signed short bitfields for Michael, but does for Eric. What’s going on? Maybe different versions? |
|
Using: root@x064:[/data/prj/aixtools/python/tests]cat -n bits.c Using: root@x064:[/data/prj/aixtools/python/tests]xlc -qversion I get: root@x064:[/data/prj/aixtools/python/tests]xlc -c bits.c On 23-Aug-16 01:06, Martin Panter wrote:
I do not understand what you mean by "supports" - yes, it can be
|
|
On 23-Aug-16 21:23, Michael Felt wrote:
|
|
I am able to replicate what Michael has provided (i.e., xlc does not support signed short). Sorry for the confusion: I overlooked that the compiler is emitting the error with may patch and assuming 'unsigned'. So it seems like there are two things to address: (1) The bit fields should be explicitly marked as 'signed', since that appears to be the desired intent. (2) What to do about the test case. It seems like we all agree on (1). For (2), is this something that should be stubbed out on AIX/xlc, resolved by xlc for supporting implementation-defined (short) bit-fields, or remove the short members in the struct since C99 (6.7.2.1) allows "a qualified or unqualified version of _Bool, signed int, unsigned int, or some other implementation-defined type"; thus removing the ambiguity for implementation-defined behavior? |
|
Michael: When posting to the bug tracker, please trim irrelevant parts of old messages. It makes it hard to see if you actually added anything new.
Proper support for “signed short” according to standard C I guess would mean if you define struct BITS {
signed short M: 1;
} b;
b.M = -1;then reading back b.M gives -1. However I realized test_bitfields tests overflowing values rather than negative values. In any case, I think we have established that neither of these cases work with XLC. I haven’t changed the signed int A–I fields yet. That was part of my patch. I was waiting for confirmation about the __xlC__ check, before committing the whole thing. Eric: I proposed to conditionally skip the test; see disable-signed-short.patch. Since many other compilers apparently pass the test and support signed short, we should probably keep the test. |
|
As I was not responding properly (too verbose) - I'll reread the thread for the initial patch suggestion - and hope it still fits the current 'master'. My apologies for the long silence on this. |
|
On 30/07/2016 02:51, Martin Panter wrote:
As some time has passed, I assume you mean look at this change: struct BITS {
signed int A: 1, B:2, C:3, D:4, E: 5, F: 6, G: 7, H: 8, I: 9;
signed short M: 1, N: 2, O: 3, P: 4, Q: 5, R: 6, S: 7;
};
Not helping the compilation in any case - same messages as before.
michael@x071:[/data/prj/python/git/xlc-python3-3.7]xlc_r -DNDEBUG -O
michael@x071:[/data/prj/python/git/xlc-python3-3.7]xlc_r
|
|
On 04/08/2016 10:58, Martin Panter wrote:
I'll figure this out. HOWEVER, Just want to mention that both parts of (Have not done the compiler detect part yet, so it still returns +33 class C_Test(unittest.TestCase): ====================================================================== Traceback (most recent call last):
File
"/data/prj/python/git/xlc-python3-3.7/Lib/ctypes/test/test_bitfields.py",
line 40, in test_ints
self.assertEqual(getattr(b, name), func(byref(b),
name.encode('ascii')))
AssertionError: -1 != 1====================================================================== Traceback (most recent call last):
File
"/data/prj/python/git/xlc-python3-3.7/Lib/ctypes/test/test_bitfields.py",
line 51, in test_shorts
self.assertEqual(getattr(b, name), func(byref(b),
name.encode('ascii')))
AssertionError: -1 != 1
|
|
I know it is not earth shattering - but it will permit one more test to pass. Please review the PR. Thx. |
|
Again, the PR worked months ago. I expect it to still work. So, I guess the question is: is there any interest. After several weeks of waiting after the last ttt - still waiting :) |
|
well, update: the bpo-34603 merged 16 days ago has broken this PR - that has been waiting for nearly 10 months. Unhappy camper. And, just as a short reminder - there were earlier ¨patches (that I just copied) going back more than 2 years. Please, some attention. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: