[AIX] test_math: test_nextafter(float('nan'), 1.0) does not return a NaN on AIX #86489

vstinner · 2020-11-11T11:34:58Z

BPO	42323
Nosy	@malemburg, @rhettinger, @mdickinson, @vstinner, @aixtools
PRs	bpo-42323: Fix math.nextafter() for AIX libc #24265 bpo-42323: Fix math.nextafter() on AIX #24381

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-01-30.00:00:29.184>
created_at = <Date 2020-11-11.11:34:57.991>
labels = ['tests', '3.10']
title = "[AIX] test_math: test_nextafter(float('nan'), 1.0) does not return a NaN on AIX"
updated_at = <Date 2021-01-30.00:00:29.183>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2021-01-30.00:00:29.183>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2021-01-30.00:00:29.184>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2020-11-11.11:34:57.991>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 42323
keywords = ['patch']
message_count = 24.0
messages = ['380751', '380754', '380755', '380756', '380769', '380774', '380778', '381096', '381111', '381115', '381119', '381128', '381164', '381166', '381238', '381260', '381262', '383535', '383548', '385326', '385343', '385348', '385953', '385960']
nosy_count = 7.0
nosy_names = ['lemburg', 'rhettinger', 'mark.dickinson', 'vstinner', 'stutzbach', 'David.Edelsohn', 'Michael.Felt']
pr_nums = ['24265', '24381']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue42323'
versions = ['Python 3.10']

vstinner · 2020-11-11T11:34:57Z

https://buildbot.python.org/all/#/builders/302/builds/338

FAIL: test_nextafter (test.test_math.MathTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/aixtools/buildarea/3.9.aixtools-aix-power6/build/Lib/test/test_math.py", line 1968, in test_nextafter
    self.assertIsNaN(math.nextafter(NAN, 1.0))
  File "/home/aixtools/buildarea/3.9.aixtools-aix-power6/build/Lib/test/test_math.py", line 2015, in assertIsNaN
    self.fail("Expected a NaN, got {!r}.".format(value))
AssertionError: Expected a NaN, got 1.0.

The test:

        # NaN
        self.assertIsNaN(math.nextafter(NAN, 1.0))   # <=== HERE
        self.assertIsNaN(math.nextafter(1.0, NAN))
        self.assertIsNaN(math.nextafter(NAN, NAN))

The Linux manual page says: "If x or y is a NaN, a NaN is returned."
https://man7.org/linux/man-pages/man3/nextafter.3.html

But it seems like the AIX libc doesn't implement this rule. Should we implement this rule in Python on AIX?

The strange thing is that it worked previously. test.python of build 338:

platform.platform: AIX-2-00F9C1964C00-powerpc-32bit
sysconfig[HOST_GNU_TYPE]: powerpc-ibm-aix7.2.4.0
platform.architecture: 32bit

The latest green build is built 347. test.pythoninfo of build 347:

platform.architecture: 32bit
platform.platform: AIX-2-00F9C1964C00-powerpc-32bit
sysconfig[HOST_GNU_TYPE]: powerpc-ibm-aix7.2.0.0

Was the machine updated two days ago (2020-11-09), between build 338 and build 347?

mdickinson · 2020-11-11T12:39:26Z

If AIX were one of our officially supported platforms, then yes, I'd say that we should add a workaround to handle special cases ourselves, similarly to what we already do for a number of math module functions (like math.pow, for example).

But given that it's only a "best effort" platform, I'm not convinced that it's worth the effort or the extra complication in the codebase.

-0 from me, I guess.

mdickinson · 2020-11-11T12:41:05Z

Is there any reasonable channel for reporting the issue upstream?

vstinner · 2020-11-11T13:26:31Z

My worry is that I'm getting emails about AIX buildbot failures. I see different options:

Skip the test on AIX
Fix nextafter() on AIX
Turn off AIX buildbot email notifications
Remove thE AIX buildbot

mdickinson · 2020-11-11T16:31:16Z

My worry is that I'm getting emails about AIX buildbot failures.

That sounds more like a process problem than a CPython codebase one. The ideal would be that the machinery sending those notifications can be configured to ignore known failures when deciding whether to send email. Is that remotely feasible? (I have zero familiarity with the buildbot machinery.)

Skipping the test on AIX sounds like a reasonable option, but I kinda *want* IBM developers running the Python test suite on AIX to see those failures, in the hope that they might then be motivated to push for a fix to the relevant AIX bug. :-)

vstinner · 2020-11-11T16:54:12Z

That sounds more like a process problem than a CPython codebase one. The ideal would be that the machinery sending those notifications can be configured to ignore known failures when deciding whether to send email. Is that remotely feasible? (I have zero familiarity with the buildbot machinery.)

If a test fails all the time and not randomly, a single email is sent at the first failure.

I'm annoyed by test_threading which crash randomly on AIX: https://bugs.python.org/issue40068 It's a known issue, I already fixed 3/4 of the issue, but I didn't fix the remaining part.

In the past, I already disabled AIX email notifications simply because there was nobody to fix issues, and so emails were just spam.

but I kinda *want* IBM developers running the Python test suite on AIX to see those failures, in the hope that they might then be motivated to push for a fix to the relevant AIX bug. :-)

Well, that would be great.

I'm not sure if Michael Felt is still working on supporting AIX in Python. David Edelsohn might help.

DavidEdelsohn · 2020-11-11T17:37:17Z

nextafter is a known problem on AIX. I believe that it is being addressed in newer releases of AIX.

Michael and I are helping the IBM AIX Open Source team to increase their attention on Python, but things only move so fast.

aixtools · 2020-11-16T12:53:47Z

I have been experimenting with different hardware and AIX versions.

When building on AIX 5.3 - and the oldest libraries - test_math passes.

When I run the test on POWER8, using either xlc or gcc test_math fails with just one element of the test.

When I run the test on POWER6 I get many more errors - that I never had before. These are all after OS updates (I was not going to build for AIX 5.3 any more).

An idea I have now - that may explain the sudden change in behavior is if the libraries have been optimized to always use the DFP (decimal floating point) internally - for what, from the application perspective - is the normal - no HW acceleration for FP - interface.

I know there are ways to 'discover' this, but I'll need to write some tests so that I can see - if linking to different libraries actuates DFP performance counters yes and no.

At this point - this feels like the a potential explanation.

DavidEdelsohn · 2020-11-16T15:22:31Z

I investigated another problem with nextafter() in 2015 and opened an internal IBM AIX PMR. At the time it was not using decimal float code.

The earlier problem was the handling of -0.0. At the time, the code was hand-written assembly language that did not check for IEEE floating point corner cases.

vstinner · 2020-11-16T16:20:26Z

The earlier problem was the handling of -0.0. At the time, the code was hand-written assembly language that did not check for IEEE floating point corner cases.

I'm quite happy that my hand written tests detect bugs in nextafter() implementations ;-)

vstinner · 2020-11-16T16:48:09Z

How can we fix the buildbot? Add #ifdef in mathmodule.c to implement the special cases, but only on AIX? Skip the test?

aixtools · 2020-11-16T17:27:33Z

There seems to be a lot of interaction of OS level and compiler used.

Waiting for the next bot run to get a different compiler.

+++
AIX 6.1.6 and older libraries - no test errors reported

AIX 7.1.4 and newer libraries - when using the binary built on 6.1.6 (AIX 6.1 TL6) - no error

AIX 7.1.4, using xlc-v13.1.2 (try and buy version), same AIX level as the bot (7.1 TL4 SP8) - no error

Back to the bot: AIX 7.1 TL4 SP8 and gcc-4.7.4 - strange errors. See, e.g., https://buildbot.python.org/all/#/builders/302/builds/373/steps/5/logs/stdio with additional errors such as:

======================================================================
FAIL: testHypotAccuracy (test.test_math.MathTests) (hx='0x1.89d8c423ea0c6p+29', hy='0x1.d35dcfe902bc3p+29', x=825956484.4892814, y=980138493.1263355)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/test_math.py", line 867, in testHypotAccuracy
    self.assertEqual(hypot(x, y), z)
AssertionError: 1281747081.1271062 != 1281747081.127106

Waiting for bot run 374 - to see if the results change when the compiler changes.

I'll try moving the bot to another system - as the system the bot is on is more than just the bot. Maybe there are side-effects coming in, unexpectedly, from other sources. ** The two systems just mentioned are fresh installs.

vstinner · 2020-11-16T21:31:09Z

Back to the bot: AIX 7.1 TL4 SP8 and gcc-4.7.4

The latest GCC version is GCC 10. Is it still relevant to test GCC 4.7 released 8 years ago? (Well, I'm not sure that the C compiler explains all issues.)

DavidEdelsohn · 2020-11-16T21:33:07Z

I believe that Michael was trying to probe under what circumstances the failure appears. But, not GCC 4.7 is not relevant.

aixtools · 2020-11-17T14:23:42Z

Yes, just probing, the version of gcc is irrelevant.

What I do believe is important is that bot run 374, 375 and 376 passed - On AIX 7.1 TL4 SP8.

The failure starting with 377 is an undefined variable.

"./Modules/posixmodule.c", line 15146.53: 1506-045 (S) Undeclared identifier SPLICE_F_MOVE.
"./Modules/posixmodule.c", line 15147.57: 1506-045 (S) Undeclared identifier SPLICE_F_NONBLOCK.
"./Modules/posixmodule.c", line 15148.53: 1506-045 (S) Undeclared identifier SPLICE_F_MORE.

So, let's put this on hold. I'll get a new environment built up specific for testing python on AIX. Hopefully by Friday.

vstinner · 2020-11-17T17:38:31Z

"./Modules/posixmodule.c", line 15146.53: 1506-045 (S) Undeclared identifier SPLICE_F_MOVE.

This is unrelated: https://bugs.python.org/issue41625#msg381259 Please continue the discussion this SPLICE there.

mdickinson · 2020-11-17T17:45:58Z

[Victor]

How can we fix the buildbot? Add #ifdef in mathmodule.c to implement the special cases, but only on AIX? Skip the test?

I'm not super-keen on using #ifdefs to implement the special-case handling _just_ for AIX: that opens the door to a labyrinth of #ifdef'ery working around various different problems on various different platforms. If we're going to handle special cases ourselves, let's do it for all platforms.

But I'd also be fine with skipping the test (just on AIX, of course) for now. If we can also find a way to remind ourselves to revisit once the upstream bug has been fixed, so much the better.

aixtools · 2020-12-21T15:53:31Z

I have been doing a lot of research on this. Wish I had thought do start the way I finished.

Basically, when math.nextafter() was added all the AIX bots were on systems running AIX earlier than AIX 7.2 TL2.

When AIX 7.2 TL2 was released (roughly Q3 2017) a (major?) change was made to the nextafter() function.

root@gcc119:[/home2/root]instfix -k IV95512 -a
IV95512 Abstract: nextafter(+0.0, -0.0) returns +0.0 instead of -0.0.

IV95512 Symptom Text:
If(x==y) nextafter returns x instead of y.

At first glance - it appears the CPython code is reversing the arguments:

The lines in test_math.py are currently:
+2026 # NaN
+2027 self.assertIsNaN(math.nextafter(NAN, 1.0))
+2028 self.assertIsNaN(math.nextafter(1.0, NAN))
+2029 self.assertIsNaN(math.nextafter(NAN, NAN))

Moving line 2027 (which is what is failing) to 2029 - the other two lines pass on an AIX system with IV95512 applied.

As IEEE754 says (and seems to have always said):

https://pubs.opengroup.org/onlinepubs/9699919799:

If x or y is NaN, a NaN shall be returned.

The current test in Modules/mathmodule.c might be too simple.

I am working on a PR where I check for presence of APAR IV95512 - with the nextafter() changes.

aixtools · 2020-12-21T18:37:22Z

While my patch in working - was successful in what it attempted to do, it did not fix this test issue.

Instead - I reinstalled the bos.adt.libm-7.2.0.0 fileset, to backout of the so-called bugfix/APAR IV95512.

@david - can you take this up with AIX support - IV95112 (and more) do not seem to return NaN when one of the arguments is NaN.

vstinner · 2021-01-20T10:43:56Z

I wrote PR 24265 to fix the issue. math.nextafter(x, y) already had a special case for AIX for x==y.

test_nextafter fails on PPC64 AIX 3.x (build 749).

test_nextafter pass on POWER6 AIX 3.x (build 701).

vstinner · 2021-01-20T14:20:27Z

New changeset c1c3493 by Victor Stinner in branch 'master':
bpo-42323: Fix math.nextafter() for NaN on AIX (GH-24265)
c1c3493

vstinner · 2021-01-20T15:36:16Z

Oh, it seems like Python no long builds on PPC64 AIX 3.x buildbot :-(
https://bugs.python.org/issue42604#msg385347

vstinner · 2021-01-29T22:04:53Z

New changeset 0837f99 by Victor Stinner in branch 'master':
bpo-42323: Fix math.nextafter() on AIX (GH-24381)
0837f99

vstinner · 2021-01-30T00:00:29Z

test_nextafter fails on PPC64 AIX 3.x (build 749).

It pass again in build 788, so I close the issue:
https://buildbot.python.org/all/#/builders/438/builds/788

It would be great if the AIX libm could be fixed, but I wanted to fix the AIX buildbots, to be abl to detect other Python regressions.

vstinner added 3.10 only security fixes tests Tests in the Lib/test dir labels Nov 11, 2020

vstinner closed this as completed Jan 30, 2021

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AIX] test_math: test_nextafter(float('nan'), 1.0) does not return a NaN on AIX #86489

[AIX] test_math: test_nextafter(float('nan'), 1.0) does not return a NaN on AIX #86489

vstinner commented Nov 11, 2020

vstinner commented Nov 11, 2020

mdickinson commented Nov 11, 2020

mdickinson commented Nov 11, 2020

vstinner commented Nov 11, 2020

mdickinson commented Nov 11, 2020

vstinner commented Nov 11, 2020

DavidEdelsohn mannequin commented Nov 11, 2020

aixtools commented Nov 16, 2020

DavidEdelsohn mannequin commented Nov 16, 2020

vstinner commented Nov 16, 2020

vstinner commented Nov 16, 2020

aixtools commented Nov 16, 2020

vstinner commented Nov 16, 2020

DavidEdelsohn mannequin commented Nov 16, 2020

aixtools commented Nov 17, 2020

vstinner commented Nov 17, 2020

mdickinson commented Nov 17, 2020

aixtools commented Dec 21, 2020

aixtools commented Dec 21, 2020

vstinner commented Jan 20, 2021

vstinner commented Jan 20, 2021

vstinner commented Jan 20, 2021

vstinner commented Jan 29, 2021

vstinner commented Jan 30, 2021

[AIX] test_math: test_nextafter(float('nan'), 1.0) does not return a NaN on AIX #86489

[AIX] test_math: test_nextafter(float('nan'), 1.0) does not return a NaN on AIX #86489

Comments

vstinner commented Nov 11, 2020

vstinner commented Nov 11, 2020

mdickinson commented Nov 11, 2020

mdickinson commented Nov 11, 2020

vstinner commented Nov 11, 2020

mdickinson commented Nov 11, 2020

vstinner commented Nov 11, 2020

DavidEdelsohn mannequin commented Nov 11, 2020

aixtools commented Nov 16, 2020

DavidEdelsohn mannequin commented Nov 16, 2020

vstinner commented Nov 16, 2020

vstinner commented Nov 16, 2020

aixtools commented Nov 16, 2020

vstinner commented Nov 16, 2020

DavidEdelsohn mannequin commented Nov 16, 2020

aixtools commented Nov 17, 2020

vstinner commented Nov 17, 2020

mdickinson commented Nov 17, 2020

aixtools commented Dec 21, 2020

aixtools commented Dec 21, 2020

vstinner commented Jan 20, 2021

vstinner commented Jan 20, 2021

vstinner commented Jan 20, 2021

vstinner commented Jan 29, 2021

vstinner commented Jan 30, 2021