Using non-ascii that require UTF-8 breaks AIX testing

[aixtools]

BPO	41401
Nosy	@vstinner, @serhiy-storchaka, @aixtools
PRs	bpo-41401: Fix test_fspath_support in test_io. #21640 [3.9] bpo-41401: Fix test_fspath_support in test_io. (GH-21640) #22133 [3.8] bpo-41401: Fix test_fspath_support in test_io. (GH-21640) (GH-22133) #22135 [3.8] bpo-41401: Fix test_fspath_support in test_io. (GH-21640) #24024

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-09-07.15:28:45.682>
created_at = <Date 2020-07-26.14:07:03.898>
labels = ['3.10', 'type-bug', 'tests', '3.9', 'expert-IO']
title = 'Using non-ascii that require UTF-8 breaks AIX testing'
updated_at = <Date 2020-12-31.12:45:33.561>
user = 'https://github.com/aixtools'

bugs.python.org fields:

activity = <Date 2020-12-31.12:45:33.561>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2020-09-07.15:28:45.682>
closer = 'vstinner'
components = ['Tests', 'IO']
creation = <Date 2020-07-26.14:07:03.898>
creator = 'Michael.Felt'
dependencies = []
files = []
hgrepos = []
issue_num = 41401
keywords = ['patch']
message_count = 13.0
messages = ['374312', '374316', '374343', '374377', '374379', '374406', '374409', '374598', '375598', '376497', '376499', '376502', '376503']
nosy_count = 4.0
nosy_names = ['vstinner', 'serhiy.storchaka', 'David.Edelsohn', 'Michael.Felt']
pr_nums = ['21640', '22133', '22135', '24024']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue41401'
versions = ['Python 3.9', 'Python 3.10']

[aixtools]

bpo-41069 introduces tests for paths/files containing non-ascii characters.

On AIX - since the merge of PR21035 and PR21156 - the bots have been broken, i.e., returning test failed.

commit 700cfa8
Author: Serhiy Storchaka <storchaka@gmail.com>
Date: Thu Jun 25 17:56:31 2020 +0300

bpo-41069: Make TESTFN and the CWD for tests containing non-ascii characters. (GH-21035)

commit f925407
Author: Serhiy Storchaka <storchaka@gmail.com>
Date: Thu Jun 25 20:39:12 2020 +0300

[3.9] bpo-41069: Make TESTFN and the CWD for tests containing non-ascii characters. (GH-21035). (GH-21156)

(cherry picked from commit 700cfa8c90a90016638bac13c4efd03786b2b2a0)

++++++++

Sadly, I cannot determine - exactly - where it is going wrong as the verbose results ends (says SUCCESS, but there is an ENV change, so bot says FAILED) as:

----------------------------------------------------------------------

Ran 614 tests in 59.122s

OK (skipped=8)
Warning -- files was modified by test_io
Before: []
After: ['@test_23134518_tmpæ']
test_io failed (env changed) in 59.7 sec

== Tests result: SUCCESS ==

1 test altered the execution environment:
test_io

Total duration: 59.8 sec

[serhiy-storchaka]

Thank you for your report (and for all other heroic work for maintaining AIX). Breaking only one test on AIX looks like a good news.

Do you able to run tests manually? Could you please run the following command?

./python -m test --list-cases -uall test_io | xargs -n 1 ./python -m test -vuall test_io -m

It may help to determine what exactly the test was broken.

[aixtools]

Neat! extra arguments!!

The warnings - extracted:

== CPython 3.10.0a0 (heads/master-dirty:b1a8730, Jul 26 2020, 14:00:34) [GCC 7.2.0]
== AIX-2-00F9C1964C00-powerpc-32bit big-endian
== cwd: /home/aixtools/cpython/cpython-master/build/test_python_27984450▒
== CPU count: 32
== encodings: locale=ISO8859-1, FS=iso8859-1
0:00:00 Run tests sequentially
0:00:00 [1/1] test_io
test_fspath_support (test.test_io.CIOTest) ... ok

----------------------------------------------------------------------

Ran 1 test in 0.026s

OK
Warning -- files was modified by test_io
Before: []
After: ['@test_27984450_tmpæ']
test_io failed (env changed)

+++++
== CPython 3.10.0a0 (heads/master-dirty:b1a8730, Jul 26 2020, 14:00:34) [GCC 7.2.0]
== AIX-2-00F9C1964C00-powerpc-32bit big-endian
== cwd: /home/aixtools/cpython/cpython-master/build/test_python_28246552▒
== CPU count: 32
== encodings: locale=ISO8859-1, FS=iso8859-1
0:00:00 Run tests sequentially
0:00:00 [1/1] test_io
test_fspath_support (test.test_io.PyIOTest) ... ok

----------------------------------------------------------------------

Ran 1 test in 0.026s

OK
Warning -- files was modified by test_io
Before: []
After: ['@test_28246552_tmpæ']
test_io failed (env changed)

== Tests result: SUCCESS ==

1 test altered the execution environment:
test_io

++++

== CPython 3.10.0a0 (heads/master-dirty:b1a8730, Jul 26 2020, 14:00:34) [GCC 7.2.0]
== AIX-2-00F9C1964C00-powerpc-32bit big-endian
== cwd: /home/aixtools/cpython/cpython-master/build/test_python_33620404▒
== CPU count: 32
== encodings: locale=ISO8859-1, FS=iso8859-1
0:00:00 Run tests sequentially
0:00:00 [1/1] test_io
test_reader_writer_close_error_on_close (test.test_io.CBufferedRWPairTest) ... ok

----------------------------------------------------------------------

Ran 1 test in 0.001s

OK
Warning -- Unraisable exception
Exception ignored in: <_io.BufferedRWPair object at 0x30689750>
Traceback (most recent call last):
  File "/home/aixtools/cpython/cpython-master/Lib/test/support/__init__.py", line 613, in gc_collect
    gc.collect()
ValueError: flush of closed file
Warning -- Unraisable exception
Exception ignored in: <_io.BufferedWriter>
Traceback (most recent call last):
  File "/home/aixtools/cpython/cpython-master/Lib/test/support/__init__.py", line 613, in gc_collect
    gc.collect()
ValueError: flush of closed file
test_io failed (env changed)

== Tests result: SUCCESS ==

1 test altered the execution environment:
test_io

Hope this helps!

[serhiy-storchaka]

There are two errors in test_io, both are not related to AIX.

The first one is in test_fspath_support. It is exposed on non-UTF-8 locales. AIX by accident use ISO8859-1 locale by default.

The second one is in test_reader_writer_close_error_on_close and related to garbage collector. Seems it is only exposed when run this test separately.

[aixtools]

Excellent!!

aixtools@gcc119:[/home/aixtools/cpython/cpython-master]git pr 21640
remote: Enumerating objects: 50, done.
remote: Counting objects: 100% (50/50), done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 58 (delta 46), reused 48 (delta 46), pack-reused 8
Unpacking objects: 100% (58/58), done.
From https://github.com/python/cpython
 * [new ref]         refs/pull/21640/head -> pr_21640

aixtools@gcc119:[/home/aixtools/cpython/cpython-master]./python -m test
test_io
0:00:00 Run tests sequentially
0:00:00 [1/1] test_io
test_io passed in 1 min

== Tests result: SUCCESS ==

1 test OK.

Total duration: 1 min
Tests result: SUCCESS

On 27/07/2020 16:16, Serhiy Storchaka wrote:

Change by Serhiy Storchaka <storchaka+cpython@gmail.com>:

----------
keywords: +patch
pull_requests: +20781
stage: -> patch review
pull_request: #21640

---

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue41401\>

---

[serhiy-storchaka]

New changeset 67987ac by Serhiy Storchaka in branch 'master':
bpo-41401: Fix test_fspath_support in test_io. (GH-21640)
67987ac

[serhiy-storchaka]

Automatic backport does not work due to changes in the test.support module. Victor, do you mind to backport PR 21640 to 3.9 and 3.8?

[aixtools]

The 'master' branch bot is working again, the 3.9 branch is still broken, and the 3.8 branch seems, as yet, unaffected.

[aixtools]

As much as I wish I had the skills to do the cherry picking - I am not going to touch this.

The AIX bots for 3.9 branch continue to report broken for test_io (ENV change) - as they still wait for the backport for that branch!

[vstinner]

Automatic backport does not work due to changes in the test.support module. Victor, do you mind to backport PR 21640 to 3.9 and 3.8?

Why don't you want to do backport yourself? It's a trivial change, it only changes a single line.

Anyway, I created PR 22133.

[vstinner]

New changeset c73ee5a by Victor Stinner in branch '3.9':
bpo-41401: Fix test_fspath_support in test_io. (GH-21640) (GH-22133)
c73ee5a

[vstinner]

New changeset 84105cb by Victor Stinner in branch '3.8':
bpo-41401: Fix test_fspath_support in test_io. (GH-21640) (GH-22133) (GH-22135)
84105cb

[vstinner]

It should now be fixed. Thanks.