Python 3.12 test failures

[stratakis]

Currently in Fedora we are rebuilding twisted with Python 3.12 beta 3 and there are various test failures that we've observed.

First of all test_log hangs on test_startLoggingOverridesWarning.

By removing test_log I was able to reach 5 test failures and 41 test errors. Full build log is attached.

Sample:

[FAIL]
Traceback (most recent call last):
  File "/builddir/build/BUILD/twisted-twisted-22.10.0/build/lib/twisted/trial/test/test_warning.py", line 180, in test_warningsConfiguredAsErrors
    self.assertEqual(len(result.errors), 1)
  File "/builddir/build/BUILD/twisted-twisted-22.10.0/build/lib/twisted/trial/_synctest.py", line 441, in assertEqual
    super().assertEqual(first, second, msg)
  File "/usr/lib64/python3.12/unittest/case.py", line 885, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/lib64/python3.12/unittest/case.py", line 878, in _baseAssertEqual
    raise self.failureException(msg)
twisted.trial.unittest.FailTest: 5 != 1

[ERROR]
Traceback (most recent call last):
  File "/builddir/build/BUILD/twisted-twisted-22.10.0/build/lib/twisted/test/test_unix.py", line 345, in test_cannotListen
    s = reactor.listenUNIXDatagram(addr, p)
  File "/builddir/build/BUILD/twisted-twisted-22.10.0/build/lib/twisted/internet/posixbase.py", line 285, in listenUNIXDatagram
    p.startListening()
  File "/builddir/build/BUILD/twisted-twisted-22.10.0/build/lib/twisted/internet/udp.py", line 184, in startListening
    self._bindSocket()
  File "/builddir/build/BUILD/twisted-twisted-22.10.0/build/lib/twisted/internet/unix.py", line 516, in _bindSocket
    raise error.CannotListenError(None, self.port, le)
twisted.internet.error.CannotListenError: Couldn't listen on any:/builddir/build/BUILD/twisted-twisted-22.10.0/_trial_temp/twisted.test.test_unix/DatagramUnixSocketTests/test_cannotListen/us0ykj97/temp: AF_UNIX path too long.

[ERROR]
Traceback (most recent call last):
  File "/builddir/build/BUILD/twisted-twisted-22.10.0/build/lib/twisted/test/test_sob.py", line 38, in testStyles
    p.save(filename="persisttest." + style)
  File "/builddir/build/BUILD/twisted-twisted-22.10.0/build/lib/twisted/persisted/sob.py", line 99, in save
    self._saveTemp(filename, dumpFunc)
  File "/builddir/build/BUILD/twisted-twisted-22.10.0/build/lib/twisted/persisted/sob.py", line 72, in _saveTemp
    dumpFunc(self.original, f)
  File "/builddir/build/BUILD/twisted-twisted-22.10.0/build/lib/twisted/persisted/aot.py", line 477, in jellyToSource
    file.write(getSource(aot).encode("utf-8"))
  File "/builddir/build/BUILD/twisted-twisted-22.10.0/build/lib/twisted/persisted/aot.py", line 173, in getSource
    return indentify("app = " + prettify(ao))
  File "/builddir/build/BUILD/twisted-twisted-22.10.0/build/lib/twisted/persisted/aot.py", line 234, in indentify
    for (
  File "/usr/lib64/python3.12/tokenize.py", line 543, in _generate_tokens_from_c_tokenizer
    raise TokenError(msg, (e.lineno, e.offset)) from None
tokenize.TokenError: ('source code cannot contain null bytes', (1, 0))

[ERROR]
Traceback (most recent call last):
  File "/builddir/build/BUILD/twisted-twisted-22.10.0/build/lib/twisted/trial/runner.py", line 616, in loadPackage
    module = modinfo.load()
  File "/builddir/build/BUILD/twisted-twisted-22.10.0/build/lib/twisted/python/modules.py", line 388, in load
    return self.pathEntry.pythonPath.moduleLoader(self.name)
  File "/builddir/build/BUILD/twisted-twisted-22.10.0/build/lib/twisted/python/reflect.py", line 297, in namedAny
    topLevelPackage = _importAndCheckStack(trialname)
  File "/builddir/build/BUILD/twisted-twisted-22.10.0/build/lib/twisted/python/reflect.py", line 244, in _importAndCheckStack
    raise excValue.with_traceback(excTraceback)
  File "/builddir/build/BUILD/twisted-twisted-22.10.0/build/lib/twisted/test/test_rebuild.py", line 9, in <module>
    from twisted.python import rebuild
  File "/builddir/build/BUILD/twisted-twisted-22.10.0/build/lib/twisted/python/rebuild.py", line 16, in <module>
    from imp import reload
builtins.ModuleNotFoundError: No module named 'imp'

build.log

[hroncok]

builtins.ModuleNotFoundError: No module named 'imp'

This one is fixed by 4ba766b86

[stratakis]

There is also a regexp failure which is fixed by 4dabbf4

[stratakis]

Relevant cpython issue for the AF_UNIX path too long: python/cpython#93852

[adiroiban]

Thanks for the report. I see that 3.12.0b2 is available in GitHub Actions.

Not sure about beta3 ... but I think that this can be good enough to test.

I will create a branch and give it a try.

[adiroiban]

Thanks for the feedback.

As expected, the test suite fails on Python 3.12

https://github.com/twisted/twisted/actions/runs/5404100452/jobs/9819104387?pr=11889

I think the first steps in getting py 3.12 support are:

• identifying the tests that fail
• skip those tests on py3
• Create separate, dedicated tickets for each group of tests that are caused by the same rror
• look into fixing each ticket, one by one

I think that in 3.12 and 3.13 some legacy code will be removed, so for Twisted the fix might be deprecating the code using those modules and disabling those tests on 3.12 or newer.

I know for example that Twisted had some code to do shadow password validation.
I think that code should just be deprecated and removed.

Or if anyone is still using that code, they can submit a fix to use PAM or other tools

[itamarst]

Tests seem to be passing in CI, so going to close this.

[adiroiban]

Thanks for the update.
For the record.
Initial work was done in:

• The CI suite was updated to execute the tests using a Python 3.12 pre-release (Enable python 3.12 rc on GitHub Actions #11857)

We used the pre-release, but current trunk uses the final release

twisted/.github/workflows/test.yaml

Line 135 in 157cd8e

- python-version: '3.12'