Python 3.13: Traceback during execution of tests

[penguinpee]

Fedora is preparing for the upcoming release of Python 3.13. Executing the tests with Python 3.13.0b2 leads to tracebacks:

Usage: python auto_diff.py [ncpus]
    [ncpus] - the number of workers to run in parallel,
    if omitted it will be set to the number of processors in the system
Starting pp with 3 workers
A fatal error has occured during the function execution
Traceback (most recent call last):
  File "/builddir/build/BUILD/python-ppft-1.7.6.8-build/BUILDROOT/usr/lib/python3.13/site-packages/ppft/__main__.py", line 95, in run
    __args = pickle.loads(ppc.b_(__sargs))
AttributeError: Can't get attribute 'PartialSum' on <module 'ppft.__main__' from '/builddir/build/BUILD/python-ppft-1.7.6.8-build/BUILDROOT/usr/lib/python3.13/site-packages/ppft/__main__.py'>
 Traceback (most recent call last):
  File "/builddir/build/BUILD/python-ppft-1.7.6.8-build/BUILDROOT/usr/lib/python3.13/site-packages/ppft/tests/auto_diff.py", line 124, in <module>
    print("t_log(%lf) = %lf, t_log'(%lf) = %lf" % (x, val.x, x, val.dx))
                                                      ^^^^^
AttributeError: 'NoneType' object has no attribute 'x'

I've also observed:

Usage: python quicksort.py [ncpus]
    [ncpus] - the number of workers to run in parallel, 
    if omitted it will be set to the number of processors in the system

Starting pp with 8 workers
A fatal error has occured during the function execution
Traceback (most recent call last):
  File "/builddir/build/BUILD/python-ppft-1.7.6.8-build/BUILDROOT/usr/lib/python3.13/site-packages/ppft/__main__.py", line 97, in run
    __f = locals()[ppc.str_(__fname)]
          ~~~~~~~~^^^^^^^^^^^^^^^^^^^
KeyError: 'quicksort'
 Traceback (most recent call last):
  File "/builddir/build/BUILD/python-ppft-1.7.6.8-build/BUILDROOT/usr/lib/python3.13/site-packages/ppft/tests/quicksort.py", line 59, in <module>
    output.extend(x())
    ~~~~~~~~~~~~~^^^^^
TypeError: 'NoneType' object is not iterable

[mmckerns]

Thanks for submitting the issue. I am aware of it. There's some issues that changes in python 3.13 starting around 0a5 have impacted dill as python has made some significant changes to core infrastructure to enable multi-threading.

[penguinpee]

With uqfoundation/dill/pull/679 applied we are still seeing the issues from the second codeblock above. I try to gather some more information the coming days.

[mmckerns]

Yes, I'm seeing this as well. This is the next ticket on my schedule, and it's the only blocker to a new release. I expect to be able to get into this week or weekend, and after a few more minor tickets then produce releases. I've seen these kinds of errors before, and it's often due to some change in python that causes a module dict / global dict to not be found.

[mmckerns]

The issue is due to changes with exec and locals: python/cpython#118888
For example, if we have:

# file: test_source.py
def add(x,y):
  return x+y

Then:

Python 3.12.6 (main, Sep  7 2024, 06:27:43) [Clang 13.1.6 (clang-1316.0.21.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill.source as ds
>>> import test_source as ts
>>> def doit(x,y):
...   exec(ds.getsource(ts.add))
...   return locals()['add'](x,y)
... 
>>> doit(1,2)
3

however:

Python 3.13.0rc2 (main, Sep  7 2024, 12:20:37) [Clang 13.1.6 (clang-1316.0.21.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill.source as ds
>>> import test_source as ts
>>> def doit(x,y):
...     exec(ds.getsource(ts.add))
...     return locals()['add'](x,y)
...     
>>> doit(1,2)
Traceback (most recent call last):
  File "<python-input-17>", line 1, in <module>
    doit(1,2)
    ~~~~^^^^^
  File "<python-input-16>", line 3, in doit
    return locals()['add'](x,y)
           ~~~~~~~~^^^^^^^
KeyError: 'add'

Basically, the KeyError observed in ppft is from the same type of usage of locals.

[mmckerns]

While it's a bit more complex inside of ppft, the fix should be something like this:

Python 3.13.0rc2 (main, Sep  7 2024, 12:20:37) [Clang 13.1.6 (clang-1316.0.21.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill.source as ds
>>> import test_source as ts
>>> def doit(x,y):
...     local = {}
...     exec(ds.getsource(ts.add), {}, local)
...     return local[ds.getname(ts.add)](x,y)
...     
>>> doit(1,2)
3