Please port qiime2 to Python3.12

[tillea]

Improvement Description
When running q2-types test suite under Python3.12 some errors occured that are caused by qiime2s OutPath method.

Current Behavior
When running the Debian packaged q2-types test suite under Python3.12 in our CI there are errors like

_________ TestTransformers.test_skbio_distance_matrix_to_lsmat_format __________
self = <[AttributeError("'OutPath' object has no attribute '_raw_paths'") raised in repr()] OutPath object at 0x7f98bf983e50>
    def __str__(self):
        """Return the string representation of the path, suitable for
        passing to system calls."""
        try:
>           return self._str
E           AttributeError: 'OutPath' object has no attribute '_str'
/usr/lib/python3.12/pathlib.py:441: AttributeError
During handling of the above exception, another exception occurred:
self = <[AttributeError("'OutPath' object has no attribute '_raw_paths'") raised in repr()] OutPath object at 0x7f98bf983e50>
    @property
    def drive(self):
        """The drive prefix (letter or UNC path), if any."""
        try:
>           return self._drv
E           AttributeError: 'OutPath' object has no attribute '_drv'
...

You can see a full log

Proposed Behavior
Since the test suite passes nicely in Python3.11 I assume the problem is caused by some change in Python3.12.

It would be great if qiime could be ported to Python3.12.

Kind regards
Andreas.

[tillea]

If you need more instances of the problem you can also find examples in the q2-cutadapt build log or q2-demux build log

[s3v-]

Hi,
According to python/cpython#113976, patch below for qiime2/core/path.py should fix the problem.

Kind Regards

        self = super().__new__(cls, *args)
-       self._destructor = weakref.finalize(self, self._destruct, str(self))
+       self._destructor = weakref.finalize(self, self._destruct, self._raw_path)
        return self

[tillea]

Hi,
thanks a lot for your hint. I tried to patch:

--- a/qiime2/core/path.py
+++ b/qiime2/core/path.py
@@ -106,7 +106,7 @@ class OutPath(OwnedPath):
             # prevent a resource leak.
             os.close(fd)
         obj = super().__new__(cls, name)
-        obj._destructor = weakref.finalize(obj, cls._destruct, str(obj))
+        obj._destructor = weakref.finalize(obj, cls._destruct, obj._raw_path)
         return obj

     def __exit__(self, t, v, tb):
@@ -126,7 +126,7 @@ class InternalDirectory(_ConcretePath):
     @classmethod
     def __new(cls, *args):
         self = super().__new__(cls, *args)
-        self._destructor = weakref.finalize(self, self._destruct, str(self))
+        self._destructor = weakref.finalize(self, self._destruct, self._raw_path)
         return self

     def __new__(cls, *args, prefix=None):

and checked as an example the dada2 test suite.

$ grep -A2 -B2 _raw_path LOG
cls = <class 'qiime2.core.path.ProvenancePath'>
args = ('/tmp/qiime2-provenance-auqdfbve',)
self = <[AttributeError("'ProvenancePath' object has no attribute '_raw_paths'") raised in repr()] ProvenancePath object at 0x7f40ec441a50>

    @classmethod
    def __new(cls, *args):
        self = super().__new__(cls, *args)
>       self._destructor = weakref.finalize(self, self._destruct, self._raw_path)
E       AttributeError: 'ProvenancePath' object has no attribute '_raw_path'. Did you mean: '_raw_paths'?

/usr/lib/python3/dist-packages/qiime2/core/path.py:129: AttributeError
--
cls = <class 'qiime2.core.path.ProvenancePath'>
args = ('/tmp/qiime2-provenance-auqdfbve',)
self = <[AttributeError("'ProvenancePath' object has no attribute '_raw_paths'") raised in repr()] ProvenancePath object at 0x7f40ec441a50>

    @classmethod
    def __new(cls, *args):
        self = super().__new__(cls, *args)
>       self._destructor = weakref.finalize(self, self._destruct, self._raw_path)
E       AttributeError: 'ProvenancePath' object has no attribute '_raw_path'. Did you mean: '_raw_paths'?

/usr/lib/python3/dist-packages/qiime2/core/path.py:129: AttributeError
--
kwargs = {'prefix': 'q2-DADA2StatsFormat-'}
name = '/tmp/q2-DADA2StatsFormat-_pubr6gh', fd = 11
obj = <[AttributeError("'OutPath' object has no attribute '_raw_paths'") raised in repr()] OutPath object at 0x7f40ec1757d0>

    def __new__(cls, dir=False, **kwargs):
--
            os.close(fd)
        obj = super().__new__(cls, name)
>       obj._destructor = weakref.finalize(obj, cls._destruct, obj._raw_path)
E       AttributeError: 'OutPath' object has no attribute '_raw_path'. Did you mean: '_raw_paths'?

/usr/lib/python3/dist-packaautopkgtest [21:27:15]: test run-unit-test: -----------------------]

So this does not seem to help. Thanks a lot for your research anyway, Andreas.

[s3v-]

Hi @tillea,
Since ProvenancePath class inherit from InternalDirectory class, I've just attempted to fix qiime2's tests. on a basis of changes in pathlib ABC.
I haven't tested other plugins but I'm sure the author has more in-depth knowledge of this mechanism.

Kind Regards.