Standalone: Resource files obtained by importlib.resources.files(...).iterdir() cannot be sorted

[jhbuhrman]

Problem description:

Another PyPI package, Pyphen, assumes that objects returned from importlib.resources.files(...).iterdir() can be compared using the < operator, since it calls sorted() on the .iterdir() result, but the importlib.resources.abc.Traversable Protocol does not guarantee that objects returend by .iterdir() can be compared using the < operator. See also Kozea/Pyphen#54 for more info. Perhaps of an unfortunate misunderstanding the issue on Pyphen is already closed, while I still think that one should not rely on functionality that appears to be working but is not backed by documentation describing that it should.

Nonetheless the Nuitka developer/maintainer asked me to create an issue on Nuitka, since he wants to stay as compatible as possible with plain Python functionality.

So here we go:

Versions

> python -m nuitka --version
1.7.10
Commercial: None
Python: 3.10.4 (tags/v3.10.4:9d38120, Mar 23 2022, 23:13:41) [MSC v.1929 64 bit (AMD64)]
Flavor: CPython Official
Executable: D:\workspace\projects\nuitka-problem\.venv310\Scripts\python.exe
OS: Windows
Arch: x86_64
WindowsRelease: 10
Version C compiler: C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.37.32822\bin\Hostx64\x64\cl.exe (cl 14.3).

Installation info

> where python
D:\dev\Python310\python.exe
....

Virtual environment

Created with

python -m venv .venv310

Activated with

> .venv310\Scripts\activate
> where python
D:\workspace\projects\nuitka-problem\.venv310\Scripts\python.exe
D:\dev\Python310\python.exe
...

Installed packages

> python -m pip install -U pip
...
> python -m pip install Nuitka
...
> python -m pip freeze
Nuitka==1.7.10
ordered-set==4.1.0
zstandard==0.21.0

Source files

See attached .zip file → implib_sorted_resources_problem.zip

> dir /B/S D:\workspace\projects\nuitka-problem\implib_sorted_resources_problem
D:\workspace\projects\nuitka-problem\implib_sorted_resources_problem\main.py
D:\workspace\projects\nuitka-problem\implib_sorted_resources_problem\some_package
D:\workspace\projects\nuitka-problem\implib_sorted_resources_problem\some_package\handle_resources.py
D:\workspace\projects\nuitka-problem\implib_sorted_resources_problem\some_package\some_resources
D:\workspace\projects\nuitka-problem\implib_sorted_resources_problem\some_package\__init__.py
D:\workspace\projects\nuitka-problem\implib_sorted_resources_problem\some_package\some_resources\file_A.txt
D:\workspace\projects\nuitka-problem\implib_sorted_resources_problem\some_package\some_resources\file_B.txt

Output using plain Python

> cd implib_sorted_resources_problem
> python -m main
WindowsPath('D:/workspace/projects/nuitka-problem/implib_sorted_resources_problem/some_package/some_resources/file_A.txt')
WindowsPath('D:/workspace/projects/nuitka-problem/implib_sorted_resources_problem/some_package/some_resources/file_B.txt')

Nuitka build command

> python -m nuitka --standalone --include-package-data=some_package main.py
Nuitka-Options:INFO: Used command line options: --standalone --include-package-data=some_package main.py
Nuitka:INFO: Starting Python compilation with Nuitka '1.7.10' on Python '3.10' commercial grade 'not installed'.
Nuitka:INFO: Completed Python level compilation and optimization.
Nuitka:INFO: Generating source code for C backend compiler.
Nuitka:INFO: Running data composer tool for optimal constant value handling.
Nuitka:INFO: Running C compilation via Scons.
Nuitka-Scons:INFO: Backend C compiler: cl (cl 14.3).
Nuitka-Scons:INFO: Backend linking program with 9 files (no progress information available for this stage).
Nuitka-Scons:INFO: Compiled 9 C files using clcache with 8 cache hits and 1 cache misses.
Nuitka-Options:INFO: Included data file 'some_package\some_resources\file_A.txt' due to package 'some_package' package data.
Nuitka-Options:INFO: Included data file 'some_package\some_resources\file_B.txt' due to package 'some_package' package data.
Nuitka:INFO: Keeping build directory 'main.build'.
Nuitka:INFO: Successfully created 'main.dist\main.exe'.

Actual output using Nuitka-built version

> main.dist\main.exe
Traceback (most recent call last):
  File "D:\workspace\projects\nuitka-problem\implib_sorted_resources_problem\main.dist\main.py", line 9, in <module>
  File "D:\workspace\projects\nuitka-problem\implib_sorted_resources_problem\main.dist\main.py", line 5, in main
  File "D:\workspace\projects\nuitka-problem\implib_sorted_resources_problem\main.dist\some_package\handle_resources.py", line 7, in doit
TypeError: '<' not supported between instances of 'nuitka_resource_reader_files' and 'nuitka_resource_reader_files'

Additional info:

> dir /B D:\workspace\projects\nuitka-problem\implib_sorted_resources_problem\main.dist\some_package\some_resources
file_A.txt
file_B.txt

[kayhayen]

I see, comparison is also something I have not seen before, but sorting is of course a thing bound to be desired.

[kayhayen]

I managed to reproduce this, should be easy to add, however, not sure what the best way is. Probably converting itself and argument to str has to work. The messy way that tp_rich_compare works, we might be doing it wrong with overloading, but I don't really know better.

[kayhayen]

Thanks, that was super easy to use. The only way it could have been better would be to provide project options with # nuitka-project: as described in the user manual, but this super easy to reproduce, thanks a lot.

Thanks for your report, this is fixed on the factory branch, which is a development version under rapid development. You can try it out by going here: https://nuitka.net/doc/factory.html

Feedback if this is working is very welcome, just please do not share plans of doing it, but rather confirmations or denials of it working.

[richardscholtens]

It works now. Thank you!

[jhbuhrman]

It works now. Thank you!

Explanation: Richard works in the same project as I do. He verified this factory-version fix of the issue, and the issue is gone now.

[kayhayen]

No worries, I do not actually manage to track who said what anyway. If you have more like these, swing them my way. Nuitka is supposed to be compatible, but for all I know, there is no ultimate API description that I can test against, so this is continously catching up to what Python does and thus people expect to happen. Very happy with these reports.

[kayhayen]

This was released with 1.8 and should definitely be in 1.9, this was forgotten to be updated.