whoosh open files issues on Python 3.13

[ThomasWaldmann]

I thought supporting Python 3.13 is easy, but it looks like whoosh is making troubles on 3.13:

#1960

3.13 test failures:

https://github.com/moinwiki/moin/actions/runs/16400200464/job/46338490174

[ThomasWaldmann]

Maybe related: #1645

[roland-ruedenauer]

The errors with unclosed files in your CI test run are most likely not directly related to your changes.

As far as I am concerned, they are timing related and caused by the inappropriate use of whoosh class AsyncWriter by moin and a bug in whoosh itself. I had a look into this problem some time ago and commented my findings in #1645.

These changes resolved the issue for me: https://github.com/roland-ruedenauer/moin/tree/track-async-writer-instances

Unfortunately, the fork Sygil-Dev/whoosh-reloaded is no longer maintained (mentioned in README.md). And no other maintained fork seems to exist currently.

[UlrichB22]

I can reproduce the error with a clean installation of Ubuntu 25.04. E.g. every pytest run for test_indexing.py gives following output:

$ pytest src/moin/storage/middleware/_tests/test_indexing.py
============================================= test session starts ==============================================
platform linux -- Python 3.13.3, pytest-8.4.1, pluggy-1.6.0
rootdir: /home/moinadm/moin
configfile: pyproject.toml
collected 23 items                                                                                             

src/moin/storage/middleware/_tests/test_indexing.py ....E..E...............ss                            [100%]

==================================================== ERRORS ====================================================

[...]

src/moin/conftest.py:69: AssertionError
------------------------------------------- Captured stdout teardown -------------------------------------------
Wiki file open: /home/moinadm/moin/src/moin/_tests/wiki/index/all_revs_gsz8qvps3o38hpck.seg mode: r
Wiki file open: /home/moinadm/moin/src/moin/_tests/wiki/index/all_revs_en44nbzfjgaiy0kn.seg mode: r
Wiki file open: /home/moinadm/moin/src/moin/_tests/wiki/index/latest_meta_zpko9dc5cc1zaqpb.seg mode: r
Wiki file open: /home/moinadm/moin/src/moin/_tests/wiki/index/latest_meta_4ajbdgrw8ytoj7jo.seg mode: r
=========================================== short test summary info ============================================
ERROR src/moin/storage/middleware/_tests/test_indexing.py::TestIndexingMiddleware::test_destroy_revision - AssertionError: assert [] == [popenfile(pa...=557056), ...]
ERROR src/moin/storage/middleware/_tests/test_indexing.py::TestIndexingMiddleware::test_destroy_item - AssertionError: assert [] == [popenfile(pa...flags=557056)]
=================================== 21 passed, 2 skipped, 2 errors in 5.62s ====================================

I added the file mode for the output of “Open Wiki File” and, surprisingly, it is only a read mode.

[roland-ruedenauer]

@UlrichB22 Would you mind testing the changes I made on the branch mentioned above? I'd be interested to know, if it solves the known problems on your system too.

[UlrichB22]

Tested

@UlrichB22 Would you mind testing the changes I made on the branch mentioned above? I'd be interested to know, if it solves the known problems on your system too.

The problem has been solved with https://github.com/roland-ruedenauer/moin/tree/track-async-writer-instances under Python 3.13.

I also tested pip version 2.7.5 of whoosh-reloaded with your patch to filedb/structfile.py, and the bug was also fixed. IMO this proves that the problem is related to searcher instances and not the async-writer. The change still needs to be tested on the other Python versions and other OS platforms.

Another attempt would be to patch the original Whoosh and run the tests again.

[roland-ruedenauer]

Thanks for taking the time to test the changes.

On my system (Linux) it wasn't sufficient to just apply the patch to filedb/structfile.py.
I also require the changes related to tracking whoosh async writer instances to be applied in order for the tests to complete without an error.

The problematic pattern is explained quite simple:

• issue multiple wiki item chages (creating new items)
• each item change triggers an index update
• if changes happen quickly enough (disable logging to run into the problem more easily!), the indexing will make use of async operations (when creating instances of AsyncWriter that initially can't get hold of the whoosh writer lock)
• closing all indexers is requested
• at this point some async operations are still in progress on the non-main thread
• when attempting to remove the index folder, an error occurs because some files are still open

The async writer tracking allows waiting for all AsyncWriter instances to have completed before closing the whoosh indexers.

[UlrichB22]

IMO we should stick with Whoosh 2.7.4 if possible. Are there any known improvements in whoosh-reloaded that we can benefit from?

As a first step, we can try to fix the BufferFile class in Whoosh filedb/structfile.py with a monkey patch, perhaps @ThomasWaldmann can help with this. This should enable the use of Python 3.13.

As a second step, @roland-ruedenauer, you can create a PR for the Async Writer fix.

What is your opinion about this?

[roland-ruedenauer]

@UlrichB22 I didn't check the commit history of whoosh-reloaded.

Patching the original Whoosh should be possible using this snippet:

from whoosh.filedb.structfile import BufferFile


def buffer_file_close(self):
    super(BufferFile, self).close()
    self._buf = None


# patch class BufferFile
BufferFile.close = buffer_file_close

Patching the whoosh package is a nice idea no matter which version to go with.

[UlrichB22]

#1960 has been merged successfully.