Problem generating the index file with Python 3.6

[BadAngelZ]

Hi,
I am prototyping usage of ratarmount on a distributed environment (compute nodes) that access a shared storage (Lustre). I have huge datasets to deal with (up to 8 million files), and I would need to reduce the number of (small) files: and ratarmount seems a perfect tool for this.
I tested it on my laptop with no problem at all. The problem comes when I try to use it on the "production" environment (the compute nodes).

The problem:
When I mount an archive (be it tar or tgz) the first time, I get this error on a small archive (26 GB for ~26000 files):

$ ratarmount tf_medium.tgz
Creating offset dictionary for /XXX/tf_medium.tgz ...
Position 331354366 of 26563406918 (1.25%). Remaining time: 2 min 38 s (current rate), 2 min 38 s (average rate). Spent time: 0 min 2 s
Position 675300055 of 26563406918 (2.54%). Remaining time: 2 min 30 s (current rate), 2 min 33 s (average rate). Spent time: 0 min 4 s
Traceback (most recent call last):
  File "/home/XXX/.local/bin/ratarmount", line 8, in <module>
    sys.exit(cli())
  File "/home/XXX/.local/lib/python3.6/site-packages/ratarmount.py", line 1798, in cli
    indexMinimumFileCount        = args.index_minimum_file_count,
  File "/home/XXX/.local/lib/python3.6/site-packages/ratarmount.py", line 590, in __init__
    mountSources[key] = openMountSource(path, **options)
  File "/home/XXX/.local/lib/python3.6/site-packages/ratarmountcore/factory.py", line 149, in openMountSource
    raise CompressionError(f"Archive to open ({str(fileOrPath)}) has unrecognized format!")
ratarmountcore.utils.CompressionError: Archive to open (/XXX/tf_medium.tgz) has unrecognized format!

The file index file is generated, but is "empty": there is only the "files" table defined, and not a single record in it.

The behavior is the same with tar or tgz file (so I guess that it's not really an unrecognized format).
The problem is not related to the Lustre filesystem, since the behavior is the same with the file stored on a local file system.
On the compute node: I am able to mount the archive if I specify "--index-minimum-file-count 50000", because no index file is created.
On my laptop: I am able to mount a copy of this archive: the index file is correctly generated.
With the index file created on my laptop, I am able to mount (and use) this archive on my compute node.
I have the same behavior on 5 different compute nodes (all of them are hardware identical)

I think that I installed all the recommended dependencies, but I may have missed something.

Here are some details on one of the compute node:

• OS: Red Hat Enterprise Linux Server release 7.9 (Maipo)
• Python 3.6.8
• pip3 21.3.1
• 2x Xeon Gold 6242R (20 cores / 40 threads each)
• Memory 14x 16GB

Any advice on where to look for diagnosing the problem ?

[BadAngelZ]

I ran the same command in debug mode:

$ ratarmount tf_medium2.tgz -d 3
[Info] Try to open with rarfile
[Info] Try to open with tarfile
[Info] Detected compression None for file object: <_io.BufferedReader name='/XXXXX/tf_medium2.tgz'>
[Info] Because of the given positive index minimum file count (1000) and because no explicit index file path is given, try to open an SQLite database in memory first.
[Info] Create new index in memory because memory is to be preferred, e.g., for small archives.
Creating offset dictionary for /XXXXX/tf_medium2.tgz ...
Position 507762688 of 26568304640 (1.91%). Remaining time: 1 min 43 s (current rate), 1 min 43 s (average rate). Spent time: 0 min 2 s
Position 919007232 of 26568304640 (3.46%). Remaining time: 2 min 5 s (current rate), 1 min 52 s (average rate). Spent time: 0 min 4 s
[Info] Exceeded file count threshold (1001 > 1000) for metadata held in memory. Will try to reopen the database on disk.
[Info] Back up database from :memory: -> /XXXXX/tf_medium2.tgz.index.sqlite
[Info] Trying to open with tarfile raised an exception: no such table: parentfolders
Traceback (most recent call last):
  File "/home/XXXXX/.local/lib/python3.6/site-packages/ratarmountcore/SQLiteIndexedTar.py", line 521, in _processSerial
    self._setFileInfos(fileInfos)
  File "/home/XXXXX/.local/lib/python3.6/site-packages/ratarmountcore/SQLiteIndex.py", line 816, in setFileInfos
    self._reloadIndexOnDisk()
  File "/home/XXXXX/.local/lib/python3.6/site-packages/ratarmountcore/SQLiteIndex.py", line 577, in _reloadIndexOnDisk
    oldSqlConnection.backup(self.sqlConnection)
AttributeError: 'sqlite3.Connection' object has no attribute 'backup'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/XXXXX/.local/lib/python3.6/site-packages/ratarmountcore/factory.py", line 135, in openMountSource
    result = _BACKENDS[name](fileOrPath, **options)
  File "/home/XXXXX/.local/lib/python3.6/site-packages/ratarmountcore/factory.py", line 34, in _openTarMountSource
    return SQLiteIndexedTar(fileOrPath, **options)
  File "/home/XXXXX/.local/lib/python3.6/site-packages/ratarmountcore/SQLiteIndexedTar.py", line 817, in __init__
    self._createIndex(self.tarFileObject)
  File "/home/XXXXX/.local/lib/python3.6/site-packages/ratarmountcore/SQLiteIndexedTar.py", line 924, in _createIndex
    self._createIndexRecursively(fileObject, streamOffset=streamOffset)
  File "/home/XXXXX/.local/lib/python3.6/site-packages/ratarmountcore/SQLiteIndexedTar.py", line 945, in _createIndexRecursively
    filesToMountRecursively = metadataReader.process(fileObject, pathPrefix, streamOffset)
  File "/home/XXXXX/.local/lib/python3.6/site-packages/ratarmountcore/SQLiteIndexedTar.py", line 537, in process
    return self._processSerial(fileObject, pathPrefix, streamOffset)
  File "/home/XXXXX/.local/lib/python3.6/site-packages/ratarmountcore/SQLiteIndexedTar.py", line 525, in _processSerial
    self._setFileInfos(fileInfos)
  File "/home/XXXXX/.local/lib/python3.6/site-packages/ratarmountcore/SQLiteIndex.py", line 829, in setFileInfos
    self._tryAddParentFolders(row[0], row[2], row[3])
  File "/home/XXXXX/.local/lib/python3.6/site-packages/ratarmountcore/SQLiteIndex.py", line 797, in _tryAddParentFolders
    [(p[0], p[1], offsetheader, offset) for p in paths],
sqlite3.OperationalError: no such table: parentfolders
[Info] Try to open with zipfile
[Info] Try to open with libarchive
Traceback (most recent call last):
  File "/home/XXXXX/.local/bin/ratarmount", line 8, in <module>
    sys.exit(cli())
  File "/home/XXXXX/.local/lib/python3.6/site-packages/ratarmount.py", line 1798, in cli
    indexMinimumFileCount        = args.index_minimum_file_count,
  File "/home/XXXXX/.local/lib/python3.6/site-packages/ratarmount.py", line 590, in __init__
    mountSources[key] = openMountSource(path, **options)
  File "/home/XXXXX/.local/lib/python3.6/site-packages/ratarmountcore/factory.py", line 149, in openMountSource
    raise CompressionError(f"Archive to open ({str(fileOrPath)}) has unrecognized format!")
ratarmountcore.utils.CompressionError: Archive to open (/XXXXX/tf_medium2.tgz) has unrecognized format!

When I ran the same mount on my laptop I didn't get the "sqlite3.OperationalError: no such table: parentfolders", and I didn't find this table in the (well) generated index file...

[tailtwo]

The function sqlite3.Connection.backup() used during index reload has been added in Python 3.7 as specified by the documentation : https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.backup

It seems ratarmount suggest Python 3.6+ in the README, it would be good for it to be updated accordingly.

[mxmlnkn]

The unfortunate thing is that ratarmount did support Python 3.6 for a very long time and still has separate dependencies for it (dependencies = [ 'dataclasses; python_version < "3.7.0"', ...]). But since Python 3.6 is not available in the Github CI anymore, I overlooked this break. The backup-introducing commit is also fairly recent. It was introduced with ratarmountcore 0.6.4 / ratarmount 0.14. Using anything older may work for you, e.g., with pip install ratarmountcore==0.6.3; pip install ratarmount==0.14.

I have also tested on test.pypi.org and pip is intelligent enough to somehow find the most recent version while considering the project.requires-python = ">= 3.7" key-value in pyproject.toml. In the future, I should use this.

The problem now is that I would need to publish a "more-recent" version for pip to find that also is flagged to be compatible with Python 3.6. However, it seems wrong to simply an older ratarmountcore version on PyPI as a newer one (i.e. a forced downgrade). I guess I have to work around the usage of backup with a version check for now, publish that as a fix, rightly so, and then add the requires-python = ">= 3.6" simply as a blanket check because I cannot test newer ratarmount releases on older Python versions any longer.

I am running in circles with that damn CI:

• Python 3.6 is not available through the setup-python action anymore, also not in ubuntu-latest any longer.
• I cannot even test locally because the deadsnakes PPA does not distribute 3.6 anymore for Ubuntu 22.04+ versions.
• Trying to run the test CI in a manylinux container also yields me different out-of-support issues:
  • quay.io/pypa/manylinux_2_28_x86_64 (AlmaLinux 8) has Python 3.9-3.14 (2025-06)
  • quay.io/pypa/manylinux_2_24_x86_64 (Debian 9) has Python 3.5-3.11 (2025-06)
    • Debian 9 repository sources return 404 and I cannot install packages, probably turned off the servers.
  • quay.io/pypa/manylinux2014_x86_64 (CentOS 7) has Python 3.8-3.14 (2025-06)
  • quay.io/pypa/manylinux2010_x86_64 (CentOS 6) has Python 3.6-3.10 (2025-06)
    • My tests do not run on CentOS because they require many tools such as pixz for generating data that do not seem to be available.
• The next idea was to build an AppImage in one of the working CentOS images instead and then test with that AppImage, however using any Github action module does not work because fucking node.js is too recent and [does not work with the available GLIBC version](checkout@v3 using node20 instead of node16 actions/checkout#1809.
• The checkout action can easily be replaced with:

   steps:
   - name: Checkout
     run: |
       git clone "https://github.com/${GITHUB_REPOSITORY}.git" .
       git config --global --add safe.directory '*'
       git checkout "$GITHUB_SHA"

• The [GLIBC[(https://github.com/[bug] cannot use glibc 2.34 (amazon linux 2023) actions/upload-artifact#489) error in upload-artifact can not be so simply worked around because there seems to be no REST API for uploading artifacts that I could use via curl.
• I tried to check the source of the upload-artifact action, but I got lost when it redirected to actions/toolkit somewhere, and I am not a typescript programmer. I had hoped to see some API endpoint that I could use but alas.
• I have also tried with actions/upload-artifact@v3 instead of @v4, but I get Missing download info for actions/upload-artifact@v3 and I think I saw some deprecation warnings, ironically also regarding the node.js version in the past.
• I also tried out running only that particular AppImage build script step in a container, which seems to be a new and rather slim feature. That way, I can do the uploading in the more recent host system. This actually seems to work!
• Unfortunately, the AppImage does not build because pyfatfs requires setuptools 68 and ratarmount requires setuptools >= 77 while setuptools only support Python 3.6 up to version 59 because of this compatibility breaking change as to how to specify licenses. I hate how every time I find some time to hack on this project, I first have to fix 10 compatibility bugs because the Python build system is so fast-moving and breaking stuff.
• It does not even build with python 3.6 after undoing that change. I mean, I documented, but did not enforce for a long time, that setuptools >= 61 was required for pyprojects.toml. Unfortunately, I was not specific enough as to the reasons, but it seems to be true.
• Not even Python 3.7 is building anymore because of this dumb license keyword change. I'm giving up on Python 3.6. Luckily, I did not yet release that commit using the newer license keyword. I'll then correctly specify the requires-python restraints for that.

I am amazed you even got it installed with Python 3.6. I can't even get it installed with Python 3.8 anymore...

[mxmlnkn]

Should be fixed with 69daf26 released with ratarmountcore 0.8.2. The --index-minimum-file-count option will get ignored though, but I think that is a reasonable trade-off for a Python version that was released 9 years ago.