data_files removal: pip / setuptools is going to be not nice with GTK applications

[jeromerobert]

There is a long discussion about that here.

Some key points:

• data_files is a distutils/setuptools feature we use to install icons, translations, and freedesktop files.
• wheel is a standard for python packages
• wheel files are designed to install files only in the python site-packages (i.e. not in <prefix>/share)
• In the near future wheel files are going to be the only way to install files with pip
• distutils is deprecated. It will be removed from python 3.12 but still be available for while (how long ?) in the setuptools package

I managed to create custom setuptools commands to have pip install installing our data files at the right location (<prefix>/share). But as wheel ignore those files pip uninstall ignore them too.

I'm sure that supporting old and new python packaging tooling at the same time going to be a nightmare.

@dreua Would it be possible to stop using wheel in the Fedora package ? It worked so far because we use distutils commands. If I switch to setuptools commands the rpm would no longer include translation.

3 related goods news:

• A part from that data_files issue switching PDF Arranger to pyproject.toml does work on Linux (I checked it).
• cx_Freeze has started its migration to adapt to modern python packaging
• python-distutils-extra still relies on distutils but we'll soon no longer use it.

[dreua]

I'll take care of the Fedora package. Thanks for taking care of this, it sounds a bit like a mess and I haven't read the details but I trust you doing the right thing here.

[dreua]

Let's hope this doesn't affect too many other distributions. We should add a note to the next release to help package maintainers.

[dreua]

@jeromerobert an update about getting rid of wheel building in Fedora: When I initially packaged PDF Arranger for Fedora, building and installing wheels was possible but not the default way (iirc). I ended up doing that because that was how I got it to work and stuck with it. (Iirc that were similar issues with translations and data files just that wheels made it work, not the other way around.)

Therefore I first responded positively when you asked me to get rid of the wheels (here and in another issue I didn't find with a quick search).

Only later I reread the Fedora guidelines and and realized that nowadays building wheels is the recommended and preferred way and deviating from it should be well justified. Therefore I'd like to try sticking with wheels if possible. (We might even ask the Fedora Python-SIG for help if we can't figure it out ourselves.)

Actually I think with your latest changes it works even when following the latest guidelines and new RPM macros: #994

Do you still think we need to get rid of wheels?

Sorry for the confusion!

[jeromerobert]

Do you still think we need to get rid of wheels?

Not for the time being. Thanks to Cimbali I realize that the direction followed by Python packaging is still not clear to me. So let's wait and see.

Sorry for the confusion!

Don't be sorry, here I'm the one causing confusion ;-).

[dreua]

Regarding uninstall:

Do I understand correctly that each pip install will leave some files behind that can't be uninstalled? That wouldn't sound ideal to me, at least we'd need to document this behavior IMO. Better would be fixing it. I'm worried this could lead to hard to track bugs in case we ever move, delete or rename files or in cases where you pip uninstall to switch to an rpm install for example. Left over files may hide others and/or become outdated.

[Cimbali]

I don’t think that’s right. Installed paths are recorded in the {package}.dist-info/RECORD file, so when uninstalling/updating they will get removed even if the data_files is not supported in the future.

[jeromerobert]

Hum, then I'll double recheck.

[jeromerobert]

Indeed it works with pip install. RECORD contains lines like:

../../../share/applications/com.github.jeromerobert.pdfarranger.desktop,sha256=

It works with ./setup.py bdist_wheel on Debian. The wheel file constains things like:

pdfarranger-1.11a1.data/data/share/applications/com.github.jeromerobert.pdfarranger.desktop

It does not work with python3 -m build and according to our rpmbuild CI does not work with ./setup.py bdist_wheel on Fedora. In both cases *.mo files are missing.

On Debia I can see that only *.mo files are missing. Other data_files are properly so I suspect a bug in our setup.py.

I wonder why https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-data-files says that data_files do not work with wheels.

[jeromerobert]

Well python3 -m build --wheel does work.

[dreua]

I've been using venvs recently and it seems to work quite well with PDF Arranger, might be recommendable for development and translation:

python3 -m venv --system-site-packages venv
source ./venv/bin/activate  # VS Code / Codium can automate this for you, just close and reopen the console
pip install .            # Repeat after code changes
pdfarranger          # works
man pdfarranger # also works
deactivate  # in case you want to leave the venv

Everything should be contained inside the venv. There is of course no Desktop integration (Nautilus "Open With", Application Menus etc.). For that it's probably the best to always build a system packge (RPM, DEB or whatever) and install it as such.

Unfortunately pip install -e . does not seem to work (fails to find a .ui file). You might be able to run the pdfarranger.py directly in the venv with installed helper files though, haven't tried that yet.

[Cimbali]

I think at the moment you can use setuptools + wheel with data_files, which from the discussion you’ve linked are more likely to be replaced rather than flat-out removed (at least I do that on another package I maintain I’m not sure what issues you run into, willing to look into them).

Also you could include more files inside the package directory to reduce the impact of the data_files removal (like the non OS-specific translations and ui files).

[jeromerobert]

from the discussion you’ve linked are more likely to be replaced rather than flat-out removed

That is not my understanding. I understand that restricting the install directory and preventing PyPI packages from installing files anywhere they want in the <prefix> is a strong requirement. The reason for this is probably security.

I do that on another package I maintain I’m not sure what issues you run into

Wow, indeed Pympress setup.py is really polished. I'll have a look in deep.

https://github.com/Cimbali/pympress/blob/master/setup.py#L279-L282 <-- I did not managed to make that work. For me setuptools data_files forbid installation out side of site-packages.

Yes we could move translations used directly in python and .ui files from <prefix>/share to site-packages. But freedesktop files and translations used directly by GTK (that is by .ui files) must remain in <prefix>/share. Also some distros automatically index translation files and expect to have them <prefix>/share. That would require packagers to patch.

[jeromerobert]

I'm probably wrong. I did all my tests using pyproject.toml instead of setup.py, may be it has more restriction.

[jeromerobert]

For the record, Meld has dropped pip support.