Improvement in OneDir structure

[darpan5552]

I am frustrated of finding the executable in large list of bundled libraries and files. There are more than 40 files in the main directory of OneDir build. Exposing that directory to users might irritate them in finding the correct file to click, hence decreasing usability and effectiveness of software.
I also tried OneFile build, but facing ABI Incompatibility issues #5573

Solution:
A better way will be to shift dist/<onedir> to dist/newfolder/<onedir> and creating a symlink in newfolder to point to the executable in jungle of files. This way user opening the directory will be happy with executables, others may choose to dive deeper. If not by default, at least an option to do so will be better.

[bwoodsend]

I'd really like this too. It's been discussed a lot but we never did work out a works universally solution.

The real problem with symlinks is Windows. It's rubbish at them. First you need to make them. Python's os.symlink() doesn't support Windows. Instead you have to ctypes your way through Window's C API although this is easier than it sounds. Then there are all kinds of hidden constraints which differ according to Windows version. It's a while since I tried and I've forgotten the specifics but I seem to remember the target having to be absolute which effectively rules out its usefulness to PyInstaller. I'm also fairly certain symlink creation requires root privileges for some reason.

I'm not saying for sure that it's impossible, but I don't think Windows is capable of doing this.

I tried the next closest thing I could think of: to create a shim executable written in C which just says launch the executable with the same name as this shim but inside a _guts folder, propagating CLI arguments. That seemed to work quite well but it felt too hacky for me to keep it.

The best solution I can think of is to rework the bootloader so that it sets sys._MEIPASS, PATH and LD_LIBRARY_PATH to point to a predefined sub-directory rather than it's own location. Then put everything except the executable in this sub-directory. This will probably be a messy transition but I don't see why it can't work.

[jjuod]

As an alternative while this is being addressed, I'd like to mention InnoSetup which creates installers for tidier deployment. Currently we create a --onedir package with PyInstaller, then use an InnoSetup script to compress that into a single deployable installer .exe. On the user end, this launches a typical Windows setup wizard, which can create desktop/start menu symlinks to the actual app.exe, among other options.

[Legorooj]

The best solution I can think of is to rework the bootloader so that it sets sys._MEIPASS, PATH and LD_LIBRARY_PATH to point to a predefined sub-directory rather than it's own location. Then put everything except the executable in this sub-directory. This will probably be a messy transition but I don't see why it can't work.

This isn't hard, it's just a backwards-incompatible change. But then there's a lot of backwards-incompatible changes that need doing so I don't see why it would be too hard to do this.

[darpan5552]

@jjuod @bwoodsend
Thanks for your wonderful suggestion.
As i have mentioned, this feature request was originally inspired by #5573 , have a quick look on it.
In practice, there are lot of issues in creating compatible apps. Seems like this problem only exists for small developers, other popular softwares perfectly support wide range of operating system 😒

[KarthikAbiram]

The best solution I can think of is to rework the bootloader so that it sets sys._MEIPASS, PATH and LD_LIBRARY_PATH to point to a predefined sub-directory rather than it's own location. Then put everything except the executable in this sub-directory. This will probably be a messy transition but I don't see why it can't work.

This isn't hard, it's just a backwards-incompatible change. But then there's a lot of backwards-incompatible changes that need doing so I don't see why it would be too hard to do this.

Coming from #6075

Its good to hear that its not hard to do this :) One question - By backwards incompatible change, do you mean that a built pyinstaller can either do this way or that way - but can't do this based on a user configured flag? If so, that would be a bummer. If that is possible, this can be an additional optional flag to make the pyinstaller make the dependencies go to a sub-directory like a bin folder.

If we are able to do this, it would be great - as the EXE directory will look more professional and well received by the community.

[bwoodsend]

By backwards incompatible change, do you mean that a built pyinstaller can either do this way or that way - but can't do this based on a user configured flag?

It's very difficult to pass even the simplest of configurations to the bootloader because it's written in C but we don't want our users to need a C compiler just to use PyInstaller. Possibly, it could detect at runtime that the libpython.so is not where it is expected to be (i.e. in the same folder as the executable) but is in say _guts/libpython.so.

[rokm]

What if we implemented this as "directory sideloading"? The onefile mode already supports package sideloading, i.e., instead of embedding CArchive, it can be written to progname.pkg and placed next to progname.exe.

So bootloader could look for progname.pkg, and if it exists and is a directory, use it as its _MEIPASS for onedir mode.

This, however, raises a question of where __main__'s __file__ should point to in such cases, and it will also break current relation of os.path.dirname(sys.executable) == sys._MEIPASS for onedir mode.

And I'm not particularly keen on having to deal with two possible onedir configurations, both in CI/tests and in bug reports. I think that if onedir layout is problematic for you (i.e., you want to "hide" it from the users because it's too messy), you should either put the whole bundle into subdirectory and create (symbolic) link to the executable, or use an installer that creates start menu and desktop links (assuming we're talking Windows here).

[bwoodsend]

This, however, raises a question of where main's file should point to in such cases,

__file__ would have to point inside the subdirectory so that code like data_file = pathlib.Path(__file__).with_name("my-data-file") still works (assuming that my-data-file also goes in the subdirectory which I expect it would?).

and it will also break current relation of os.path.dirname(sys.executable) == sys._MEIPASS for onedir mode

I can't see why this would matter but I've said those words and been proven wrong enough times to know that that doesn't mean much.

you should either put the whole bundle into subdirectory and create (symbolic) link to the executable

The symlink approach knocks out Windows 7 and 8 (which was still some 25% of Windows users last time I checked) because only absolute link targets are allowed - i.e. the symlink would have to know the location of the bundle which would make it no longer be relocatable.

I'd say my biggest concerns are the added complexity to the bootloader and the 3^rd permutation to have to test on CI/CD - unless we limit which tests to run this permutation, it'll be another 40 minutes or so testing time.

[Legorooj]

I would like to note that after I release 4.5, the plan is to work towards 5.0. Which can include as many breaking changes as we want. So we could redesign the entire bundle format/structure, without worrying about compatibility/non-breaking changes as much.

[KarthikAbiram]

I would like to note that after I release 4.5, the plan is to work towards 5.0. Which can include as many breaking changes as we want. So we could redesign the entire bundle format/structure, without worrying about compatibility/non-breaking changes as much.

This is great to know and if so we can add this capability in the 5.0 version which is already planned to be a breaking change :)

[LewisGaul]

Big +1 on this enhancement from me.

I've come up with a workaround solution where I create a simple wrapper script on Windows, and use symlinks otherwise, see my create_release.py script.

After my post-processing, the directory structure looks like:

+-- minegauler/
|-- CHANGELOG.txt
|-- README.txt
|-- LICENSE.txt
`-- minegauler.bat

where minegauler.bat contains:

CD minegauler
START minegauler.exe

Possibly I'm missing some edge-case handling here, but maybe this is something that can be built-on as a workaround or even a solution in v5.x?