New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

multipackage (MERGE) boken in PyInstaller 3.0 #1527

Open
matysek opened this Issue Sep 25, 2015 · 35 comments

Comments

Projects
None yet
@matysek
Member

matysek commented Sep 25, 2015

In PyInstaller 3.0 no much attention was paid to 'multipackage' feature and it is broken:

  • MERGE should be fixed
  • MERGE tests should be migrated to pytest

@matysek matysek added this to the PyInstaller 3.1 milestone Sep 25, 2015

@htgoebel

This comment has been minimized.

Member

htgoebel commented Sep 27, 2015

Maybe a chance to rework this. PyInstaller supports loading modules from a external file, see old_suite/basic/test_pyz_as_external_file.spec and bootloader/main. We could change MERGE into using this and dramatically simplify the code.

Edit: This test-case has now been moved to test_pyz_as_external_file in functional/test_basic.py: py.test -k test_pyz_as_external_file.

@kalikaneko

This comment has been minimized.

kalikaneko commented Mar 8, 2016

I'm interested in this feature, and I might be able to help with fixing it. I did a very simple test (just an import and a print), and it seems not to be duplicating the libs, so I don't see exactly where it could be broken.

samsara 戝 ~/tmp/pyinst-MERGE/dist 
10712 ◯ : tree                                                                                                                                                                      
.
├── bar
│   └── bar
├── baz
│   └── baz
└── foo
    ├── bar
    ├── baz
    ├── bz2.x86_64-linux-gnu.so
    ├── _codecs_cn.x86_64-linux-gnu.so
    ├── _codecs_hk.x86_64-linux-gnu.so
    ├── _codecs_iso2022.x86_64-linux-gnu.so
    ├── _codecs_jp.x86_64-linux-gnu.so
    ├── _codecs_kr.x86_64-linux-gnu.so
    ├── _codecs_tw.x86_64-linux-gnu.so
    ├── _ctypes.x86_64-linux-gnu.so
    ├── foo
    ├── _hashlib.x86_64-linux-gnu.so
    ├── _json.x86_64-linux-gnu.so
    ├── libbz2.so.1.0
    ├── libcrypto.so.1.0.2
    ├── libexpat.so.1
    ├── libffi.so.6
    ├── libpython2.7.so.1.0
    ├── libreadline.so.6
    ├── libssl.so.1.0.2
    ├── libtinfo.so.5
    ├── libz.so.1
    ├── _multibytecodec.x86_64-linux-gnu.so
    ├── pyexpat.x86_64-linux-gnu.so
    ├── readline.x86_64-linux-gnu.so
    ├── resource.x86_64-linux-gnu.so
    ├── _ssl.x86_64-linux-gnu.so
    ├── termios.x86_64-linux-gnu.so
    ├── twisted.python._sendmsg.x86_64-linux-gnu.so
    └── zope.interface._zope_interface_coptimizations.x86_64-linux-gnu.so

3 directories, 32 files
(env2) 

(using d9be7da due to #1863 affecting me at least on my local enviroment)

@htgoebel could you please elaborate a bit more on the needed rewrite when you have some time?

@htgoebel

This comment has been minimized.

Member

htgoebel commented Mar 11, 2016

@kalikaneko Well, the tests (which are still in the old test-suite only) all fail, Try:

cd tests/old_suite/
./runtests.py multipackage/test_*.py

My idea for rewriting it is: Instead of appending the PKG to one of the executables, create an external file pkg which is used. The bootloader already contains code for this. But now as I'm checking the code, I'm no longer convinced that this will be a notable improvement.

@miphip

This comment has been minimized.

miphip commented Aug 25, 2016

Selected PyInstaller based on this feature. Disappointed to see it not working. What to do?

@bluebad

This comment has been minimized.

bluebad commented Nov 24, 2016

Not working still?
Now my app consists of two simple executables that build with pyinstaller so horrible large.

@letlet

This comment has been minimized.

letlet commented Dec 20, 2016

Please add this to your todo list.

@MerlijnWajer

This comment has been minimized.

Contributor

MerlijnWajer commented Jan 8, 2017

I just wanted to say that this works for me, given a small hack.

For some time I was trying to fork+exec my own application, to launch some workers due to inherent issues with multiprocessing on OS X (not pyinstaller related). To prevent unpacking overhead, I would do something like this::

def pyinstaller_env():
    """
    Returns None if not in pyinstaller. (None means: Keep current environment,
    at least to Popen)

    Otherwise returns the current environment, plus the _MEIPASS2 environment
    variable, provided to ensure Pyinstaller doesn't unpack itself again when
    we exec() it.

    For multipackage bundles, this is required for the multipackage bundles to
    function at all.
    """
    if hasattr(sys, 'frozen'):
        env = os.environ
        # If Pyinstaller encounters _MEIPASS2 in the env, it will not unpack
        # and instead use that directory for the bootloader. By using this we
        # avoid unpacking again for every fork+exec.
        env['_MEIPASS2'] = sys._MEIPASS

        return env

    return


def fork_self(args):
    """ Fork self. Possibly using a different entry point (args)
    """
    print('fork_self:', args)
    env = pyinstaller_env()

    main = subprocess.Popen([sys.argv[0]] + args, env=env)

    return main

However, the second (smaller) application does not start, with an error like this:
Error loading Python lib '/var/folders/sy/ky6wtvhs3yx_b1856h3g8w6r0000gn/T/_MEIk5dZOT/.Python': dlopen(/var/folders/sy/ky6wtvhs3yx_b1856h3g8w6r0000gn/T/_MEIk5dZOT/.Python, 10): image not found

But, using my pyinstaller_env() function when spawning the second application does work.

Maybe this is useful for others - if you fork+exec to start one of your other MERGE'd + BUNDLE'd application, just pass _MEIPASS2 (with value set to the current _MEIPASS)

(Background: my fork+exec worker combo worked fine, except that it would spawn lots of dock icons, due to the main application being a windowed application, and pyinstaller setting up every time. That's why I created a second binary, which is otherwise identical, except that it has console=True, instead of console=False.)

@tallforasmurf

This comment has been minimized.

Contributor

tallforasmurf commented Jan 8, 2017

Would this be suitable as a Wiki recipe?

I don't get any hits on a search in the repository for _MEIPASS2 so I wonder where in the bootloader this is tested?

@MerlijnWajer

This comment has been minimized.

Contributor

MerlijnWajer commented Jan 8, 2017

_MEIPASS2 is mentioned here - https://github.com/pyinstaller/pyinstaller/wiki/Recipe-Multiprocessing

I didn't find it in the documentation either, but that recipe made me try the same on OS X/Linux, and it also seems to work there.

_MEIPASS2 is found here: bootloader/src/pyi_main.c

I don't know if this is the right 'solution', so I don't think I am qualified to comment on whether this counts as a recipe or not.

@tallforasmurf

This comment has been minimized.

Contributor

tallforasmurf commented Jan 9, 2017

Yeah, it's confusing. I'm confused because github search still doesn't turn up that pyi_main.c hit in the 41 results for '_MEIPASS2'. But you have fingered the place it matters, here. The bootloader starts, fetches that envar, and at line 122 makes a critical decision.

If _MEIPASS2 was not defined, this is the original bootloader thread and it starts the process of creating the temp folder and unpacking into it. When that completes it will fork itself, set _MEIPASS2 and we come back to line 122 now in a second thread. Because the envar is set, it knows this is the second instance, unpacking is complete, and it can proceed to kick off the unpacked user program.

So by pre-setting the envar and then launching the same executable in a subprocess, you are preventing the bootloader at the head of that executable from going through the unpacking-and-forking business. It just proceeds into the user code in its subprocess thread.

If you made a subprocess out of a different executable it would be a mistake to set that envar because that different executable would not unpack itself, would try to import modules from the parent program's temp folder and would likely fail with an import error.

The multiprocessing recipe you point to seems to be doing this same thing. (The same code appears in this functional test and this one.

However it looks odd to me for a couple of reasons, one is this:

if sys.platform.startswith('win'):
...
            finally:
...
                    # On some platforms (e.g. AIX) 'os.unsetenv()' is not
                    # available. 

Since the code is restricted to Windows, why is it nattering about AIX?

@MerlijnWajer

This comment has been minimized.

Contributor

MerlijnWajer commented Jan 9, 2017

Yes, that is indeed what _MEIPASS2 does. However - your note on 'different executable' is not entirely correct. In this case (MERGE target), only one of the binaries (first one passed to MERGE) contains all of the dependencies. If that one unpacks first, and then the others are started with that _MEIPASS2, it just seems to work.

The multiprocessing recipe mentions Windows specifically because multiprocessing on Windows, using python2, can't do fork+exec (because it is Windows) and you need the hacks that they mention in the recipe. Why they mention AIX is beyond me. It may just be general coding practice / a habbit regarding os.unsetenv. Multiprocessing on python3 will likely require similar hacks/recipes for all platforms.

@ghost

This comment has been minimized.

ghost commented Jan 9, 2017

@MerlijnWajer Some tests fail unexpectedly at the moment because the loader doesn't setup the environment correctly for the parent thread sometimes. You can see a test failure here, but they have also occurred on linux. Can you guess what might be causing this?

@MerlijnWajer

This comment has been minimized.

Contributor

MerlijnWajer commented Jan 10, 2017

Are you sure that this is related to the MERGE target? I haven't looked at pyinstaller tests before, so I don't know offhand what is going wrong there. It's also Windows, which I don't usually test pyinstaller on.

@htgoebel - any comments on my previous notes on this? Could this be related to the broken tests? I guess I should perhaps dive into the bootloader code and see if this can be fixed/changed, but I don't know what the intended behaviour is.

@ghost

This comment has been minimized.

ghost commented Jan 11, 2017

@MerlijnWajer

This comment has been minimized.

Contributor

MerlijnWajer commented Jan 11, 2017

@xoviat: Just to check, I don't see how this relates to the MERGE issue?

@ghost

This comment has been minimized.

ghost commented Jan 11, 2017

I don't think it does now. Originally I thought that it was related to the bootloader, but @htgoebel has come up with a more plausible explanation.

@ghost

This comment has been minimized.

ghost commented Jan 26, 2017

I may look at this after #2341 is merged.

@ghost

This comment has been minimized.

ghost commented Jan 28, 2017

I've made some progress at github.com/xoviat/mergemodule. The main roadblock now is that Using the single package approach, the bootloader is unable to determine the correct script to start for both executables.

@ghost

This comment has been minimized.

ghost commented Jan 28, 2017

Unfortunately, It appears that I will have to backtrack because append_pkg cannot be used to create a package that an executable can distinguish the specific scripts to load.

@Felix-neko

This comment has been minimized.

Felix-neko commented Apr 6, 2017

Cannot you please restore the feature? It is really useful for us.

We develop a Python package and have about 10 demo Python scripts that show its capabilities.
We want to distribute the demo scripts with PyInstaller but we want to put all 10 generated exe files into same folder.

@lvu

This comment has been minimized.

lvu commented Apr 20, 2017

This would be a really useful feature; it is the only thing that stops me moving from py2exe to pyinstaller.

@matatk

This comment has been minimized.

matatk commented Jun 25, 2017

I recently discovered PyInstaller; it works very well, thanks! It seems that my program also needs this feature. I'm not sure that I would understand the codebase well enough to contribute patches, but I would be happy to help test out any solutions on which you may be working.

(I noticed the mention of subzero in another issue, and will check it out, but it would be great to see this feature restored to solve the problem fully.)

@Terrabits

This comment has been minimized.

Terrabits commented Jul 8, 2017

I don't know pyinstaller well enough to know if this is a bad idea, but if I just copy the second executable into the directory of the first they both work with the same shared library. I'm using Python 3.5.1, Pyinstaller 3.2.1 and MacOS 10.12.5. It at least works for this environment.

For executables with non-conflicting dependencies this might be an option.

@htgoebel

This comment has been minimized.

Member

htgoebel commented Aug 19, 2017

This requires larger changes to the bootloader and will not make it into release 3.3.

@ErikBjare

This comment has been minimized.

Contributor

ErikBjare commented Aug 25, 2017

Just discovered this feature in the docs. I'm building a multi-program bundle using PyInstaller by having one spec file for each program and building them individually. When done I simply copy all the files into the same directory (seems to be what @Terrabits is doing as well). The project is ActivityWatch.

Haven't had any issues with it, so I'm wondering what the practical differences are.

@codewarrior0

This comment has been minimized.

Member

codewarrior0 commented Aug 25, 2017

The idea behind MERGE mostly applies to onefile executables.

With onedir exes, all of the Python bytecode is archived into the main executable, while the rest of the app dependencies (including data files and shared libraries) are stored in the executable's folder. This is why it often works to just merge two onedir exes file-wise.

With onefile exes, every single file is archived into the main exe (and extracted to a temp folder at runtime). Putting two of these exes in the same folder means that there will be many files that are archived twice, with one copy in each exe. What MERGE allows is to have the first exe contain all of the dependencies, and the second exe just have references to the first one instead of second copies of, for example, python.dll.

@Felix-neko

This comment has been minimized.

Felix-neko commented Dec 18, 2017

Please fix it, it can be really important

@htgoebel

This comment has been minimized.

Member

htgoebel commented Dec 18, 2017

If one needs this, please provide a pull request or sponsor a project grant.

@MerlijnWajer

This comment has been minimized.

Contributor

MerlijnWajer commented Dec 19, 2017

@htgoebel Could you comment on the larger changes required to the bootloader?

@ghost

This comment has been minimized.

ghost commented Dec 22, 2017

Are you a C programmer? Look at the PRs I submitted and fix them.

@MerlijnWajer

This comment has been minimized.

Contributor

MerlijnWajer commented Dec 22, 2017

Yeah, C is no problem. Can you link me to the relevant PRs? You have rather a lot. :)

@ghost

This comment has been minimized.

ghost commented Dec 22, 2017

See gh-2416. Note that you'll need to extract the relevant changes manually (ugh!) and then clean them up so that just the bootloader is modified. In addition, you should try to avoid moving functions around (as I did to try to make up for my poor C knowledge at the time I submitted the PR).

@uSpike

This comment has been minimized.

uSpike commented Jul 2, 2018

Hi, just wondering if there's any progress that was made on this issue? It seems that #2416 was abandoned and the user was deleted.

@htgoebel

This comment has been minimized.

Member

htgoebel commented Jul 2, 2018

#2416 is huge, and nobody started a new approach.

@uSpike

This comment has been minimized.

uSpike commented Jul 2, 2018

@htgoebel I don't understand the specifics of the bootloader change, but I could implement some of your comments in #2416. Would you consider merging separate PRs for bootloader changes / the rest of the changes (tests, etc)? I can handle pytest changes well.

@htgoebel htgoebel removed this from the PyInstaller 3.4 milestone Aug 28, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment