Multipackage bundles MERGE - duplication of all shared dependencies (ie, libpython etc)

[lopsided]

Hi,

I hope I've just missed something simple here, but I've been going around in circles with this for a while now. When I try to make multipackage bundles in onedir mode I just get all the dependencies loaded into both directories. I can even delete the copies from the non-master copies and all the scripts still run. That solution of course is not possible for the onefile mode where each generated executable file is the same size!

I started with the guide in the docs but hit up against the problem of any "dependent" scripts failing outright (no python found) -- this issue: #6053. I adjusted according to that and then this: https://github.com/pyinstaller/pyinstaller/blob/develop/tests/functional/specs/test_multipackage4.spec to get basically that test case:

script[1/2].py:

import sys
from pathlib import Path
cwd = Path.cwd()
print('cwd', cwd)
sys.argv.append('--data-dir=' + str(cwd))
print('All done script [1/2]!')

merged.spec:

import os
import sys

block_cipher = None
upx = True
debug = True

script_main = 'script1'
script_dep = 'script2'

a = Analysis(
    [script_main + '.py'],
    pathex=['.'],
    binaries=[],
    datas=[],
    hiddenimports=[],
    hookspath=[],
    hooksconfig={},
    runtime_hooks=[],
    excludes=[],
    win_no_prefer_redirects=False,
    win_private_assemblies=False,
    cipher=block_cipher,
    noarchive=False,
)

b = Analysis(
    [script_dep + '.py'],
    pathex=['.'],
    binaries=[],
    datas=[],
    hiddenimports=[],
    hookspath=[],
    hooksconfig={},
    runtime_hooks=[],
    excludes=[],
    win_no_prefer_redirects=False,
    win_private_assemblies=False,
    cipher=block_cipher,
    noarchive=False,
)

MERGE(
    (a, script_main, os.path.join(script_main, script_main)),
    (b, script_dep, os.path.join(script_dep, script_dep))
)

pyz = PYZ(a.pure)  # , a1.zipped_data, cipher=block_cipher)
exe = EXE(
    pyz,
    a.scripts,
    a.dependencies,
    exclude_binaries=True,
    name=os.path.join('build', 'pyi.' + sys.platform, script_main, script_main),
    debug=debug,
    bootloader_ignore_signals=False,
    strip=False,
    upx=upx,
    console=True,
    disable_windowed_traceback=False,
    argv_emulation=False,
    target_arch=None,
    codesign_identity=None,
    entitlements_file=None,
)
coll1 = COLLECT(
    exe,
    a.binaries,
    a.zipfiles,
    a.datas,
    strip=False,
    upx=upx,
    upx_exclude=[],
    name=os.path.join('dist', script_main)
)

pyzB = PYZ(b.pure)  # , b.zipped_data, cipher=block_cipher)
exeB = EXE(
    pyzB,
    b.scripts,
    b.dependencies,
    exclude_binaries=True,
    name=os.path.join('build', 'pyi.' + sys.platform, script_dep, script_dep),
    debug=debug,
    bootloader_ignore_signals=False,
    strip=False,
    upx=upx,
    console=True,
    disable_windowed_traceback=False,
    argv_emulation=False,
    target_arch=None,
    codesign_identity=None,
    entitlements_file=None,
)
coll2 = COLLECT(
    exeB,
    b.binaries,
    b.zipfiles,
    b.datas,
    strip=False,
    upx=upx,
    upx_exclude=[],
    name=os.path.join('dist', script_dep)
)

The generated dist/script[1/2] directories contain basically all the same files:

dist/script1:
total 26M
drwxr-xr-x. 1 me me  180 Oct  9 12:10 .
drwxr-xr-x. 1 me me   28 Oct  9 12:10 ..
-rw-r--r--. 1 me me 780K Oct  9 12:10 base_library.zip
-rwxr-xr-x. 1 me me 3.2M Oct  3 14:59 libcrypto.so.1.1
drwxr-xr-x. 1 me me 2.8K Oct  9 12:10 lib-dynload
-rwxr-xr-x. 1 me me 178K Sep 16  2021 liblzma.so.5
-rwxr-xr-x. 1 me me  21M Sep 16  2021 libpython3.9.so.1.0
-rwxr-xr-x. 1 me me 107K Oct  3 21:05 libz.so.1
-rwxr-xr-x. 1 me me 805K Oct  9 12:10 script1

dist/script2:
total 25M
drwxr-xr-x. 1 me me  148 Oct  9 12:10 .
drwxr-xr-x. 1 me me   28 Oct  9 12:10 ..
-rwxr-xr-x. 1 me me 3.2M Oct  3 14:59 libcrypto.so.1.1
drwxr-xr-x. 1 me me 2.7K Oct  9 12:10 lib-dynload
-rwxr-xr-x. 1 me me 178K Sep 16  2021 liblzma.so.5
-rwxr-xr-x. 1 me me  21M Sep 16  2021 libpython3.9.so.1.0
-rwxr-xr-x. 1 me me 107K Oct  3 21:05 libz.so.1
-rwxr-xr-x. 1 me me 808K Oct  9 12:10 script2

Running ./dist/script2/script2 I can see that it detects the deps in script1 and seems to be copying/extracting them to /tmp, eg:

LOADER: Coping file /home/me/pyi-tests/test_merge/dist/script2/../script1/libpython3.9.so.1.0 to /tmp/_MEI8vYKUn

And if I delete all-but the executable in script2, my script still runs fine.

(Tested using fresh pipenv with no packages installed).

So, how to stop all these files appearing?

Many thanks!

[rokm]

You really shouldn't be using MERGE with onedir, because you gain no benefits, while your frozen application(s) gain onefile semantics - i.e., they copy the "shared" data to temporary directory on each run. The semantics of MERGE make sense only with two or more onefile programs.

To have multiple executables share dynamic libraries and data files in onedir mode, use a single COLLECT instead. See this article that was linked in #6634.

But if you want to use MERGE nevertheless, you should probably omit

    b.binaries,
    b.zipfiles,
    b.datas,

in the second COLLECT.

[lopsided]

Thanks for the reply, I totally didn't get that from the docs/test cases!

Removing the MERGE and combining the two COLLECTs into a single one as per the linked article worked great. Thanks!

coll = COLLECT(
    exe,
    a.binaries,
    a.zipfiles,
    a.datas,
    exeB,
    b.binaries,
    b.zipfiles,
    b.datas,
    strip=False,
    upx=upx,
    upx_exclude=[],
    name='scripts',
)

[lopsided]

As I was working up to fixing my onefile case that gave multiple massive executables I will also put my final, working solution here as @rokm's hint for the onedir fix led me to the fix here. Any other suggestions/improvements welcome!

block_cipher = None
upx = True
debug = True

script_main = 'script1'
script_dep = 'script2'

a = Analysis(
    [script_main + '.py'],
    pathex=['.'],
    binaries=[],
    datas=[],
    hiddenimports=[],
    hookspath=[],
    hooksconfig={},
    runtime_hooks=[],
    excludes=[],
    win_no_prefer_redirects=False,
    win_private_assemblies=False,
    cipher=block_cipher,
    noarchive=False,
)

b = Analysis(
    [script_dep + '.py'],
    pathex=['.'],
    binaries=[],
    datas=[],
    hiddenimports=[],
    hookspath=[],
    hooksconfig={},
    runtime_hooks=[],
    excludes=[],
    win_no_prefer_redirects=False,
    win_private_assemblies=False,
    cipher=block_cipher,
    noarchive=False,
)

MERGE(
    (a, script_main, script_main),
    (b, script_dep, script_dep)
)

pyz = PYZ(a.pure)  # , a1.zipped_data, cipher=block_cipher)
exe = EXE(
    pyz,
    a.scripts,
    a.binaries,
    a.zipfiles,
    a.datas,
    a.dependencies,
    name=script_main,
    debug=debug,
    bootloader_ignore_signals=False,
    strip=False,
    upx=upx,
    upx_exclude=[],
    runtime_tmpdir=None,
    console=True,
    disable_windowed_traceback=False,
    argv_emulation=False,
    target_arch=None,
    codesign_identity=None,
    entitlements_file=None,
)

pyzB = PYZ(b.pure)  # , b.zipped_data, cipher=block_cipher)
exeB = EXE(
    pyzB,
    b.scripts,
    # b.binaries,  <---\
    # b.zipfiles,  <---   >  don't add the binaries/zipfiles/datas!
    # b.datas,  <---    /
    b.dependencies,
    name=script_dep,
    debug=debug,
    bootloader_ignore_signals=False,
    strip=False,
    upx=upx,
    upx_exclude=[],
    runtime_tmpdir=None,
    console=True,
    disable_windowed_traceback=False,
    argv_emulation=False,
    target_arch=None,
    codesign_identity=None,
    entitlements_file=None,
)

[kingmo888]

good ! cool!!!