Organizing parts of a large application

[fredrikaverpil]

Hi again @mherrmann!

In src/main/python, it seems like the intention is we can add multiple folders here in case our application is made out of multiple packages/parts. I've noticed that the tutorial example can import anything which resides here (very nice!) when running using python -m fbs run.

For example, I've got an application I'm experimenting with, which I might want to "fbs-ify" which is separated into many parts. Trying to simplify, here's an example of what my app could be made out of:

1. Base app UI
2. Models and modules (callable from UI or headlessly)
3. Sub-apps (QMainWindow or QWidget based Python MVC-style UIs launched from the base UI)

I'm thinking I could organize them in fbs by placing each "part" in a subfolder under src/main/python, as they could then be imported via a simple "import part" statement. However, I'm noticing that they won't get included in the app bundle when freezing (on Windows).

To make a frozen app work, I have to place a copy of everything in the "resources" folder prior to the freeze, and it will successfully get bundled with the standalone app. So I'm starting to wonder how I should plan the file structure when using fbs... I don't want to put the same file in two locations so to speak.

What would you advise I do here?

My goal is to be able to have the standalone fbs application be able to import a certain module, but also allow for a headless way of importing the same module (meaning using Python).

EDIT: A very simple example is to just add an extrafile.py next to the tutorial's main.py and import this file in main.py. This will work fine with python -m fbs run but the extrafile.py file will not be included with the standalone app when performing a python -m fbs run, and thus the standalone app will crash.

[mherrmann]

Hey Fredrik,

I just tried the externalfile sample you mentioned and it works for me (see diff below). Either way, I think
I would use a custom build script based on fbs. Something like the following:

from fbs.cmdline import command
from fbs.builtin_commands import freeze
from fbs.freeze import run_pyinstaller
from os.path import dirname

import fbs.cmdline
import fbs.builtin_commands

@command
def freeze_all():
    # Freeze the "Base" app:
    fbs.builtin_commands.freeze()
    _freeze_headless()

def _freeze_headless():
    run_pyinstaller(...) # etc.

if __name__ == '__main__':
    project_dir = dirname(__file__)
    fbs.cmdline.main(project_dir)

The way I have it in fman is that I have such a file as build.py in the root project directory (ie. next to requirements.txt.) Then you can do

python -m fbs freeze_all

I think PyInstaller will complain if you run it twice with the same target directory. I would try running it with different target directories and then "brutally" copying them over each other.

I'm afraid I'm not 100% sure I understood the requirements you outlined. Are the sub-apps standalone applications as well? Or are the simply windows displayed by the Base app? If the latter, I think I would try the following directory structure:

• src/main/python/
  • core/ <- Importable by everything else, for the "models and modules"
  • gui/
    • main.py <- The "Base" app
    • subapp_1/
    • subapp_2/
  • cmdline/ <- The "headless" app, imports core
    • main.py <- main script for the cmdline app

But of course that's just my best guess. I hope but am not sure it will work exactly like that for you :)

Diff:

diff --git a/src/main/python/tutorial/externalfile.py b/src/main/python/tutorial/externalfile.py
index e69de29..e21af34 100644
--- a/src/main/python/tutorial/externalfile.py
+++ b/src/main/python/tutorial/externalfile.py
@@ -0,0 +1,2 @@
+def f():
+       print('hi')
\ No newline at end of file
diff --git a/src/main/python/tutorial/main.py b/src/main/python/tutorial/main.py
index 3237366..3fab972 100644
--- a/src/main/python/tutorial/main.py
+++ b/src/main/python/tutorial/main.py
@@ -1,8 +1,10 @@
+from tutorial.externalfile import f
 from tutorial.application_context import AppContext
 
 import sys
 
 if __name__ == '__main__':
+    f()
     appctxt = AppContext()
     exit_code = appctxt.run()
     sys.exit(exit_code)

[fredrikaverpil]

Okay, I realize I asked too many questions at once 😉

I just tried the externalfile sample you mentioned and it works for me

Ah yes, this did work... I must've done something wrong... 👍

I'm afraid I'm not 100% sure I understood the requirements you outlined.

Your example structure is almost what I'm trying to achieve, and I'll now just focus on the main problem I have. I'd like to achieve a freeze where the gui and the cmdline are bundled as executables and the core is a folder with python scripts (a module).

So, something like this for the files organization:

.
└── src
    └── main
        └── python
            ├── cmdline
            │   ├── __init__.py
            │   └── main.py
            ├── core
            │   ├── __init__.py
            │   └── somemodule.py
            └── gui
                ├── __init__.py
                └── main.py

And something like this for the freeze (excluding all Python libraries etc):

.
├── cmdline.exe
├── core
│   ├── __init__.py
│   └── somemodule.py
└── gui.exe

Since both gui and cmdline imports core, it would be nice to somehow be able to set up the development so that I can develop and run these apps but then also commence a freeze which would generate the desired bundle (and which would be possible to make an installer of).

So, to clarify; I need the core module as pure python, as it will be interpreted by different third-party applications which all have embedded Python. They cannot use the binaries so to speak.

Does this all make sense?

EDIT: I could just copy core into the package before making an installer out of it, I presume... but could this be avoided?
Regardless, I still need to organize the project somehow, and make it possible to run fbs run and fbs freeze... which is the part I haven't been able to wrap my head around... I tried your build.py script, but I wasn't able to figure out how I could define the secondary app (cmdline) to be built using _freeze_headless.run_pyinstaller().

[mherrmann]

Yes, that makes sense. Thanks for clarifying :-)

I think I would simply copy the core sources into the target/App directory as a step of your freeze_all command.

You're probably right. Maybe you can't use run_pyinstaller(...) as-is. But you can more or less copy its code:

from fbs import path
from fbs.platform import is_mac
from subprocess import run
run(['pyinstaller', '--name', 'cmdline', '--noupx', '--distpath', path('target'), '--specpath', path('target/PyInstaller_cmdline'), '--workpath', path('target/PyInstaller_cmdline'), path('src/main/python/cmdline/main.py')])

Hope this is what you meant?

[mherrmann]

(P.S.: You probably can also simply call your freeze_all command freeze. It should override fbs's one.)

[fredrikaverpil]

Very nice, thank you!

This works great, except for that I get this structure:

.
├── cmdline
│   └── cmdline.exe
└── gui
    ├── core (copied into place)
    └── gui.exe

I'm just doing what you previously recommended:

def _freeze_cmdline():
    run([
        'pyinstaller', '--name', 'cmdline', '--noupx', '--distpath',
        path('target'), '--specpath',
        path('target/PyInstaller_cmdline'), '--workpath',
        path('target/PyInstaller_cmdline'),
        path('src/main/python/cmdline/main.py')
    ])

Do you think I could get the cmdline.exe to be built in the gui directory?

Right now, I just copied the cmdline.exe and cmdline.manifest over to the gui folder and deleted the cmdline folder. Most likely, everything inside of the cmdline folder needs to be copied over onto the gui folder as well, as part of the freeze, in case it doesn't already exists there...

[fredrikaverpil]

I'm going to make a public fork out of fbs-tutorial, by the way, so that anyone who wishes to do similar stuff could look at this as another tutorial example.

[mherrmann]

Do you think I could get the cmdline.exe to be built in the gui directory?

I'm not sure; You could try PyInstaller's -y option, but I don't know if that deletes the existing directory first. If it does, I think you'll have to copy the two directories together in a separate step.

I'm going to make a public fork out of fbs-tutorial

Sounds good :-)

[fredrikaverpil]

Ok, I'll close this issue as I think I've got most questions answered ;) thanks again!

[mherrmann]

Glad I could help :-)

[fredrikaverpil]

One last question on this setup... 😉 about the core module in the example above...
-- It is being used by the gui app and the cmdline app. If I were to remove core, neither apps would run when running e.g. with python -m fbs run.

But after the freeze, core is bundled within the frozen binaries. So, the core module I'm copying into the build directory (before performing python -m fbs installer) is not being used by gui.exe or cmdline.exe.

Now, let's speculate that I would actually want the frozen gui.exe and cmdline.exe to import the manually copied core module and fail to run unless core exists as pure python files inside of my frozen target directory.

Do you think such a thing would be possible?

[fredrikaverpil]

Here's my fork: https://github.com/fredrikaverpil/fbs-tutorial/tree/multibundle

[mherrmann]

Yes, I would assume that if you delete the copy of core bundled by PyInstaller, then as your app starts it will automatically pick up the source version. If not, it would probably be enough if you appended your app's directory to sys.path.

[fredrikaverpil]

Hm. I know that all python files are bundled under the .app on macOS etc, but I'm on Windows right now. And when I look inside of target/Tutorial, I don't see a core folder anywhere (unless I have copied into this location myself).

But I'll try a sys.path.insert(0, PATH) and see if that will pick up my manually copied core module when running Tutorial.exe.

[fredrikaverpil]

Very cool, it worked by doing a sys.path.insert! 😄

[mherrmann]

😄

[fredrikaverpil]

I just hope all of this will work once auto-updating comes to fbs 😀 because it's awesome!

[mherrmann]

We'll make it work :)

[fredrikaverpil]

Hm, unfortunately, I was mistaken. I can't import the core module using sys.path.insert... trying to figure out a way to do it...

[mherrmann]

Are you sure? I would have thought sys.path.insert(0, os.path.dirname(sys.executable)) would work.

[fredrikaverpil]

Yes. But I may have found a way with importlib:

import sys
import os

sys.path.insert(0, os.path.join(os.path.dirname(sys.executable), 'modules'))

core = importlib.import_module('core')  # works, imports Tutorial/modules/core/__init__.py
# import core  # doesn't work, will import Tutorial/core/__init__.pyc

Hm. It's not very nice... as you're bound to forget to not just import core or do e.g. a from core import main which will then somehow revert core back to the frozen version...

[fredrikaverpil]

I can't seem to get this to work in the long run. Even if I import core as described in my previous post, a from core import main will "revert" the core module back into read the compiled/frozen version of core.

[mherrmann]

Why is it important to read from the "source copy" of core? Do users modify it?

[fredrikaverpil]

I have a Python API which I could bundle with the installer. It would be somewhat intuitive if the main apps (which uses this Python API) would actually use those files. But I guess it's not essential.