Repository navigation
Feature: Splash screen based on Tcl/Tk - #4887
Conversation
There was a problem hiding this comment.
Looks good! This is quite a lot of work, so thanks for doing this!
Travis is failing to compile the bootloader, so that'll need fixing. I think it's a linux problem.
Of course I will add documentation/changelog in the next days and squash/reword the commits
Understood. We don't nitpick on those until PRs are marked as ready to review.
Other than that, I just have one thing to ask about...
Legorooj
left a comment
There was a problem hiding this comment.
Looks good. I've got another couple of things I'd like to ask about. The commit messages are, as you said they'd be, nice and neat. I've got a quick-to-fix changelog entry nitpick, which I'll push myself, but otherwise this looks good! I'd like a review from @htgoebel or @bjones1, as my C skills aren't nearly as good as theirs or yours (IE I can understand the C code you've written, but I wouldn't be able to write something that complex myself - I only learned C a few months ago).
Thanks for such a major feature, with clean and easy to understand code!
| try: | ||
| # Test if a module named 'pyi_splash' is locally installed. | ||
| # This prevents that a potentially required dependency is not | ||
| # packed | ||
| import pyi_splash # noqa: F401 | ||
| except ImportError: | ||
| module_dir = os.path.join(PACKAGEPATH, 'fake-modules') | ||
|
|
||
| api.search_dirs = [module_dir] | ||
| logger.info('Adding pyi_splash module to application dependencies.') | ||
| else: | ||
| logger.info('A local module named "pyi_splash" is installed. ' | ||
| 'Use the installed one instead.') | ||
| return |
There was a problem hiding this comment.
My idea behind this check was, that I wanted to prevent issues in the future, where maybe someone installed a package named like pyi_splash (not necessarily a module to control the splash screen, but maybe a name collision with another project), which would be overwritten by our module. If such a module is installed I assumed the user explicitly installed and wanted to use it.
There was a problem hiding this comment.
Maybe we could change this to something like PyInstaller.pyi_splash? Update the hooks and rewrite the module a bit so that it just does nothing when unfrozen - so you can call it in unfrozen code. @bjones1 do you have an opinion?
There was a problem hiding this comment.
As I worked on the pyi_splash module I had the same idea (naming it like PyInstaller.pyi_splash), but I discarded that idea, because I thought that would make the PyInstaller Python API somewhat inconsistent, since the "general" API is for building and that one module is for a user application. But the more I keep thinking of renaming the module the more I like the idea.
There was a problem hiding this comment.
So, the core question is: where, conceptually, does the pyi_splash module belong? From what I understand, it is a module that will only be available in a frozen application and enables the frozen application to control the splash screen. The possibilities:
- It is a fake module -- it replaces an existing system modules, such as
sites. But this isn't correct, since it's not (or shouldn't be) overriding a system module. - It is a module that should be included as part of the modulegraph search process. However, it can't operate outside the frozen executable, so this isn't it.
- It's a part of the loader, used to get the frozen Python environment bootstrapped. While another part of this PR does reside in the bootloader, this module isn't required for bootstrapping.
- It's a part of the PyInstaller build process. No, because it runs when the frozen executable runs.
- It's a run-time hook. These help set up the frozen environment for specific modules, and belong only in that context. While it's not required for splash screen operation (or is it -- will the splash screen close on its own?), this seems the closest description of what it does.
So, I think the name should be pyi_rth_splash.py or something similar, and it listed as a run-time hook.
There was a problem hiding this comment.
I thought about putting the module into the runtime-hook too, but I had some problems implementing it as one:
-
the runtime hook would need to be executed before the users script and after all other boot script, so it would need to be inserted into
Analysis.scriptsat a specific position after Analysis has finished. I think that would break very easily. -
To still provide the module with its current name the script would need to modify
sys.modulesand do some weird loading. I think that would break very easily too.
I somewhat explained my choice here in the previous PR.
Maybe you have a better way to implement this on the python side, but I didn't find any maintainable/satisfying way.
There was a problem hiding this comment.
I'm happy to leave it as a non-runtime hook if that fits more nicely. Ispyi_splash a module that the user imports to control the splash screen, which can only be imported/used in a frozen executable, and should therefore be included in the frozen executable?
There was a problem hiding this comment.
The pyi_splash module is to control the splash screen from within a frozen application. If someone tries to import it from a non frozen python interpreter the module cannot be found. So it only serves a purpose if the application is in a frozen state. If if it is not included in the frozen application and the bootloader starts a splash screen, there will be no way to close it.
|
@Chrisg2000 I've come across a slight problem testing it out locally: As the warnings suggest, the splash screen doesn't work. However, I inserted the following diff: @@ -215,11 +215,13 @@ class Splash(Target):
self.binaries.extend(toc for toc in tk_tree
if toc[0] in self.splash_requirements)
# Check if all requirements were found
def _filter(_item):
- if _item not in (datas + self.binaries).filenames:
+ fnames = (datas + self.binaries).filenames
+ if _item not in fnames:
+ print(_item.lower() in (x.lower() for x in fnames))
# Item is not bundled, so warn the user about it.
# This actually may happen on some tkinter installations
# on which there is no license.terms file
logger.warning("The local Tcl/Tk installation is missing"
" the file %s. The behavior of the splash"Which gave me the following output: I then changed the splash source again: @@ -215,11 +215,12 @@ class Splash(Target):
self.binaries.extend(toc for toc in tk_tree
if toc[0] in self.splash_requirements)
# Check if all requirements were found
def _filter(_item):
- if _item not in (datas + self.binaries).filenames:
+ fnames = (datas + self.binaries).filenames
+ if _item.lower() not in (x.lower() for x in fnames) and _item not in fnames:
# Item is not bundled, so warn the user about it.
# This actually may happen on some tkinter installations
# on which there is no license.terms file
logger.warning("The local Tcl/Tk installation is missing"
" the file %s. The behavior of the splash"Hey presto, warnings gone. But the splash still doesn't work: Any ideas? I've tried everything I can think of. Oh and the splash section of the spec is: I tweaked it a bit of course. |
|
@Legorooj Good catch there! I found the reason why those files were excluded: The TOC class only puts pyinstaller/PyInstaller/building/datastruct.py Lines 23 to 39 in 7eb1c66 So my solution is then to not use the @@ -216,8 +216,10 @@ class Splash(Target):
if toc[0] in self.splash_requirements)
# Check if all requirements were found
+ fnames = [toc[0] for toc in (datas + self.binaries)]
+
def _filter(_item):
- if _item not in (datas + self.binaries).filenames:
+ if _item not in fnames:
# Item is not bundled, so warn the user about it.
# This actually may happen on some tkinter installations
# on which there is no license.terms fileThis resolves the problem of the excluded files, but I am not sure what causes the bootloader to not find the resources which are appended, since you should at least see a message with the prefix |
|
The tests fail. I've also rebuilt the bootloaders - with MSVC 2019. The pytest output of the tests is here. I'm attempting to debug now... |
|
I can't help myself, but I still have the feeling that it has to do with the compilation of the bootloader, since the pytest log should contain messages with the prefix An idea to debug the problem would be to modify the 1At least the message |
Legorooj
left a comment
There was a problem hiding this comment.
You were right. I hadn't built the bootloaders properly - I'd only built the 64bit ones for some reason. It all works - well works ish. There's 2 bugs. I've left a comment for one below - and a fix. It's quite trivial.
The other is I tried using the splash with a very high definition photo - 3k. The result was that the splash worked fine - it just splashed across the entire screen, covering everything, and I could only see a quarter of the image cause most if was off the screen. This needs fixing with a maximum size for the image.
I also got the opposite result when using the test image; it was tiny and I could barely see it.
|
As a quick (personal) note: I will be off the grid for the next two weeks and won't work/comment on this PR. |
|
The thing about the huge/small images: I had the concern, that such images would be used, but there is no real way to deal with that, except scaling the image. In a test version I implemented a bilinear image scaling function in the bootloader, which scaled the image to a specific portion of the screens width at runtime (e.g. scaling the image up to be 40% of the screens width). My implementation unfortunately produced some image artifact, so I discarded the idea. Another suggestion I could think about is to add an option to the |
|
@Chrisg2000 maybe. I'll look into it more. And take whatever time you need. |
|
So, back online I changed some minor things:
The |
|
Thanks for the changes. I'll check them out in the next day or 2. Maybe we can get this merged in for the 4.0 release; you never know :-). |
There was a problem hiding this comment.
Can't see anything wrong with this and it's working locally! Really awesome work on this!
Just one nitpick: Please add:
A huge thanks to `@Chrisg2000 <https://github.com/Chrisg2000>`_for programming this feature.Into the changelog entries.
Other than that, @htgoebel and @bjones1 please review this amazing peice of work.
What am I doing wrong? |
Looks like you have 2 single quotes. Python takes a single quote as the start of a string. So you're giving it an empty string Try |
|
Sorry, I think I was a bit unclear/out of context. |
If Pillow is missing and image needs to be resized, raise a ValueError instead of None, to avoid a more confusing error further down the road. Similarly, if lack of Pillow precludes image conversion, mention that in the error message to make it clear what needs to be done to resolve the error.
It turns out that Tcl/Tk on macOS (or rather, the underlying UI
toolkit) disallows UI operations from any thread other than
the main one. This makes the current splash screen implementation,
which operates in a secondary thread, inherently incompatible with
macOS.
The frozen application with splash screen enabled crashes when
calling Tk_Init() in the splash screen thread:
```
2021-05-13 19:37:02.116 program[3394:49760] *** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'NSWindow drag regions should only be invalidated on the Main Thread!'
*** First throw call stack:
(
0 CoreFoundation 0x00007fff205f45bf __exceptionPreprocess + 242
1 libobjc.A.dylib 0x00007fff2032cbbb objc_exception_throw + 48
2 CoreFoundation 0x00007fff2061cdae -[NSException raise] + 9
3 AppKit 0x00007fff22cc9872 -[NSWindow(NSWindow_Theme) _postWindowNeedsToResetDragMarginsUnlessPostingDisabled] + 321
4 AppKit 0x00007fff22cb444a -[NSWindow _initContent:styleMask:backing:defer:contentView:] + 1296
5 AppKit 0x00007fff22cb3f33 -[NSWindow initWithContentRect:styleMask:backing:defer:] + 42
6 libtk8.6.dylib 0x0000000107ff3e61 TkMacOSXMakeRealWindowExist + 757
7 libtk8.6.dylib 0x0000000107ff3a3e TkWmMapWindow + 63
8 libtk8.6.dylib 0x0000000107f587ec Tk_MapWindow + 158
9 libtk8.6.dylib 0x0000000107f611f7 MapFrame + 59
10 libtcl8.6.dylib 0x0000000107e5a6d8 TclServiceIdle + 75
11 libtcl8.6.dylib 0x0000000107e3efd0 Tcl_DoOneEvent + 340
12 libtk8.6.dylib 0x0000000107fe6944 TkpInit + 766
13 libtk8.6.dylib 0x0000000107f59a37 Initialize + 2411
14 program 0x0000000106f87f31 program + 20273
15 libsystem_pthread.dylib 0x00007fff20481954 _pthread_start + 224
16 libsystem_pthread.dylib 0x00007fff2047d4a7 thread_start + 15
)
libc++abi: terminating with uncaught exception of type NSException
Abort trap: 6
```
Therefore, explicitly disable building with splash screen on macOS
by raising SystemExit in Splash constructor, following the same
behavior as if tkinter is missing. Also add/adjust notes about
lack of macOS support throughout the docs.
|
Alright, this PR is now rebased back on top of I've also fixed the race condition at exit that I mentioned before, and improved the error messages when Unfortunately, the current design that has splash screen running in a secondary thread does not work on macOS at all, because the UI framework there disallows UI-related operations from any thread but the main one. So the frozen application crashes when bootloader tries to initialize Tk. Therefore, trying to enable splash screen in macOS will now raise SystemExit error, and the lack of macOS support is documented in the affected parts of documentation. I suppose in follow-up work, we could try pushing archive unpacking into separate thread, and have splash screen displayed in the main thread, then tear it down once we're ready to start the embedded python interpreter. This will limit the functionality compared to the current one (right now, it is possible to interact with splash screen from the frozen python code), but should be compatible across all OSes. But I'd rather see this done afterwards, and have this merged now as-is to prevent it from falling out of sync with |
|
Alright! Here goes... |
|
I'm confused by this message about macOS. I've been using the splash screen for months now on Mac with no problem. Did this break recently? |
|
What macOS version and what python version/distribution were/are you using? |
|
Right now I'm on Catalina 10.15.7 with Python 3.7.9. I believe this was working on older versions too. I've been installing from this git pull request, but haven't updated in a few weeks. |
|
Is that python.org python, or homebrew? |
|
Sorry I don't remember where I installed from. Probably python.org judging by the path: |
|
Can you check |
|
Can you try installing the And for the reference, here the backup of this branch before I pushed the rebased version. But I get the same behavior with both the original and the rebased version (although it could be a problem with my system...). |
|
I do have python@3.8 and python@3.9 installed with homebrew but they are in the Cellar. In this example I used a venv pointing to Python3.7 which is not homebrew (I can tell from the |
|
Okay I tried the penultimate commit of the develop branch and it's not working. Looks like the same kind of errors as mentioned in that last commit. I also get this error: 291 WARNING: The local Tcl/Tk installation is missing the file tk/license.terms. The behavior of the splash screen is therefore undefined and may be unsupported. I tried going back a few commits but I cannot get it to work now from the develop branch. This was definitely working in the past, and it still does work in a different venv. I'm trying to figure out what's different. |
|
Can you get it running with the original version? As I mentioned, I made a backup: https://github.com/rokm/pyinstaller/tree/splash-feature-v2-backup But searching around, I found mentions of this warning: |
|
Ah, I see your version of "NSWindow drag regions should only be invalidated on the Main Thread!" is technically still a warning, whereas in my case, it is uncaught exception. |
|
Also, if you have a venv where things still work, could you make a simple hello world example so I can try running it on my test system? |
|
Shoot, I accidentally deleted that venv so now I'm stuck. In fact now that I think about it, I may have been using the system installation of PyInstaller before, not the venv. I've tried installing from your backup branch, and from this PR, and from ChrisG2000's repo, but I can't get anything to work. I also tried upgrading to 3.9.0 from python.org and that made no difference. Using your backup branch I get this error: I spent about 2 hours debugging this, without success. |
|
Just to clarify, chrisg2000 repo branch that was used to open this PR is the same as develop now, because I had to force-push the rebased version into it to have the PR updated. So that and develop are bound to behave the same. My backup branch is the same as what this PR was before the rebase was pushed. And the above error is quite expected, because at the time when the PR was open, the state of Tcl/Tk support on macOS in PyInstaller was rather messy. So if you got it working, it must have been a very specific combination that managed to get you past the errors in Tcl/Tk collection itself. (But I think that's a separate problem from main-thread enforcement from UI toolkit, which probably also came in with one of system updates). |
|
I did some more testing, and it looks like that UI-in-main-thread enforcement came as part of graphics system overhaul in macOS 10.14 Mojave. But whether it is just a warning or an actual exception depends on the version of macOS SDK that the program (in our case, PyInstaller's bootloader) is built against (or, rather, the version that the program claims to be built against). If the declared macOS SDK version is old enough (e.g., 10.9), the offending program gets just a warning. If it is 10.14 or newer, it gets an exception. We recently introduced the reported SDK version adjustment to Python.org 3.9 universal2/macos11 variant comes with Tcl/Tk 8.6.10 instead, but is also built against SDK 11.0, so with bootloaders rebuilt with XCode tools 12.x, the frozen binary reports 11.0 SDK, and terminates with uncaught exception. But, if I manually adjust the reported SDK version to 10.9, it produces just a warning and displays the splash screen. Dropping reported SDK version below 11.x opts us out of Big Sur specific features, though (like dynamic library cache searching for discovery of system libraries, version being reported as 11.x instead of 10.16). The So my guess would be that if you managed to get the original branch working, you were working with recent Tcl/Tk (which rules out python.org builds with their 8.6.8). And when you were rebuilding bootloaders, you were using toolchain (XCode version) with SDK version lower than 10.14. (Or alternatively, the UI-in-main-thread enforcement came with one of system updates, and you were testing before that happened). Considering all the above, I think I stand by my decision to disable splash screen on macOS for now. At least until it is reworked to operate in the main thread... and even then we will probably keep running afoul of that Tcl/Tk 8.6.8 that comes with python.org builds. |
|
my pyi_splash.update_text() doesn't work the splash part of my .spec file: compared with the test .spec file (test_pyi_splash.py works) i guess pyinstaller did not add those lines to my spec file |
From the docs: |
|
Is it possible that the splash window on loading is not on the top the system window? |
|
Splash screen is great for slow loading apps, however, static image is misleading as it does not convey progress. Allowing a GIF or APNG support for splash screen images would be better. Tkinter seems to support GIF by cycling through frames, e.g. https://stackoverflow.com/questions/28518072/play-animations-in-gif-with-tkinter. |
|
If you have a feature request related to splash screens, please feel free to open a new issue. |
Edit: Note
Original post
So, following on #4581 now a new implementation of the splash screen based on Tcl/Tk (the same library that tkinter uses). The idea of the splash screen came from #4354.
This change implements a splash screen based on Tcl/Tk, which is bundled by PyInstaller at compile time. The splash screen is now (unlike #4581) able to show the progress when unpacking a one-file archive.
The bootloader extracts the necessary files in onefile mode and starts a minimal Tcl/Tk environment in which a script is executed that starts the splash screen. Although this feature is platform independent, I have only tested it on Windows.
For the communication between Python and the splash screen the pyi_splash Python module is (again) bundled as fake modules. This has not changed much compared to #4581.
Of course I will add documentation/changelog in the next days and squash/reword the commits.