PyQt's qwindows.dll is corrupted by UPX

[bioinformatist]

What's the problem (or question)?

I try use PyInstaller to make a bundle of my program, when pack my codes and dependencies without calling UPX, it works well (both in one-folder or one-file mode). But things go wrong when I perform UPX compression on them.

What should have happened?

When I try run my program, it raises:

"This application failed to start because it could not find or load the Qt platform plugin "windows"
in "".

Reinstalling the application may fix this problem.

When PyInstaller is packing my codes, the parameter it calls UPX with is like:

Executing - upx --lzma -q C:\Users\USER\AppData\Roaming\pyinstaller\bincache01_py36_64bit\qt5widgets.dll

Do you have an idea for a solution?

I've check all related issue (no matter open & close) in this repo and pyinstaller's repo, and it brings me that may be there's something wrong with qwindows.dll. But I can find directory qt5_plugins\platforms which contains qwindows.dll in the location of my program.
I also try to add path (-p) to make it contain the dll file (path like C:\ProgramData\Anaconda3\Library\plugins\platforms) when use pyinstaller, but it not works as well.
And setting the run-time environment variable set QT_QPA_PLATFORM_PLUGIN_PATH="C:\ProgramData\Anaconda3\Library\plugins\platforms" or set QT_QPA_PLATFORM_PLUGIN_PATH="C:\Users\USER\Desktop\Gao-s-SB\src\gaosb\dist\main\qt5_plugins\platforms" not works, too, which just change the error message to something like this:

This application failed to start because it could not find or load the Qt platform plugin "windows"
in ""C:\ProgramData\Anaconda3\Library\plugins\platforms"".

Reinstalling the application may fix this problem.

Besides, I do use the latest PyInstaller in dev branch by pip install https://github.com/pyinstaller/pyinstaller/archive/develop.zip

How can we reproduce the issue?

1. You can get code from my repo first
2. In cmd, change the directory to where the main.py exists (Gao-s-SB/src/gaosb)
3. Run commands like `pyinstaller -y --clean -p ".." --add-binary "dependencies;dependencies" main.py
4. Change the directory to dist/main created just now and run the main.exe file. The error will occur.

Please tell us details about your environment.

• UPX version used (upx --version): Current latest version: 3.94
• Host Operating System and version: Win7 SP1 64bits
• Host CPU architecture: Intel x86_64
• Target Operating System and version: Win7 SP1 64bits
• Target CPU architecture: Intel x86_64

[markus-oberhumer]

Does it work with upx 3.91 ?

[bioinformatist]

No, UPX 3.91 does not work as well.

I have some new findings recently, when run pyinstaller in single-folder mode, with replacing the folder qt5_plugins in the package with the file in the folder of Python's interpreter, such as C:\ProgramData\Anaconda3\Library\plugins on my machine (I use Anaconda Python Distribution), then everything goes well. It seems qwindows.dll is not corrupted by UPX, but these plugins of pyqt5 do.

3.91 does not works, too.

[bioinformatist]

I have just tried to replace only qwindows.dll with unUPXed DLL file, everything works well. So it is convince that it is qwindows.dll that is corrupted by UPX.

[tycho]

I just dug into this, I think it's because UPX compresses some sections it shouldn't, such as .qtmetad, which I believe is how the Qt plugin system figures out it's a plugin:

SECTION HEADER #4
.qtmetad name
     120 virtual size
   A9000 virtual address (000000006C6A9000 to 000000006C6A911F)
     200 size of raw data
   A6A00 file pointer to raw data (000A6A00 to 000A6BFF)
       0 file pointer to relocation table
       0 file pointer to line numbers
       0 number of relocations
       0 number of line numbers
40600040 flags
         Initialized Data
         RESERVED - UNKNOWN
         RESERVED - UNKNOWN
         Read Only

RAW DATA #4
  000000006C6A9000: 51 54 4D 45 54 41 44 41 54 41 20 20 71 62 6A 73  QTMETADATA  qbjs
  000000006C6A9010: 01 00 00 00 FC 00 00 00 0B 00 00 00 E8 00 00 00  ....ü.......è...
  000000006C6A9020: 1B 03 00 00 03 00 49 49 44 00 00 00 3E 00 6F 72  ......IID...>.or
  000000006C6A9030: 67 2E 71 74 2D 70 72 6F 6A 65 63 74 2E 51 74 2E  g.qt-project.Qt.
  000000006C6A9040: 51 50 41 2E 51 50 6C 61 74 66 6F 72 6D 49 6E 74  QPA.QPlatformInt
  000000006C6A9050: 65 67 72 61 74 69 6F 6E 46 61 63 74 6F 72 79 49  egrationFactoryI
  000000006C6A9060: 6E 74 65 72 66 61 63 65 2E 35 2E 33 1B 0D 00 00  nterface.5.3....
  000000006C6A9070: 09 00 63 6C 61 73 73 4E 61 6D 65 00 19 00 51 4D  ..className...QM
  000000006C6A9080: 69 6E 69 6D 61 6C 49 6E 74 65 67 72 61 74 69 6F  inimalIntegratio
  000000006C6A9090: 6E 50 6C 75 67 69 6E 00 3A 40 A1 00 07 00 76 65  nPlugin.:@¡...ve
  000000006C6A90A0: 72 73 69 6F 6E 00 00 00 11 00 00 00 05 00 64 65  rsion.........de
  000000006C6A90B0: 62 75 67 00 15 16 00 00 08 00 4D 65 74 61 44 61  bug.......MetaDa
  000000006C6A90C0: 74 61 00 00 38 00 00 00 03 00 00 00 34 00 00 00  ta..8.......4...
  000000006C6A90D0: 14 03 00 00 04 00 4B 65 79 73 00 00 1C 00 00 00  ......Keys......
  000000006C6A90E0: 02 00 00 00 18 00 00 00 07 00 6D 69 6E 69 6D 61  ..........minima
  000000006C6A90F0: 6C 00 00 00 8B 01 00 00 0C 00 00 00 0C 00 00 00  l...............
  000000006C6A9100: A0 00 00 00 58 00 00 00 94 00 00 00 84 00 00 00  ....X...........
  000000006C6A9110: 2A 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  *...............

I'm not sure if just preserving this section is sufficient or not, though.

[Mattiwatti]

@tycho is correct, this is caused by the removal of the .qtmetad section. Since this section only provides metadata and no code, and also does not need relocations, it is even possible to fix qwindows.dll after compressing it by simply re-adding a section named .qtmetad with characteristics 0x50000040 and (on my installation) raw size 0x200 and virtual size 0x110. You can do this with e.g. CFF explorer. If you then copy back the contents from the original file in a hex editor, it will work again.

Okay... if that sounds like a tedious waste of time, that's because it is, so I don't recommend it. But it should help clarify the cause. I'm not sure why the Qt developers didn't simply use the existing resource directory for this, as it's already perfectly suitable for it and would probably even save a few bytes due to alignment. But then again, there's no law against making up weird PE sections for no good reason, so I won't judge...

Edit:
I've investigated this further by looking at the Qt source, and there is actually no need to have a .qtmetad section, which is strange considering they took the effort to add it. But the metadata is actually found by mapping the entire DLL into memory and scanning it for the magic "qTMETADATA " pattern (this is findPatternUnloaded() in src/corelib/plugin/qlibrary.cpp). If this magic value is not found, the DLL is rejected as a plugin candidate. If it is found, the entire file is still unmapped and loaded again later as a proper DLL image.

That's all for today's lesson on 'how not to find 200 bytes in your own DLL'...

[wangwenx190]

UPX 3.94, same problem.

[Guts]

Just to report that UPX 3.95, same problem.

[jreiser]

So, all you smart MSWindows hackers, where's the pull request for UPX to "ignore" PE section .qtmetad, probably by appending its contents (upon first encounter) to the end of the previous PE section, then splitting that PE section when writing the compressed result?
The best way to encourage someone else to work on the problem is to supply an actual literal test case: upload ("Attach files by ...") or provide the URL, or send as a Zip'ed attachment, some actual .dll file that fails, such as PyQt's qwindows.dll.

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days.

[github-actions]

This issue was closed because it has been stalled for 30 days with no activity. Please feel free to reopen.