Bug report: upx: actions-runner: vtkrenderingfreetype-9.2.dll

[MWTJC]

Things working fine when running in local terminal manually but not working when using GitHub-action's local executor: actions-runner (tested on both Nuitka 1.9.7 and 2.0 with same venv and para, actions-runner: 2.312.0, windows11):

Nuitka: Starting Python compilation with Nuitka '2.0' on Python '3.10' commercial grade 'not installed'.
......
Nuitka-Plugins:upx: Uncached file, compressing 'vtkrenderingfreetype-9.2.dll' may take a while.
Traceback (most recent call last):
  File "D:\actions-runner\_work\GBT_Calculator\GBT_Calculator\GBT_Calculator310\lib\site-packages\nuitka\plugins\Plugins.py", line 82, in withPluginProblemReporting
    yield
  File "D:\actions-runner\_work\GBT_Calculator\GBT_Calculator\GBT_Calculator310\lib\site-packages\nuitka\plugins\Plugins.py", line 585, in onCopiedDLLs
    plugin.onCopiedDLL(dll_path)
  File "D:\actions-runner\_work\GBT_Calculator\GBT_Calculator\GBT_Calculator310\lib\site-packages\nuitka\plugins\standard\UpxPlugin.py", line 142, in onCopiedDLL
    self._compressFile(filename=dll_filename, use_cache=not self.upx_nocache)
  File "D:\actions-runner\_work\GBT_Calculator\GBT_Calculator\GBT_Calculator310\lib\site-packages\nuitka\plugins\standard\UpxPlugin.py", line 129, in _compressFile
    copyFile(filename, upx_cache_filename)
  File "D:\actions-runner\_work\GBT_Calculator\GBT_Calculator\GBT_Calculator310\lib\site-packages\nuitka\utils\FileOperations.py", line 999, in copyFile
    shutil.copyfile(source_path, dest_path)
  File "C:\Python310\lib\shutil.py", line 254, in copyfile
    with open(src, 'rb') as fsrc:
OSError: [Errno 22] Invalid argument: 'dist\\qt_run.dist\\vtkrenderingfreetype-9.2.dll'
FATAL: upx: Plugin issue while working on 'DLL 'dist\qt_run.dist\vtkrenderingfreetype-9.2.dll''. Please report the bug with the above traceback included.
Nuitka-Reports: Compilation report written to file 'nuitka-crash-report.xml'.
Nuitka-Reports: Compilation crash report written to file 'nuitka-crash-report.xml'.

python -m nuitka --version

2.0
Commercial: None
Python: 3.10.9 (tags/v3.10.9:1dd9be6, Dec  6 2022, 20:01:21) [MSC v.1934 64 bit (AMD64)]
Flavor: CPython Official
Executable: D:\actions-runner\_work\GBT_Calculator\GBT_Calculator\GBT_Calculator310\Scripts\python.exe
OS: Windows
Arch: x86_64
WindowsRelease: 10
Version C compiler: D:\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.38.33130\bin\Hostx64\x64\cl.exe (cl 14.3).

• I am trying to build a python program through GitHub-action's local executor: actions-runner. The computer on which actions-runner is installed is Windows11: 22631.3007, and the relevant parts of the yml file are as follows:

jobs:
  build:
    runs-on: self-hosted
    steps:
        - name: checkout repo
          uses: actions/checkout@v3
             
        - name: venv
          run: |
            python -m venv GBT_Calculator310
            GBT_Calculator310\Scripts\Activate.ps1
            
        - name: install dependences
          run: |
             GBT_Calculator310\Scripts\pip install -r requirements.txt

        - name: Nuitka build (local)
          run: |
            GBT_Calculator310\Scripts\python -m nuitka --upx-binary=D:\\TOOLBOX\\upx-4.2.2-win64\\upx.exe --lto=yes --include-data-dir=./src/=src --standalone --windows-disable-console --output-dir=dist --enable-plugin=pyside6,upx qt_run.py

• That is: I installed a non-store version of python3.10 and told actions-runner to used venv to create a virtual environment for this build, and install the latest version of nuitka directly from pip install -r requirements.txt

requirements.txt:

PySide6
pyvista==0.43.2
pyvistaqt
vtk==9.2.2
loguru
pypinyin
pandas
datatable
misaka
nuitka

python -m pip freeze

......
pyvista==0.43.2
pyvistaqt==0.11.0
vtk==9.2.2
......

• I have no idea what line of code references vtkrenderingfreetype-9.2.dll, pyvista seems to be based on vtk.

import pyvista as pv
from pyvistaqt import QtInteractor

para:

--upx-binary=D:\\TOOLBOX\\upx-4.2.2-win64\\upx.exe 
--lto=yes 
--include-data-dir=./src/=src 
--standalone 
--windows-disable-console 
--output-dir=dist 
--enable-plugin=pyside6

crash-report.xml:

......
<module name="pyvista.core.utilities.geometric_sources" kind="CompiledPythonModule" usage="import" reason="Instructed by user to follow to all modules." source_path="${sys.prefix}\lib\site-packages\pyvista\core\utilities\geometric_sources.py" distribution="pyvista">
    <optimization-time pass="1" time="0.27" />
    <optimization-time pass="2" time="0.04" />
    <distribution-usages>
      <distribution-usage name="numpy" />
      <distribution-usage name="vtk" />
    </distribution-usages>
    <module_usages>
      <module_usage name="vtkmodules.vtkRenderingFreeType" finding="absolute" line="9" />
      ......
</module>
......
<module name="pyvista.plotting._vtk" kind="CompiledPythonModule" usage="import" reason="Instructed by user to follow to all modules." source_path="${sys.prefix}\lib\site-packages\pyvista\plotting\_vtk.py" distribution="pyvista">
    <optimization-time pass="1" time="0.09" />
    <optimization-time pass="2" time="0.01" />
    <distribution-usages>
      <distribution-usage name="vtk" />
    </distribution-usages>
    <module_usages>
      ......
      <module_usage name="vtkmodules.vtkRenderingFreeType" finding="absolute" line="116" />
      ......
</module>
......
<module name="vtkmodules.all" kind="CompiledPythonModule" usage="plugin:implicit-imports" reason="Instructed by user to follow to all modules." source_path="${sys.prefix}\lib\site-packages\vtkmodules\all.py" distribution="vtk">
    <optimization-time pass="1" time="0.27" />
    <optimization-time pass="2" time="0.02" />
    <module_usages>
      ......
      <module_usage name="vtkmodules.vtkRenderingFreeType" finding="relative" line="22" />
      ......
</module>
......
<module name="vtkmodules.vtkRenderingFreeType" kind="PythonExtensionModule" usage="import" reason="Extension module needed for standalone mode." source_path="${sys.prefix}\lib\site-packages\vtkmodules\vtkRenderingFreeType.cp310-win_amd64.pyd" distribution="vtk">
    <optimization-time pass="1" time="0.01" />
    <optimization-time pass="2" time="0.00" />
    <module_usages />
  </module>
......
  <included_dll name="vtkrenderingfreetype-9.2.dll" dest_path="vtkrenderingfreetype-9.2.dll" source_path="${sys.prefix}\lib\site-packages\vtkmodules\vtkrenderingfreetype-9.2.dll" package="vtkmodules" ignored="no" reason="Used by 'vtkmodules\vtkChartsCore.pyd'" />
......

• I think there may be something wrong with the environment variable in actions-runner that causes this bug, but I don't know how to fix it...
  thanks.

[MWTJC]

Sorry to disturb, I found my ignorance blinded me, I fixedit by adding echo PATH=$PATH >> $GITHUB_ENV when creating venv.

[kayhayen]

You ought to use Nuitka-Action, as an aside. https://github.com/Nuitka/Nuitka-Action

With wrong PATH I would have expected it to not find upx at all and error exit with an absence message, since it checks that binary to exist. To me OSError: [Errno 22] Invalid argument might be AV or UPX holding a lock after execution still, but I am not sure of that. In that case, it might be a race. Need to see if it repeats. Your machine might do strange things there, and we have used extra balls for actions subjected to AV before.

[kayhayen]

The AV interfering, much like it does with C compile build results, would make the most sense. Due to caching, it probably also becomes very unlikely to happen, so no urgency there, esp. as the UPX plugin is not the most important of them all.

[MWTJC]

Thank you very much.
Due to hardware problems, Hyper-V cannot be started (As an aside: wsl2 does not boot but wsl1 does on my server) on my server with actions-runner installed. Therefore, when I actually use Nuitka/Nuitka-Action@v1.1, the following error will occur in github action:

Run Nuitka/Nuitka-Action@v1.1
  with:
    nuitka-version: main
    script-name: qt_run.py
    standalone: true
    enable-plugins: pyside6
    working-directory: .
    onefile: true
    assume-yes-for-downloads: true
    output-dir: build
Run echo "NUITKA_CACHE_DIR=D:\actions-runner\_work\_actions\Nuitka\Nuitka-Action\v1.1/nuitka/cache" >> $GITHUB_ENV
  echo "NUITKA_CACHE_DIR=D:\actions-runner\_work\_actions\Nuitka\Nuitka-Action\v1.1/nuitka/cache" >> $GITHUB_ENV
  echo "PYTHON_VERSION=$(python --version | awk '{print $[2](https://github.com/MWTJC/GBT_Calculator/actions/runs/7735414040/job/21091036565#step:2:2)}' | cut -d '.' -f 1,2)" >> $GITHUB_ENV
  shell: C:\Windows\System[3](https://github.com/MWTJC/GBT_Calculator/actions/runs/7735414040/job/21091036565#step:2:3)2\bash.EXE --noprofile --norc -e -o pipefail {0}
<3>WSL (230) ERROR: CreateProcessEntryCommon:[5](https://github.com/MWTJC/GBT_Calculator/actions/runs/7735414040/job/21091036565#step:2:5)02: execvpe /bin/bash failed 2
<3>WSL (230) ERROR: CreateProcessEntryCommon:505: Create process not expected to return
Error: Process completed with exit code 1.

So I had to run nuitka like a local command when using actions-runner.
In addition, I found that the upx compression failure I proposed was somewhat random. After further investigation, I found that it was related to windows defender, which would react to part of nuitka's upx cache and part of the dll after nuitka's upx compression. I guess a possible solution is for nuitka users to manually add upx compression exceptions to avoid build failures or windows defender reacting to built files for delivery while retaining some volume compression. I am trying to package the build files for delivery using NSIS. However, it is still impossible to avoid the large file size of the program after installation.
Thanks.

You ought to use Nuitka-Action, as an aside. https://github.com/Nuitka/Nuitka-Action

With wrong PATH I would have expected it to not find upx at all and error exit with an absence message, since it checks that binary to exist. To me OSError: [Errno 22] Invalid argument might be AV or UPX holding a lock after execution still, but I am not sure of that. In that case, it might be a race. Need to see if it repeats. Your machine might do strange things there, and we have used extra balls for actions subjected to AV before.

The AV interfering, much like it does with C compile build results, would make the most sense. Due to caching, it probably also becomes very unlikely to happen, so no urgency there, esp. as the UPX plugin is not the most important of them all.

[kayhayen]

Your runner failing like that in the caching part, a well established component, it seems to be the problem with your runner, so I won't care much. I will handle my theory there, that some sort AV caused permission issue can exist for upx just like when we update resources, but I do not consider looking into your details which are entirely unreproducible to me and indicate a very strange setup with some sorts of limitations not commonly found.

[MWTJC]

Yes, my usage scenario is far removed from the general usage scenario, so I will consult relevant documents to understand relevant knowledge first or try to use more usual usage to avoid this particular failure I encountered.
Thanks.

[MWTJC]

Sorry for the waste of time, here is a follow-up note that has nothing to do with nuitka:
For Github action, everything works fine;
For my Windows actions-runner without sh support (not working even after using Vampire/setup-wsl), I solved the environment variable problem by using Poetry and replaced every python run with poetry run , and circumvented the upx plug-in problem by adding exclusion list in AV.
Thanks.
Shall I close this issue as not planned?

[kayhayen]

If it's not already clear, I am ignoring your side of the issue here entirely. What I mean to do for this issue still, once I find the time of course, is to give the upx compressor extra balls when copying the compressed DLLs, since that is a theoretically true problem with Windows and AV. Your side of things is as invalid as it gets, but I don't care much in which ways your system is broken or repaired after knowing it is broken and your result is being invalid. That said, the problem I considered possible, I will try and make it solved still, but not now.

[MWTJC]

I see.
All the best.
Thanks.