i686-w64-mingw32 build crashes as of v1.4.10

[tomasz1986]

Relevant components

• Standalone tray application (based on Qt Widgets)
• Plasmoid/applet for Plasma desktop
• Dolphin integration
• Command line tool (syncthingctl)
• Integrated Syncthing instance (libsyncthing)
• Backend libraries

Environment and versions

• Versions of syncthingtray, qtutilities and c++utilities: 1.4.10/11, N/A, N/A
• Qt version: 6.6.1
• C++ compiler (name and version): N/A
• C++ standard library (name and version): N/A
• Operating system (name and version): Windows 10 Enterprise LTSC 2019 x86

Bug description

Syncthing Tray used to work with no issues up to v1.4.9. However, when trying to run v1.4.10 or v1.4.11, the program only shows up in background for 1-2 seconds, and then it crashes immediately with no graphical output. Nothing is recorded in the Windows Event Viewer too.

I suspect that the issue may be related to the specific version of the OS which is Windows 10 Enterprise LTSC 2019 x86. However, Qt6 does state that they support "Windows 10 (1809 or later)" which is exactly the version here. On the other hand, the problem may also be related to the OS being 32-bit, however I've got no other device running Windows x86 to confirm the suspicion. I may try doing it on a 32-bit VM later on if necessary.

Edit: The Qt5 versions crashes the same way, so it's not about Qt.

Steps to reproduce

1. Run Syncthing Tray v1.4.9 on Windows 10 v1809 x86.
2. The program starts with no problems.
3. Close the program completely.
4. Try running Syncthing Tray v1.4.10 or v1.4.11.
5. The program starts in background and immediately crashes with no output.

Expected behavior

The program should open and run as is the case with v1.4.9 and older versions.

Screenshots

Additional context

[GeB666]

Same for me on a new install. Appears for a moment then immediately crashes
Windows 10 home
Version 22H2
OS build 19045.3803

[tomasz1986]

@GeB666 Is your OS also 32-bit?

[GeB666]

@tomasz1986 No, its 64-bit

[Martchus]

@GeB666 Ok, so your OS is 64-bit. And what version of Syncthing Tray are you using (32-bit or 64-bit)? Considering @tomasz1986 filed #217 I assume the 64-bit version is generally working. It also works for me. So your issue might be a different one and you need to provide more information.

@tomasz1986 Can you use the CLI (e.g. to print the help text or to use the CLI of the built-in Syncthing)? Does it show anything in the terminal when starting from a terminal via the CLI wrapper? (If you use MSYS2 you can also just use the normal binary instead of the CLI wrapper.)

Note that the 32-bit version requires an i686 CPU (and wouldn't work if you only have e.g. i586).

[Martchus]

Looks like the v1.4.11 i686 release crashes in WINE when opening the about dialog:

Unhandled exception: illegal instruction in wow64 32-bit code (0x023b92b2).
Register dump:
 CS:0023 SS:002b DS:002b ES:002b FS:0063 GS:006b
 EIP:023b92b2 ESP:03f09980 EBP:03f09c10 EFLAGS:00010206(  R- --  I   - -P- )
 EAX:00000005 EBX:03f09a64 ECX:00000000 EDX:000000b9
 ESI:0233aaba EDI:03f09ae4
Stack dump:
0x03f09980:  0000027c 00000010 0000000f 03f09fff
0x03f09990:  0000027c 00000010 00000010 ffffffff
0x03f099a0:  0000027c 00000010 00000011 020021ff
0x03f099b0:  0000027c 00000010 00000012 000000ff
0x03f099c0:  0000027c 00000010 00000013 000000ff
0x03f099d0:  0000027c 00000010 00000014 00000000
Backtrace:
=>0 0x023b92b2 in syncthingtray-1.4.11-i686-w64-m (+0x1fb92b2) (0x03f09c10)
  1 0x036ebc50 in syncthingtray-1.4.11-i686-w64-m (+0x32ebc50) (0x10fb7070)
  2 0x474e5543 (0x432b2b00)
0x023b92b2 syncthingtray-1.4.11-i686-w64-m+0x1fb92b2: ud2
Modules:
Module  Address                 Debug info      Name (156 modules)
PE        400000- 3d0c000       Export          syncthingtray-1.4.11-i686-w64-mingw32
ELF     6a000000-7231b000       Deferred        libllvm.so.17
ELF     75a00000-77ccd000       Deferred        iris_dri.so
ELF     77dca000-77e00000       Deferred        libtinfo.so.6
ELF     799bd000-799e6000       Deferred        libdrm_intel.so.1
ELF     79ae6000-79b20000       Deferred        libedit.so.0
ELF     79c62000-79c80000       Deferred        libelf.so.1
ELF     79c80000-79d0a000       Deferred        libglx_mesa.so.0
ELF     79d0a000-79d85000       Deferred        libgldispatch.so.0
ELF     79d85000-79dbf000       Deferred        libglx.so.0
ELF     79dbf000-79e30000       Deferred        libgl.so.1
PE-Wine 79f30000-79f3d000       Deferred        powrprof
PE-Wine 79f40000-79f4a000       Deferred        bcryptprimitives
PE-Wine 79f50000-79f62000       Deferred        msv1_0
PE-Wine 79f70000-79f7e000       Deferred        kerberos
PE-Wine 79f80000-79f97000       Deferred        winex11
PE-Wine 79fa0000-79fae000       Deferred        wtsapi32
PE-Wine 79fb0000-79fc7000       Deferred        jsproxy
PE-Wine 79fd0000-7a00c000       Deferred        winhttp
PE-Wine 7a010000-7a01d000       Deferred        version
PE-Wine 7a020000-7a054000       Deferred        uxtheme
PE-Wine 7a060000-7a06e000       Deferred        userenv
PE-Wine 7a070000-7a0d6000       Deferred        setupapi
PE-Wine 7a0e0000-7a0ff000       Deferred        secur32
PE-Wine 7a100000-7a1fd000       Deferred        oleaut32
PE-Wine 7a200000-7a224000       Deferred        netapi32
PE-Wine 7a230000-7a23f000       Deferred        ncrypt
PE-Wine 7a240000-7a24d000       Deferred        nsi
PE-Wine 7a250000-7a265000       Deferred        dnsapi
PE-Wine 7a270000-7a294000       Deferred        iphlpapi
PE-Wine 7a2a0000-7a312000       Deferred        dwrite
PE-Wine 7a320000-7a32e000       Deferred        dwmapi
PE-Wine 7a330000-7a362000       Deferred        d3d9
PE-Wine 7a370000-7a386000       Deferred        d3d12
PE-Wine 7a390000-7a4a1000       Deferred        opengl32
ELF     7a4a3000-7a4b0000       Deferred        libpciaccess.so.0
PE-Wine 7a4b0000-7a73a000       Deferred        wined3d
PE-Wine 7a740000-7a774000       Deferred        dxgi
PE-Wine 7a780000-7a7da000       Deferred        d3d11
PE-Wine 7a7e0000-7a8aa000       Deferred        crypt32
PE-Wine 7a8b0000-7a8c6000       Deferred        compstui
PE-Wine 7a8d0000-7a900000       Deferred        winspool
PE-Wine 7a900000-7a917000       Deferred        shcore
PE-Wine 7a920000-7a967000       Deferred        shlwapi
PE-Wine 7a970000-7b27e000       Deferred        shell32
PE-Wine 7b280000-7b29b000       Deferred        imm32
PE-Wine 7b2a0000-7b3e4000       Deferred        comctl32
PE-Wine 7b3f0000-7b4eb000       Deferred        comdlg32
PE-Wine 7b4f0000-7b506000       Deferred        bcrypt
PE-Wine 7b510000-7b51c000       Deferred        authz
PE-Wine 7b520000-7b546000       Deferred        ws2_32
PE-Wine 7b550000-7b565000       Deferred        coml2
ELF     7b565000-7b570000       Deferred        libdrm_nouveau.so.2
PE-Wine 7b570000-7b5ee000       Deferred        rpcrt4
PE-Wine 7b5f0000-7b638000       Deferred        combase
PE-Wine 7b640000-7b740000       Deferred        ole32
PE-Wine 7b740000-7b770000       Deferred        win32u
PE-Wine 7b770000-7b7ef000       Deferred        gdi32
PE-Wine 7b7f0000-7b996000       Deferred        user32
PE-Wine 7b9a0000-7b9c1000       Deferred        msacm32
ELF     7b9c2000-7b9d0000       Deferred        libdrm_amdgpu.so.1
PE-Wine 7b9d0000-7ba9b000       Deferred        ucrtbase
PE-Wine 7baa0000-7bac6000       Deferred        sechost
PE-Wine 7bad0000-7bb6f000       Deferred        msvcrt
PE-Wine 7bb70000-7bbae000       Deferred        advapi32
PE-Wine 7bbb0000-7bc70000       Deferred        winmm
PE-Wine 7bc70000-7bef7000       Deferred        kernelbase
PE-Wine 7bf00000-7bf60000       Deferred        kernel32
PE-Wine 7bf60000-7bfff000       Deferred        ntdll
ELF     7c004000-7c013000       Deferred        libdrm_radeon.so.1
ELF     7c013000-7c01e000       Deferred        libxcb-xfixes.so.0
ELF     7c01e000-7c027000       Deferred        libxcb-sync.so.1
ELF     7c027000-7c051000       Deferred        libgcc_s.so.1
ELF     7c057000-7c07d000       Deferred        libglapi.so.0
ELF     7c07d000-7c100000       Deferred        libvulkan.so.1
ELF     7c304000-7c309000       Deferred        libxcb-present.so.0
ELF     7c309000-7c323000       Deferred        libdrm.so.2
ELF     7c323000-7c328000       Deferred        libx11-xcb.so.1
ELF     7c328000-7c32f000       Deferred        libxcb-dri3.so.0
ELF     7c36a000-7c37d000       Deferred        libxcb-randr.so.0
ELF     7c37d000-7c39b000       Deferred        libxcb-glx.so.0
ELF     7c39b000-7c3a0000       Deferred        librt.so.1
ELF     7cb02000-7cb07000       Deferred        libxshmfence.so.1
ELF     7cb07000-7cb0c000       Deferred        libxcb-shm.so.0
ELF     7cb0c000-7cb13000       Deferred        libxcb-dri2.so.0
ELF     7cb23000-7cb28000       Deferred        msv1_0.so
ELF     7cb28000-7cbc9000       Deferred        opengl32.so
ELF     7cc51000-7cd04000       Deferred        libzstd.so.1
ELF     7cd04000-7ce00000       Deferred        libgcrypt.so.20
ELF     7ce00000-7d205000       Deferred        libcrypto.so.3
ELF     7d207000-7d20f000       Deferred        kerberos.so
ELF     7d244000-7d26b000       Deferred        libgpg-error.so.0
ELF     7d26b000-7d316000       Deferred        libpcre2-8.so.0
ELF     7d316000-7d400000       Deferred        libsystemd.so.0
ELF     7d409000-7d413000       Deferred        secur32.so
ELF     7d413000-7d41a000       Deferred        netapi32.so
ELF     7d41a000-7d420000       Deferred        dwrite.so
ELF     7d420000-7d45f000       Deferred        liblzma.so.5
ELF     7d45f000-7d484000       Deferred        liblz4.so.1
ELF     7d484000-7d491000       Deferred        libcap.so.2
ELF     7d491000-7d4c4000       Deferred        libselinux.so.1
ELF     7d4c4000-7d4d6000       Deferred        libresolv.so.2
ELF     7d4d6000-7d4dd000       Deferred        libkeyutils.so.1
ELF     7d4dd000-7d53a000       Deferred        libdbus-1.so.3
ELF     7d53a000-7d54a000       Deferred        libkrb5support.so.0
ELF     7d54a000-7d561000       Deferred        libk5crypto.so.3
ELF     7d561000-7d62e000       Deferred        libkrb5.so.3
ELF     7d62e000-7d644000       Deferred        libavahi-client.so.3
ELF     7d644000-7d654000       Deferred        libavahi-common.so.3
ELF     7d654000-7d6ad000       Deferred        libgssapi_krb5.so.2
ELF     7d6ad000-7d76f000       Deferred        libcups.so.2
ELF     7d76f000-7d774000       Deferred        dnsapi.so
ELF     7d774000-7d77a000       Deferred        crypt32.so
ELF     7d7c1000-7d7ca000       Deferred        libxfixes.so.3
ELF     7d7ca000-7d7d7000       Deferred        libxcursor.so.1
ELF     7d7d7000-7d7eb000       Deferred        libxi.so.6
ELF     7d7eb000-7d7fa000       Deferred        libxrandr.so.2
ELF     7d7fa000-7d807000       Deferred        libxrender.so.1
ELF     7d807000-7d835000       Deferred        libxcb.so.1
ELF     7d835000-7d98c000       Deferred        libx11.so.6
ELF     7d98c000-7da1a000       Deferred        winex11.so
ELF     7db1a000-7dbb0000       Deferred        libgmp.so.10
ELF     7dbb0000-7dbfb000       Deferred        libhogweed.so.6
ELF     7dbfb000-7dc4f000       Deferred        libnettle.so.8
ELF     7dc4f000-7de00000       Deferred        libunistring.so.5
ELF     7de00000-7e033000       Deferred        libgnutls.so.30
ELF     7e037000-7e03c000       Deferred        libxcomposite.so.1
ELF     7e03c000-7e052000       Deferred        libxext.so.6
ELF     7e052000-7e06a000       Deferred        libtasn1.so.6
ELF     7e06a000-7e200000       Deferred        libp11-kit.so.0
ELF     7e5a3000-7e5aa000       Deferred        libxxf86vm.so.1
ELF     7e5aa000-7e5b4000       Deferred        libffi.so.8
ELF     7e5b4000-7e5d5000       Deferred        libidn2.so.0
ELF     7e5d5000-7e5df000       Deferred        libjitterentropy.so.3
ELF     7e5e0000-7e5e5000       Deferred        libcom_err.so.2
ELF     7e5e5000-7e5eb000       Deferred        winspool.so
ELF     7e60f000-7e61c000       Deferred        bcrypt.so
ELF     7e886000-7e8b3000       Deferred        libexpat.so.1
ELF     7e8b3000-7e907000       Deferred        libfontconfig.so.1
ELF     7e907000-7e92a000       Deferred        libbrotlicommon.so.1
ELF     7e92a000-7e96e000       Deferred        libpng16.so.16
ELF     7e96e000-7ea77000       Deferred        libm.so.6
ELF     7ea77000-7ec00000       Deferred        win32u.so
ELF     7ee78000-7ee7d000       Deferred        libxinerama.so.1
ELF     7ee7d000-7ee8b000       Deferred        libbrotlidec.so.1
ELF     7ee8b000-7ee9f000       Deferred        libbz2.so.1
ELF     7ee9f000-7eeb8000       Deferred        libz.so.1
ELF     7eeb8000-7ef78000       Deferred        libfreetype.so.6
ELF     7ef7b000-7ef80000       Deferred        libxau.so.6
ELF     7ef80000-7ef86000       Deferred        ws2_32.so
PE-Wine 7fd90000-7fdc2000       Deferred        rsaenh
PE-Wine 7fdd0000-7fddb000       Deferred        msimg32
ELF     f7b48000-f7c00000       Export          ntdll.so
ELF     f7c00000-f7e35000       Deferred        libc.so.6
ELF     f7f74000-f7fa9000       Deferred        ld-linux.so.2
ELF     f7fa9000-f7fae000       Deferred        <wine-loader>
Threads:
process  tid      prio    name (all IDs are in hex)
00000020 (D) Z:\hdd\downloads\syncthingtray-1.4.11-i686-w64-mingw32.exe\syncthingtray-1.4.11-i686-w64-mingw32.exe
	00000024    0 <== 
	00000104    0     
	00000108    0     
	0000010c    0     
	00000110    0     
	00000114    0     
	00000118    0     
	0000011c    0     
	00000120    0     wine_threadpool_worker
	00000124    0     wine_threadpool_waitqueue
	00000128    0     wine_rpcrt4_server
	0000013c    0     
00000038 services.exe
	0000003c    0     
	00000040    0     wine_rpcrt4_server
	0000004c    0     wine_rpcrt4_io
	00000070    0     wine_rpcrt4_io
	000000a0    0     wine_rpcrt4_io
	000000b8    0     wine_rpcrt4_io
	000000cc    0     
	000000e4    0     wine_rpcrt4_io
00000044 winedevice.exe
	00000048    0     
	00000054    0     
	00000058    0     wine_sechost_service
	0000005c    0     
	00000060    0     
	00000064    0     
	000000c4    0     
	000000c8    0     
00000068 winedevice.exe
	0000006c    0     
	00000074    0     
	00000078    0     wine_sechost_service
	0000007c    0     
	00000080    0     
	00000084    0     
	00000098    0     
	0000009c    0     
00000088 plugplay.exe
	0000008c    0     
	000000a4    0     
	000000a8    0     wine_sechost_service
	000000ac    0     wine_rpcrt4_server
00000090 explorer.exe
	00000094    0     
	000000d0    0     
	000000d4    0     wine_rpcrt4_server
000000b0 svchost.exe
	000000b4    0     
	000000bc    0     
	000000c0    0     wine_sechost_service
000000dc rpcss.exe
	000000e0    0     
	000000ec    0     
	000000f0    0     wine_sechost_service
	000000f4    0     wine_rpcrt4_server
	000000f8    0     wine_rpcrt4_server
	000000fc    0     wine_rpcrt4_io
00000134 conhost.exe
	00000138    0     
System information:
    Wine build: wine-9.0-rc1
    Platform: x86_64 (guest: i386)
    Version: Windows 10
    Host system: Linux
    Host version: 6.6.6-1-default

This is also reproducible with v1.4.10. This is not reproducible with v1.4.9 so there's definitely a regression between v1.4.9 and v1.4.10 and probably the crash I'm able to reproduce has the same source than your crash.

[Martchus]

There were indeed no changes in Qt (as also the Qt 5 builds are affected in the very same way and I haven't updated Qt 5 at all between those builds).

There were also no relevant changes in the code that crashes.

Maybe the toolchain update from mingw-w64-gcc-12.2.0-1-x86_64 to mingw-w64-gcc-13.1.0-1-x86_64 (https://gitlab.archlinux.org/archlinux/packaging/packages/mingw-w64-gcc/-/commit/025acaaa02145a395d3a87cf5f10f9ab1d693c8b) causes this regression. This would be easy to cross-check by building v1.4.11 with the older toolchain. This would be the next step for me to investigate. Let's see when I'll come around to do that, so feel free to try it out yourself meanwhile.

[tomasz1986]

@tomasz1986 Can you use the CLI (e.g. to print the help text or to use the CLI of the built-in Syncthing)? Does it show anything in the terminal when starting from a terminal via the CLI wrapper? (If you use MSYS2 you can also just use the normal binary instead of the CLI wrapper.)

Note that the 32-bit version requires an i686 CPU (and wouldn't work if you only have e.g. i586).

Unfortunately, there is no output at all when trying to run the CLI executable (although --help does print out the standard information). The specific CPU is Intel Atom Z3740D (https://intel.com/content/www/us/en/products/sku/78416/intel-atom-processor-z3740d-2m-cache-up-to-1-83-ghz/specifications.html). It actually is a 64-bit CPU, however the device has a 32-bit UEFI, hence it can only run Windows x86.

[Martchus]

I now had the change to test natively under Windows and there it indeed crashes even sooner. That's also the case on a 64-bit Windows and a newer version of Windows so I simplified the issue title.

It does not crash instantly, though. As you've already mentioned, printing --help is possible but also some of the other CLI options (e.g. triggering the web UI of an already running instance) work just fine. I still suspect there's some miscompilation with the new GCC version.

[Martchus]

I've just been updating GCC to 13.2.1 (now using my own build instead of Arch Linux's 13.1.0 build, see Martchus/PKGBUILDs@12153c8) and I have rebuilt Syncthing Tray. In WINE it already no longer crashes. I'll update the binaries in the release section when it also looks good under Windows (for now you'll find binaries on https://martchus.no-ip.biz/repo/arch/ownstuff/os/x86_64). I haven't rebuilt Qt 6 because apparently rebuilding Syncthing Tray itself already does the trick. Maybe it is a problem with one of the libraries that come from GCC or a miscompilation affecting just a unit of Syncthing Tray itself.

[Martchus]

Looks like the binaries created using GCC 13.2.1 work fine under Windows. So I'm considering this issue resolved. (I created https://gitlab.archlinux.org/archlinux/packaging/packages/mingw-w64-gcc/-/issues/6 as a follow-up to get Arch Linux's packaging fixed but for the sake of building Syncthing Tray I can just use my own GCC build.)

[tomasz1986]

I can confirm that the fixed version works! Thank you very much.