Apple M1 Docker not working so hot

[brandonros]

$ wine notepad
0009:err:environ:run_wineboot failed to start wineboot c00000e5
000d:err:environ:run_wineboot failed to start wineboot c00000e5
000d:err:module:__wine_process_init failed to load L"C:\\windows\\system32\\notepad.exe", error c0000018

What other information can I provide? I'm running ./docker-wine --rdp from

Darwin Brandons-MacBook-Air.local 20.3.0 Darwin Kernel Version 20.3.0: Thu Jan 21 00:06:51 PST 2021; root:xnu-7195.81.3~1/RELEASE_ARM64_T8101 arm64

[brandonros]

@scottyhardy any suggestions?

[scottyhardy]

Unfortunately it doesn't look like I can build for ARM64 due to the reliance on i386 arch for wine. If you're running with Rosetta you can apparently run Intel-based docker containers with a hefty performance hit due to emulation. There may be other problems with getting X11 working on macOS with M1 chipsets but I can't test that as I only have an Intel Mac.

The build has been broken for a while which may also be related to the error you received when using --rdp. It might be worth testing again as it's working now, but performance probably won't be great.

[scottyhardy]

Doing some further reading, it is possible to install or build an ARM64 based version of wine, but you’re limited to 64bit applications that have been compiled for ARM which only exists for some newer Win10 apps. If there’s enough interest I’ll do ARM builds but otherwise I think the x86 build with emulation will be more useful for most users.

[brandonros]

how much effort is that on your end? @scottyhardy

[scottyhardy]

@brandonros fortunately not too much effort.

Turns out I needed to build from source as I couldn't find the binaries anywhere. I've just added a pull request and waiting for the build to complete. Once successful, it'll push the new image to scottyhardy/docker-wine:arm64. You will also be able to run it with just ./docker-wine --arm64 which essentially just uses the new image tag and adds --platform=linux/arm64 to the docker run command. The image also contains the RDP server and I've had both X11 and RDP modes working on my Mac when testing (mine's an Intel Macbook Pro but it seems to run ARM via emulation with decent performance).

I haven't included the gecko/mono install files or winetricks in the arm64 build as I'm pretty sure I'll need to build them from source as well but I haven't looked into it. I've at least performed the usual wine notepad test which works but I haven't tried anything else.

[brandonros]

let me know when the tag is pushed + published + ready to test please, thank you

i see you just merged

[scottyhardy]

@brandonros it’s ready to go now. Let me know if you hit any problems.

[brandonros]

Which tag example @scottyhardy ?

https://hub.docker.com/r/scottyhardy/docker-wine/tags

You might want to also get GitHub release/tags up to par/matching if it isn't too much effort for you: https://github.com/scottyhardy/docker-wine/releases

a bit confusing that they are out of sync

shall I try latest, stable-6.0.2 or devel-7.0-rc1-arm64?

[brandonros]

docker run -it   --rm   --hostname="$(hostname)"   --env="RDP_SERVER=yes"   --publish="3389:3389/tcp"   --platform=linux/arm64   scottyhardy/docker-wine:devel-7.0-rc1-arm64 /bin/bash

Works flawlessly. Thank you.

[brandonros]

wine Downloads/WinOLS_Testversion.64Bit.exe 
00fc:err:module:load_wow64_ntdll failed to load L"\\??\\C:\\windows\\syswow64\\ntdll.dll" error c0000135
00fc:err:wow:load_64bit_module failed to load dll c0000135

as expected, non-arm64 binaries don't work. i wonder what our options are in terms of translation/emulation. what's the best route?

[brandonros]

Weird.

docker run -it   --rm   --hostname="$(hostname)"   --env="RDP_SERVER=yes"   --publish="3389:3389/tcp"   --platform=linux/amd64   scottyhardy/docker-wine:stable-6.0.2 /bin/bash

I don't fully understand why that would segfault when you try to run wine notepad inside the RDP context. I thought we'd abstract away the fact that the underlying hardware is running arm64 because we're having Docker emulate (through Rosetta I think?) amd64 -> arm64?

[scottyhardy]

The tags used for arm64 builds are tagged as:

• arm64
• devel-version-arm64
• devel-version-arm64-timestamp

Release tags are just arbitrary versions for the code itself, if I continue to use them I think I’ll drop semver and use calendar versions instead. Builds are automated to run weekly and are tagged by the branch and version of wine actually installed in the image.

I saw the segfault in testing too. I assumed that was because of the ARM emulation on my Intel Mac. It’s probably something broken in the build - I’ll also try doing a traditional AMD64 build and see if it exhibits the same behaviour.

Interestingly, I noticed that even though the Ubuntu image is using ARM64 platform (you see architecture is AARCH64 with uname -a), if you use ps -ef you see almost all processes are actually run with QEMU. I’m not exactly sure why that’s the case and googling turned up nothing. Not sure if this potentially helps with using x86 compiled binaries on ARM but I thought it was a little odd.

As far as running Windows apps, I think you’re best off looking for Windows 8/10 apps compiled for ARM or compile some code yourself.

[brandonros]

I saw the segfault in testing too. I assumed that was because of the ARM emulation on my Intel Mac. It’s probably something broken in the build - I’ll also try doing a traditional AMD64 build and see if it exhibits the same behaviour.

I would say the issue is almost worth leaving open just for you and I to both investigate that. On one hand, you've done a great job getting ARM64 working. I don't know if you want to add to the README how people can control --platform=linux/amd64 vs --platform=linux/arm64 and the whole emulation thing.

Unfortunately, I can't get the manufacturer of this software to add ARM64 support. So I'm kind of stuck with... "what is my best option on ARM64 to run this AMD64 Windows EXE"

So far, I have not found the answer. Let me tell you :P

No VirtualBox, no VMWare, Parallels was a bust, UTM virtual machine is kind of a bust. I'm pretty sure I could get QEMU to work but I'm afraid the performance is just totally going to suck lol.

[scottyhardy]

The problem with running Wine compiled for ARM means that the Windows binaries you use must also be compiled for ARM. I don't think there's any way around this restriction and there's just not a lot of readily available programs that fit that category outside of the standard Microsoft Office apps.

I think if your aim is to be able to run x86/amd64 Windows programs on ARM, then using the original amd64 image(s) with emulation would be best. From my testing, the docker-ce package on Linux or Docker Desktop on macOS includes QEMU for CPU emulation out-of-the-box and just works without any changes needed. Just use docker run scottyhardy/docker-wine /bin/bash and it should give a warning that the image CPU doesn't match and then just run with emulation. You could throw in --platform=linux/amd64 but I intentionally used separate tags for the ARM images so that the amd64 image is always used by default. I still want to keep the ARM images for others to play with or for specific use cases, but I'll continue to tag them with -arm64 so users will need to intentionally choose to use ARM as it's the inferior way to use the image in my opinion.

[scottyhardy]

@brandonros Also, just a note that Hypervisors will only let you install guest VMs with the same CPU architecture as the host (well, technically you can run an "enhanced compatibility mode" with some hypervisors to essentially downgrade CPU features for one or more hosts to match other host CPUs within a cluster to allow live migrations of VMs between hosts, but they still need to be within the same CPU family and can't be too many generations apart).

If you want to run x86 on ARM, emulation either on the host OS or within the VM would still be required. Since you're using a docker image and docker itself ships with QEMU for emulating other CPU architectures then it makes sense to use that rather than include emulation within a container image.

[brandonros]

I think if your aim is to be able to run x86/amd64 Windows programs on ARM, then using the original amd64 image(s) with emulation would be best.

I think I had 2 problems:

1. There was no arm64 working tag for this image. That's resolved.
2. I want to run amd64 but I can't, it segfaults.

Do you have any guesses as to why #2 happens/what I need to provide to you log wise so we can understand?

[brandonros]

#109 (comment)

This is from a purely amd64 context, just running on underlying Apple M1 hardware. So yes, "emulation" somewhere in the stack.

[scottyhardy]

Ok, I missed that part earlier where you got the segfault running the Intel image on your M1 Mac. That is a problem so I'm going to re-open this issue.

After some further reading I've confirmed that Rosetta is only used for some of the additional docker binaries (e.g. docker scan) but qemu is used for emulating other CPU architectures for containers (see https://docs.docker.com/desktop/mac/apple-silicon/). This is because docker containers run in a virtual machine hosted by Hyperkit on macOS, which I assume is a Linux VM.

In addition, the article linked above also says:

However, attempts to run Intel-based containers on Apple silicon machines under emulation can crash as qemu sometimes fails to run the container.

I'm pretty confident the cause of the segfaults relates to using qemu so the optimal solution would be to use an ARM image for M1 Macs, but then we have the issue with incompatibility with x86 Windows binaries. I've just stumbled over another project named Hangover which looks to bridge that gap with a custom qemu build for running x86 binaries with Wine built on multiple arches including ARM: https://github.com/AndreRH/hangover

I'll take a look at building it and will potentially use it to replace the current arm64 image. It may take some time though - each build takes 8+ hours normally just for the 64-bit build of Wine. I expect the build times will be pushed out quite a bit more with the additional packages being built.

[brandonros]

@scottyhardy

It's weird because when I am running the Intel image on my ARM processor, the RDP + Ubuntu part of it works fine. It's the Wine part specifically that fails. I really think if you and I dove deeper into it, we'd have some success.

A real alternative I was looking at (especially with recent progress being made on qemu-system-aarch64 being able to support https://wiki.qemu.org/Features/HVF) was running the Windows 11 AARCH64 insider preview because of this: https://www.pcmag.com/news/microsofts-x86-64-bit-emulator-for-arm-devices-is-exclusive-to-windows

But I couldn't get that working either, lol. I tried UTM and qemu-system-aarch64, all kinds of different kinds of errors. It for sure feels like we're early.

Do you have access to an arm64/aarch64/Apple M1 computer to test? I'd love to give you the really easy steps to reproduce. In my mind, there's no reason this Wine executable should fail when the Intel emulation overall is working enough for RDP to work.

[scottyhardy]

@brandonros No, I don't have any ARM devices available which makes troubleshooting a bit of a pain.

Segfaults relate to memory and I just remembered an issue I hit earlier on with RDP and needed to add --shm-size="1g" due to insufficient memory: scottyhardy/docker-remote-desktop#3 (comment) 66970a7

If that doesn't fix your issue, then I suppose the first step is to see whether or not I also get a segfault when I replicate whatever you're doing on my Intel Mac with the same x86 image. If I do then there may be some chance for me to troubleshoot a little easier, but if it doesn't error for me then the main difference is the CPU and there may not be a lot we can do.

[brandonros]

Steps to reproduce:

$ uname -a
Darwin Brandons-MacBook-Air.local 21.2.0 Darwin Kernel Version 21.2.0: Sun Nov 28 20:29:10 PST 2021; root:xnu-8019.61.5~1/RELEASE_ARM64_T8101 arm64

$ docker run -it   --rm   --hostname="$(hostname)"   --env="RDP_SERVER=yes"   --publish="3389:3389/tcp"   --platform=linux/amd64   scottyhardy/docker-wine:stable-6.0.2 /bin/bash
xrdp-sesman[77]: (77)(274909663552)[DEBUG] libscp initialized

xrdp-sesman[77]: (77)(274909663552)[DEBUG] Testing if xrdp-sesman can listen on 127.0.0.1 port 3350.

xrdp-sesman[77]: (77)(274909663552)[DEBUG] Closed socket 4 (AF_INET6 ::ffff:127.0.0.1 port 3350)

xrdp-sesman[80]: (80)(274909663552)[INFO ] starting xrdp-sesman with pid 80

xrdp-sesman[80]: (80)(274909663552)[INFO ] listening to port 3350 on 127.0.0.1

xrdp[82]: (82)(274910156608)[INFO ] address [0.0.0.0] port [3389] mode 1

[20211221-22:34:38] [INFO ] address [0.0.0.0] port [3389] mode 1
xrdp[82]: (82)(274910156608)[INFO ] listening to port 3389 on 0.0.0.0

[20211221-22:34:38] [INFO ] listening to port 3389 on 0.0.0.0
xrdp[82]: (82)(274910156608)[INFO ] xrdp_listen_pp done

[20211221-22:34:38] [INFO ] xrdp_listen_pp done
xrdp[82]: (82)(274910156608)[DEBUG] Closed socket 6 (AF_INET6 :: port 3389)

[20211221-22:34:38] [DEBUG] Closed socket 6 (AF_INET6 :: port 3389)
wineuser@Brandons-MacBook-Air:/$ xrdp[85]: (85)(274910156608)[INFO ] starting xrdp with pid 85

xrdp[85]: (85)(274910156608)[INFO ] address [0.0.0.0] port [3389] mode 1

xrdp[85]: (85)(274910156608)[INFO ] listening to port 3389 on 0.0.0.0

xrdp[85]: (85)(274910156608)[INFO ] xrdp_listen_pp done

wineuser@Brandons-MacBook-Air:/$ 

Then, use Microsoft Remote Desktop to RDP to 127.0.0.1:3389 as wineuser. Ignore the invalid certificate error.

Open Terminal in the Xubuntu GUI.

From the Terminal inside the running Docker container:

Validate we're running x86_64

wineuser@Brandons-MacBook-Air:~$ uname -a
Linux Brandons-MacBook-Air.local 5.10.76-linuxkit #1 SMP PREEMPT Mon Nov 8 11:22:26 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Try to run notepad:

$ wine notepad
wine: created the configuration directory '/home/wineuser/.wine'
qemu: uncaught target signal 11 (Segmentation fault) - core dumped
Segmentation fault

Try with a clean .wine directory:

wineuser@Brandons-MacBook-Air:~$ rm -rf ~/.wine
wineuser@Brandons-MacBook-Air:~$ wine notepad
wine: created the configuration directory '/home/wineuser/.wine'
qemu: uncaught target signal 11 (Segmentation fault) - core dumped
Segmentation fault
wineuser@Brandons-MacBook-Air:~$ winecfg
'qemu: uncaught target signal 11 (Segmentation fault) - core dumped
Segmentation fault

Adding --shm-size="1g" did not fix the issue.

[brandonros]

6.0.2 stable tag at least got RDP working, now with latest tag, RDP just gives you a full res black screen. 6.0.2 used to do that for a second, but then it would draw the screen.

root@Brandons-MacBook-Air:/# strace winecfg
/usr/bin/strace: test_ptrace_get_syscall_info: PTRACE_TRACEME: Function not implemented
/usr/bin/strace: ptrace(PTRACE_TRACEME, ...): Function not implemented
/usr/bin/strace: PTRACE_SETOPTIONS: Function not implemented
/usr/bin/strace: detach: waitpid(1291): No child processes
/usr/bin/strace: Process 1291 detached

not really sure what to do at this point. i added --cap-add=SYS_PTRACE

[brandonros]

Looks like docker/for-mac#5123 is related but I don't see a resolution

I'm running the latest Docker Desktop 4.3.1 (72247)

not sure waht version of QEMU that bundles... might just need to wait for it to fix itself?

[scottyhardy]

Ok, wow - that just dies straight away. At least when I'm using the arm64 image on my Intel Mac it successfully runs notepad and doesn't segfault until after exiting the program. I don't think there's a lot more we can do about getting the Intel image working on M1 Macs as I'd prefer to avoid messing with qemu in the docker desktop VM. There's no guarantee the issue with qemu will ever be resolved though, so I'm going to go ahead with testing the hangover solution I proposed earlier.

As for the black screen on RDP - that looks to be related to the sudo changes I just added recently. I'm seeing the same on the arm64 image and I originally had the same problem on the scottyhardy/docker-remote-desktop base image when I first added sudo there. I don't understand what the relationship between the two is but it seems to be an issue with the entrypoint script that I need to troubleshoot.

[brandonros]

I agree with you that QEMU + Docker Desktop/docker-engine really aren't your problem. It sucks because... I don't really know what to report on docker/for-mac as an issue. You agree that strace should work in theory, right?

We can close this as not your issue. Just stinks that it happens specifically when trying to run Intel wine/winecfg while on an ARM Mac.

[scottyhardy]

I'd start by reporting this behaviour with strace. It seems to be related to qemu, as I have no issues on an Intel image, but with ARM I get pretty much the same as you:

wineuser@Scotts-MacBook-Pro:~$ strace -f -s 10000 -o ./strace.log wine notepad
/usr/bin/strace: test_ptrace_get_syscall_info: PTRACE_TRACEME: Function not implemented
/usr/bin/strace: ptrace(PTRACE_TRACEME, ...): Function not implemented
/usr/bin/strace: PTRACE_SETOPTIONS: Function not implemented
/usr/bin/strace: detach: waitpid(867): No child processes

In this case, wine notepad works fine without using strace, so it's just strace that's the issue. Oh and I tried with --privileged and --cap-add=SYS_PTRACE but none of that worked. The fact that we both only have issues when emulation is occurring leads to QEMU as the likely culprit.

[brandonros]

I think I came up with something crazy.

iximiuz/docker-to-linux#23

Going to see if I can combine it roughly with https://gist.github.com/brandonros/0d6a275c280ba9bcca9269aa3221db6d

[scottyhardy]

@brandonros would you mind testing out my hangover docker-wine image? Tag is hangover-arm64. I'm pretty confident you should at least be able to run wine notepad since it's an ARM image, but the built-in custom qemu component should also allow you to run regular x86 Windows binaries. Please let me know how well it works for you, as I'm thinking of promoting this image to share the latest tag with the standard Intel image so it will be downloaded by default on ARM-based machines.

[brandonros]

@scottyhardy Is this supposed to be an x86_64 image or arm64?

[brandonros]

wineuser@Brandons-MacBook-Air:~$ HOQEMU="/usr/local/lib/hangover/build/qemu/x86_64-windows-user/qemu-x86_64.exe.so"
wineuser@Brandons-MacBook-Air:~$ PATH="${PATH}:/usr/local/lib/hangover/build/wine-host/loader"
wineuser@Brandons-MacBook-Air:~$ which wine
/usr/local/lib/hangover/build/wine-host/loader/wine
wineuser@Brandons-MacBook-Air:~$ wine notepad
bash: /usr/local/lib/hangover/build/wine-host/loader/wine: No such file or directory
wineuser@Brandons-MacBook-Air:/usr/local/lib/hangover/build/wine-host$ ./wine
./wine: 107: exec: /usr/local/lib/hangover/build/wine-host/loader/wine: not found
wineuser@Brandons-MacBook-Air:/usr/local/lib/hangover/build/wine-host$ ls -l
total 19800
-rw-r--r--   1 root root 18920470 Dec 23 21:28 Makefile
-rw-r--r--   1 root root  1179095 Dec 23 21:28 config.log
-rwxr-xr-x   1 root root    91302 Dec 23 21:28 config.status
drwxr-xr-x 908 root root    36864 Dec 23 22:04 dlls
drwxr-xr-x   2 root root     4096 Dec 23 21:54 fonts
drwxr-xr-x   3 root root    12288 Dec 23 21:54 include
drwxr-xr-x   5 root root     4096 Dec 23 21:28 libs
drwxr-xr-x   2 root root     4096 Dec 23 22:04 loader
drwxr-xr-x   2 root root     4096 Dec 23 21:54 nls
drwxr-xr-x   2 root root     4096 Dec 23 21:29 po
drwxr-xr-x 107 root root     4096 Dec 23 21:28 programs
drwxr-xr-x   2 root root     4096 Dec 23 22:04 server
lrwxrwxrwx   1 root root       28 Dec 23 21:28 wine -> ../../wine/tools/winewrapper
wineuser@Brandons-MacBook-Air:/usr/local/lib/hangover/wine/tools$ ./winewrapper
winewrapper: could not locate the Wine build tree

[scottyhardy]

Ok, d'oh! I kinda just assumed that build would work the same as the one on my machine. Looks like I missed adding the platform to the build and may have other issues. Sorry, next time I'll actually test it myself first before I ask you 🤦

[brandonros]

Don't say that at all. I am more than happy to test. I appreciate your help here 1000x.

For what it's worth, it's almost unsuably slow. But let's see if we can get this working.

Things I've tried:

• qemu-system-aarch64 windows 11 ISO
• qemu-system-aarch64 windows 11 VHDX
• docker-wine
• UTM
• Parallels
• docker-to-linux (didn't work lol)

[scottyhardy]

Have you tried Win 11 in Parallels? I thought ARM win 11 was supposed to have better x86 emulation…?

[Morakhiyasaiyam]

What about adding box64 and box86 to arm image it can run x86 and amd64 apps for linux and than run wine for amd64 with it ?

[brandonros]

@scottyhardy thoguhts ^?

[scottyhardy]

box64 and box86 are both x86/amd64 apps, so won't run in an arm64 native container

[Morakhiyasaiyam]

box64 and box86 are both x86/amd64 apps, so won't run in an arm64 native container

They are arm64 binary and meant to be run on arm64

[scottyhardy]

Oh ok, I wasn't aware they were compiled for arm64. I'll have to do a little more investigation. There may already be docker images for these if that's the case.

[Morakhiyasaiyam]

Oh ok, I wasn't aware they were compiled for arm64. I'll have to do a little more investigation. There may already be docker images for these if that's the case.

Yes there are
Look at here https://github.com/Weilbyte/box
But without gui
I requested for that but didn't implemented yet