Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Startup error after modifying configuration. #37

Closed
nongze opened this issue Mar 2, 2024 · 33 comments
Closed

Startup error after modifying configuration. #37

nongze opened this issue Mar 2, 2024 · 33 comments

Comments

@nongze
Copy link

nongze commented Mar 2, 2024

When I added configuration to Docker Compose.yml to modify RAM and CPU, there was a startup error, whether it was the first creation or modification after creation.

    environment:
      VERSION: "win11"
      RAM_SIZE: "4G"
      CPU_CORES: "4"
root@orangepi5:/root/win# docker-compose up
[+] Building 0.0s (0/0)                                                                                                      docker:default
[+] Running 1/1
 ✔ Container windows  Recreated                                                                                                        0.2s 
Attaching to windows
windows  | ❯ Starting Windows for Docker v1.08...
windows  | ❯ For support visit https://github.com/dockur/windows-arm
windows  | 
windows  | ❯ Booting Windows using QEMU emulator version 8.2.1 ...
windows  | ❯ ERROR: qemu-system-aarch64: Failed to put registers after init: Invalid argument
windows exited with code 0

If the default configuration is used, there will be no errors.

@kroese
Copy link
Contributor

kroese commented Mar 4, 2024

I cannot think of a reason why this happens. Because even without setting these values in your Dockerfile, they are still set to the defaults (4G and 2 cores) so if anything was wrong with the order I set them for example, it would also happen with the defaults.

I will see if I can reproduce this and fix it, but currently I have no ARM hardware available anymore (used my Pi5 for something else now), so it might take some time before I can fix this.

@yuunnn-w
Copy link

yuunnn-w commented Mar 9, 2024

I cannot think of a reason why this happens. Because even without setting these values in your Dockerfile, they are still set to the defaults (4G and 2 cores) so if anything was wrong with the order I set them for example, it would also happen with the defaults.

I will see if I can reproduce this and fix it, but currently I have no ARM hardware available anymore (used my Pi5 for something else now), so it might take some time before I can fix this.

I suspect the issue might be due to the qemu version you are using being too old, leading to abnormal parameter passing. The error message is: ERROR: qemu-system-aarch64: Failed to put registers after init: Invalid argument . Moreover, the error occurs regardless of whether the parameters in the configuration file are removed or not. The qemu version we are using is 8.2.1, and the hardware we use is OrangePi 5 Plus.

@samirdjelal
Copy link

I've got the same issue on OrangePi 5 Plus, I've used the default docker-compose without any changes or custom env vars

windows  | ❯ Extracting Windows 11 for ARM bootdisk...
windows  | ❯ Extracting Windows 11 for ARM environment...
windows  | ❯ Extracting Windows 11 for ARM setup...
windows  | ❯ Extracting Windows 11 for ARM image...
windows  | ❯ Adding XML file for automatic installation...
windows  | ❯ Building Windows 11 for ARM image...
windows  | ❯ Creating a 64G growable disk image in raw format...
windows  | ❯ Booting Windows using QEMU emulator version 8.2.1 ...
windows  | ❯ ERROR: qemu-system-aarch64: Failed to put registers after init: Invalid argument
windows  | ❯ Starting Windows for Docker v1.08...
windows  | ❯ For support visit https://github.com/dockur/windows-arm

@kroese
Copy link
Contributor

kroese commented Mar 13, 2024

For anyone experiencing this issue, can you add:

environment:
  DEBUG: "Y"

to your compose file. And then copy and paste the QEMU commandline arguments here. And also a copy of the arguments when it does not give that error. So we can compare both and see what the difference is between them?

@Explosion-Scratch
Copy link

@kroese Happening to me as well:

windows  | ❯ Starting Windows for Docker v1.09...
windows  | ❯ For support visit https://github.com/dockur/windows-arm
windows  |
windows  | ❯ Host: c84093f0d383  IP: 172.24.0.2  Gateway: 172.24.0.1  Interface: eth0  MAC: 02:7E:CC:0F:F0:80
windows  | nameserver 127.0.0.11
windows  |
windows  | ++ /usr/sbin/dnsmasq --dhcp-range=20.20.20.21,20.20.20.21 --dhcp-host=02:7E:CC:0F:F0:80,,20.20.20.21,QEMU,infinite --dhcp-option=option:netmask,255.255.255.0 --dhcp-option=option:dns-server,20.20.20.1 --dhcp-option=option:router,20.20.20.1 --address=/host.lan/20.20.20.1
windows  |
windows  | ❯ Booting Windows using QEMU emulator version 8.2.1 ...
windows  | Arguments: -nodefaults -cpu host -smp 4,sockets=1,dies=1,cores=4,threads=1 -m 4G -machine type=virt,secure=off,dump-guest-core=off,accel=kvm -enable-kvm -display vnc=:0,websocket=5700 -device ramfb -monitor telnet:localhost:7100,server,nowait,nodelay -daemonize -D /run/shm/qemu.log -pidfile /run/shm/qemu.pid -name windows,process=windows,debug-threads=on -serial pty -device qemu-xhci,id=xhci -device usb-kbd -device usb-tablet -netdev tap,ifname=qemu,script=no,downscript=no,id=hostnet0 -device virtio-net-pci,romfile=,netdev=hostnet0,mac=02:7E:CC:0F:F0:80,id=net0 -object iothread,id=io2 -drive id=cdrom0,media=cdrom,if=none,format=raw,readonly=on,file=/storage/win11arm64.iso -device virtio-scsi-pci,id=scsi0,iothread=io2,addr=0x5 -device scsi-cd,bus=scsi0.0,drive=cdrom0,bootindex=10 -drive id=cdrom1,media=cdrom,if=none,format=raw,readonly=on,file=/run/drivers.iso -device usb-storage,drive=cdrom1 -drive file=/storage/data.img,if=none,id=drive-userdata,format=raw,cache=none,aio=native,discard=on,detect-zeroes=on -device virtio-scsi-pci,id=hw-userdata,iothread=io2,bus=pcie.0,addr=0xa -device scsi-hd,bus=hw-userdata.0,channel=0,scsi-id=0,lun=0,drive=drive-userdata,id=userdata,rotation_rate=1,bootindex=3 -drive file=/storage/windows.rom,if=pflash,unit=0,format=raw,readonly=on -drive file=/storage/windows.vars,if=pflash,unit=1,format=raw -device virtio-balloon-pci,id=balloon0,bus=pcie.0,addr=0x4 -object rng-random,id=objrng0,filename=/dev/urandom -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pcie.0,addr=0x1c
windows  |
windows  | ❯ ERROR: qemu-system-aarch64: Failed to put registers after init: Invalid argument
windows exited with code 0
`docker-compose.yml`
version: "3"
services:
  windows:
    container_name: windows
    image: dockurr/windows-arm
    environment:
      VERSION: "win11"
      RAM_SIZE: "4G"
      CPU_CORES: "4"
      DEBUG: "Y"
    volumes:
      - /home/tjs/Media/Windows:/storage
    devices:
      - /dev/kvm
    cap_add:
      - NET_ADMIN
    ports:
      - 8006:8006
      - 3389:3389/tcp
      - 3389:3389/udp
    stop_grace_period: 2m
    restart: on-failure

@Explosion-Scratch
Copy link

I also removed all environment variables except for VERSION and DEBUG and it's still happening.

@Explosion-Scratch
Copy link

I'm also running the Orange Pi 5B using Armbian

Screenfetch
❯ screenfetch
                          ./+o+-       tjs@tjsorange
                  yyyyy- -yyyyyy+      OS: Ubuntu 22.04 jammy
               ://+//////-yyyyyyo      Kernel: aarch64 Linux 5.10.160-legacy-rk35xx
           .++ .:/++++++/-.+sss/`      Uptime: 3d 1h 28m
         .:++o:  /++++++++/:--:/-      Packages: 1802
        o:+o+:++.`..```.-/oo+++++/     Shell: zsh 5.8.1
       .:+o:+o/.          `+sssoo+/    Disk: 1.5T / 6.0T (26%)
  .++/+:+oo+o:`             /sssooo.   CPU: ARM Cortex-A55 Cortex-A76 @ 8x 1.8GHz
 /+++//+:`oo+o               /::--:.   RAM: 4996MiB / 15718MiB
 \+/+o+++`o++o               ++////.
  .++.o+++oo+:`             /dddhhh.
       .+.o+oo:.          `oddhhhh+
        \+.++o+o``-````.:ohdhhhhh+
         `:o+++ `ohhhhhhhhyo++os:
           .o:`.syhhhhhhh/.oo++o`
               /osyyyyyyo++ooo+++/
                   ````` +oo+++o\:
                          `oo++.

@kroese
Copy link
Contributor

kroese commented Mar 25, 2024

I can find very little information about this error. I suspect this is some incompatibility between the OrangePI kernel and KVM, because all of you are running into this problem on a OrangePI device.

From this post: https://forum.armbian.com/topic/29785-orange-pi-5-plus-fails-to-start-ovmf-efi-firmware-under-kvm/ it seems like it can be solved by upgrading the OrangePI to the newer Debian Trixie distribution (which has a newer kernel I guess).

@stlorenz
Copy link

I have the same on a Rockchip Device Turing RK1

This has the same chip as the OrangePi RK3588, will test the upgrade.

@stlorenz
Copy link

The update runs into errors:
Processing triggers for man-db (2.10.2-1) ... perl: /lib/aarch64-linux-gnu/libc.so.6: version GLIBC_2.36' not found (required by /lib/aarch64-linux-gnu/libcrypt.so.1)`
Same for the bash package

@morsDeiGH
Copy link

morsDeiGH commented May 8, 2024

Rockchip RK3588 have 4 core Cortex-A76 and 4 core Cortex-A55.
If disable 4 core Cortex-A76 or 4 core Cortex-A55 and after this container running normally, but with both core, we get an error "qemu-system-aarch64: Failed to put registers after init: Invalid argument".

  1. sudo chcpu --disable 0,1,2,3
  2. docker compose up
  3. sudo chcpu --enable 0,1,2,3

The problem seems to be due to different kernels

This was referenced May 9, 2024
@rogerdk
Copy link

rogerdk commented May 27, 2024

I have come across the same problem. I'm also using an OrangePi5
Could I pass custom parameters for testing?
in this thread (Joshua-Riek/ubuntu-rockchip#731 (comment)) they propose to use something like: --vcpus=4,cpuset=4-7.

However, I have tried to disable the cpus before booting and I cannot avoid this error:

'ERROR: Timeout while waiting for QEMU to boot the machine!'

even though the cpu seems high. it's as if despite not being able to connect the windows runs.

@kroese
Copy link
Contributor

kroese commented May 27, 2024

@rogerdk That thread is about libvirt which is a different software, the only similarity is that it also uses QEMU under the hood. But you cannot take parameters for that software, and use them with QEMU or this container, because we dont use libvirt at all.

The problem is that the OrangePi has a mix of different cores (big and little) and that KVM fails to handle it.

Maybe if you set it like this:

    environment:
      CPU_CORES: "1"

it will work, because its limited to a single core so there will be no mixing of types, but I am not sure.

The real solution will probably be to do some CPU pinning via QEMU numa parameters like this:

    environment:
      ARGUMENTS: "-numa node,cpus=0-2"

But I am not sure exactly which command will be needed, its explained on https://www.qemu.org/docs/master/system/qemu-manpage.html but its is very complicated.

So the solution by @morsDeiGH looks much easier, to disable the little cores outside of QEMU/the container, so that you dont need to do any CPU pinning anymore.

@sund3RRR
Copy link

sund3RRR commented Jun 2, 2024

I think the easiest solution is to override /run/entry.sh and change this line:

- { qemu-system-aarch64 ${ARGS:+ $ARGS} >"$QEMU_OUT" 2>"$QEMU_LOG"; rc=$?; } || :
+ { taskset -c $CPU_TASK_SET qemu-system-aarch64 ${ARGS:+ $ARGS} >"$QEMU_OUT" 2>"$QEMU_LOG"; rc=$?; } || :

Then add this env to the environment section of compose file:

environment:
  - CPU_TASK_SET: "4,5,6,7"

@kroese
Copy link
Contributor

kroese commented Jun 2, 2024

@sund3RRR I will create a new release, with this feature added, thanks!

@sund3RRR
Copy link

sund3RRR commented Jun 2, 2024

Off-topic. Great work, man, thank you for windows-arm!

I don't quite understand qemu-kvm related stuff, but is it real to passthrough mali-g610 gpu? Or at least accelerate 3d with virgl adapter? I want to speed up rdp session because cpu performance is not enough :(

@kroese
Copy link
Contributor

kroese commented Jun 2, 2024

@sund3RRR I want to detect the Orange Pi, so the flag will be automaticly applied. Do you know an easy way? For example, I can check if the kernel name ends with -rk35xx but I dont know if all Orange Pi's always use a kernel with that name?

@sund3RRR
Copy link

sund3RRR commented Jun 2, 2024

@sund3RRR I want to detect the Orange Pi, so the flag will be automaticly applied. Do you know an easy way? For example, I can check if the kernel name ends with -rk35xx but I dont know if all Orange Pi's always use a kernel with that name?

Hmmm. I don't know. I think this check is not enough.
At least my orangepi's kernel name is 6.1.0-1014-rockchip

And technically orangepi can run mainline linux since 6.8 kernel version with edk2-rk3588 uefi

@kroese
Copy link
Contributor

kroese commented Jun 2, 2024

Can you show me how it detects the CPU in the Docker log (the first line where it says CPU/RAM/etc)

@sund3RRR
Copy link

sund3RRR commented Jun 2, 2024

Can you show me how it detects the CPU in the Docker log (the first line where it says CPU/RAM/etc)

Sure

❯ Starting Windows for Docker v2.09...
❯ For support visit https://github.com/dockur/windows-arm
❯ CPU: Cortex A55 | RAM: 6/8 GB | DISK: 362 GB (ext4) | HOST: 6.1.0-1014-rockchip...

@kroese
Copy link
Contributor

kroese commented Jun 2, 2024

Thanks! I created a new release now (v2.10), which should have the issue fixed automaticly when it detects the RK3588 chipset.

Can you please run it and see if it works now (without adding that environment variable).

@sund3RRR
Copy link

sund3RRR commented Jun 2, 2024

It works! Well done :)

@kroese kroese closed this as completed Jun 2, 2024
@sund3RRR
Copy link

sund3RRR commented Jun 2, 2024

P.S. It doesn't work with base qemux/qemu-arm image due to

❯ Booting image using QEMU v8.2.4...
taskset: failed to execute exec: No such file or directory

Need to patch

- taskset -c "$CPU_PIN" exec qemu-system-aarch64 ${ARGS:+ $ARGS}
+ taskset -c "$CPU_PIN" qemu-system-aarch64 ${ARGS:+ $ARGS}

@nongze
Copy link
Author

nongze commented Jun 7, 2024

Can we support 8 cores in this way?

environment:
  - CPU_TASK_SET: "0,1,2,34,5,6,7"

@kroese
Copy link
Contributor

kroese commented Jun 7, 2024

@sund3RRR Thanks! Should be fixed now in qemu-arm v2.05. Let me know if it works okay now.

@nongze No, the whole problem is that it only works with 0 till 4 OR 4 till 7 (because they have different speeds).

@nongze
Copy link
Author

nongze commented Jun 8, 2024

After my attempt, it is feasible.

root@orangepi5:~# cat win/docker-compose.yml
version: "3"
services:
  windows:
    environment:
      CPU_TASK_SET: "1,2,3,4,5,6,7"
      VERSION: "win11"
      RAM_SIZE: "4G"
      CPU_CORES: "8"
    device_cgroup_rules:
      - 'c *:* rwm'
    container_name: windows
    image: dockurr/windows-arm
    devices:
      - /dev/kvm
    cap_add:
      - NET_ADMIN
    ports:
      - 8006:8006
      - 3389:3389/tcp
      - 3389:3389/udp
    stop_grace_period: 2m
    restart: on-failure
    networks:
      macnet0:
        ipv4_address: 192.168.1.100

networks:
  macnet0:
    external: true

image

@sund3RRR
Copy link

sund3RRR commented Jun 8, 2024

After my attempt, it is feasible.

root@orangepi5:~# cat win/docker-compose.yml
version: "3"
services:
  windows:
    environment:
      CPU_TASK_SET: "1,2,3,4,5,6,7"
      VERSION: "win11"
      RAM_SIZE: "4G"
      CPU_CORES: "8"
    device_cgroup_rules:
      - 'c *:* rwm'
    container_name: windows
    image: dockurr/windows-arm
    devices:
      - /dev/kvm
    cap_add:
      - NET_ADMIN
    ports:
      - 8006:8006
      - 3389:3389/tcp
      - 3389:3389/udp
    stop_grace_period: 2m
    restart: on-failure
    networks:
      macnet0:
        ipv4_address: 192.168.1.100

networks:
  macnet0:
    external: true

image

What image and kernel are you using on host?

@kroese
Copy link
Contributor

kroese commented Jun 8, 2024

@nongze That doesnt work. You set CPU_CORES to 8, that means it will create 8 "virtual" cores from the 4 real cores. You can even create a 100 core machine that way, but you should never set CPU_CORES higher than the amount of real cores, because it will make the system slower instead of faster.

Also you set CPU_TASK_SET but I dont use that name nowhere in my code, so it will be totally ignored that value. I use the name CPU_PIN.

So if you want to do a real test, rename CPU_TASK_SET to CPU_PIN and see if it still works.

@nongze
Copy link
Author

nongze commented Jun 8, 2024

I have given up, four cores are enough.
Off topic: Can I replace the image with another Linux system or have other projects?

@sund3RRR
Copy link

sund3RRR commented Jun 8, 2024

I have given up, four cores are enough. Off topic: Can I replace the image with another Linux system or have other projects?

You can use qemux/qemu-arm

@kroese
Copy link
Contributor

kroese commented Jun 8, 2024

@nongze This Windows container is a spin-off of https://github.com/qemus/qemu-arm , so to run Linux-like VM's you can use that one.

@nongze
Copy link
Author

nongze commented Jun 15, 2024

I have another question, how do you optimize Windows 11? I run it on a regular QEMU-KVM and it always restarts. Is there a download link for the system image?

@nongze
Copy link
Author

nongze commented Jun 15, 2024

I want to use it in a PVE virtual environment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

9 participants