Windows Dev Kit 2023 Boot Work

[nickdepinet]

Discussion started in Issue #7, splitting to it's own issue for tracking:

I've recently gotten ahold of a windows dev kit 2023, which appears to be a surface pro 9 (5g) / sq3 in a box, and I'm attempting to boot linux on it with these patches.

Progress so far -

Booting base Ubuntu 22.04.1 LTS arm image - Hangs at:
Booting a command list

Synchronous Exception at 0x0000000C1B17927C
Building a "default" profile image with the image tool - Hangs at:
Loading initramfs...
Booting into Surface kernel...
EFI stub: Booting Linux Kernel...
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services...
Barring better suggestions, I'm going to try to extract the firmware with the script from aarch64-firmware and stuff it into the arch image above and see where that gets booting

---

1. could possibly be fixed by linux-surface/grub@55b95a3. So you could try to use e.g. the pre-built grub from https://github.com/linux-surface/grub-image-aarch64/releases/tag/fedora-37-2 with your Ubuntu kernel. Alternatively, just use the image from 2) which contains the patched grub, but remove the devicetree option from the boot entry. That should get you booting in ACPI mode.

2. is way before any firmware gets loaded, so I doubt that that will make any difference. Can you make sure that efi=novamap is set (press e in the grub menu to edit)? Also, this is booting with the SPX DT, and I'm not sure if that's a good idea. So as I've said above, you can remove the devicetree option in the grub config and it should hopefully get you a bit further.

You can also experiment with earlycon=efifb.

In the long run, you'll probably want to have a look at the DT of the sc8280xp (that is 8cx Gen 3; it's also used in the Thinkpad X13s) and use that as a base. Also, you might want to join the oftc/#aarch64-laptops IRC. There are a lot more experienced people who also might have some ideas.

Originally posted by @qzed in #7 (comment)

[nickdepinet]

Booting the image from number 2 above with the devicetree line removed gets into a boot loop. Adding earlycon=efifb console=efifb boot_delay=200 to the boot parameters gets us to see that we're reaching the following lines before instantly rebooting:

there are a bunch of BAR and firmware errors further up the log as well that may be worth investigating

[nickdepinet]

swapping in sc8280xp-lenovo-thinkpad-x13s.dtb as the devicetree on boot gets to same line & rebootloop behavior as above

[qzed]

So at this point you'll probably really want to head over to oftc/#aarch64-laptops :) They'll know much more about what might be going wrong here.

Also if you're starting off with the thinkpad DTS you might need to remove/disable a couple of parts. AFAIK sudden restarts are generally happening when you access something you shouldn't (yet) access, like powered-off MMIO, invalid memory, ...

Particular stuff that you might want to remove:

• HID-over-I2C input nodes (https://github.com/torvalds/linux/blob/f141df371335645ce29a87d9683a3f79fba7fd67/arch/arm64/boot/dts/qcom/sc8280xp-lenovo-thinkpad-x13s.dts#L206-L257)
• Backlight (https://github.com/torvalds/linux/blob/f141df371335645ce29a87d9683a3f79fba7fd67/arch/arm64/boot/dts/qcom/sc8280xp-lenovo-thinkpad-x13s.dts#L19-L43)

maybe more...

[nickdepinet]

Stripped a bunch of stuff out of the dts into a new copy and attempted booting with it - same result. oftc/#aarch64-laptops says that the big thing is that I need to add an override for the smmu. Some googling around found this patch that seems relevant to what needs to be done: https://patchwork.kernel.org/project/linux-arm-kernel/patch/20210509022607.17534-1-shawn.guo@linaro.org/#24198673

[nickdepinet]

This does indeed appear to be the right path - whitequark logs direct us to similar issue for samsung galaxy book go here: aarch64-laptops/debian-cdimage#21. Created following PR to add wdk2023 acpi to smmu platlist here: linux-surface/kernel#130. Working on getting my local build environment setup for cross-compiling linux kernel for test

[qzed]

Ahh, I completely forgot about this being a thing... That explains at least why it boot-looped with ACPI. However, I think the DTS should have the SMMU described properly, so not sure why that fails similarly.

[nickdepinet]

No longer boot looping in ACPI mode with a kernel built with the above patch. Now we're getting rcu_preempt stalls:
.

Booting with a dtb empty except for including the sc8280xp.dtsi still causes bootloop behavior.

I've also attached the acpidump.txt here: linux-surface/acpidumps#24

[qzed]

I'd guess some MMIO region or something is being accessed that hasn't been powered up yet... but hard to tell what exactly...

[hexdump0815]

@nickdepinet - just out of curiosity: did you make any further progress in getting this device to boot mainline linux?

[xlazom00]

Any progress?
Btw @qzed are you mainlining your fixes ?

[qzed]

@xlazom00 Hopefully at some point. But that depends on someone upstreaming a device tree for the sc8180x platform. People have been working on that but I don't know the current status.

[xlazom00]

@qzed btw motherboard in dev kit is same as surface pro 9
so do you know if there is some dts for surface pro 9 somewhere ?

[nickdepinet]

@nickdepinet - just out of curiosity: did you make any further progress in getting this device to boot mainline linux?

I haven't had time to visit this in a while, so no further progress on my end. There's been some interesting qcom improvements in the arm64 upstream tree that may let us get further, but I haven't had a chance to build a new kernel and give it a go

[qzed]

@qzed btw motherboard in dev kit is same as surface pro 9
so do you know if there is some dts for surface pro 9 somewhere ?

I'm not aware of any, sorry.

[shawnguo2]

@nickdepinet I haven't got time to debug, but that RCU stall issue seems caused by cpuidle support. You can try to disable CONFIG_CPU_IDLE and see if it helps.

[jwijenbergh]

Has anyone seen merckhung/linux_ms_dev_kit@195ccc9?

Source: https://twitter.com/merckhung/status/1602205575902683136

[qzed]

That looks like what you'd need to get it booted via a DT. Not sure about the remoteproc firmware paths (from what I've heard they're signed individually for each device, so you might have to copy them from Windows) but those aren't required to get the basics up and running.

[xlazom00]

@qzed can you plz cherrypick that to your 6.2 linux kernel
THX

[qzed]

@xlazom00 I have thought about it, but decided not to do this, sorry. The reason for that is that I can't maintain this. I don't have a dev-kit, so at best I can keep things compiling. The sc8280xp platform is still moving fairly quickly, so it's likely that things will break and I won't be able to fix these things, or even detect any breakage. Due to this, I've decided to focus on the SPX exclusively, which is time-consuming enough at this point.

[xlazom00]

I was able to boot with Johan's Hovald(@jhovold) git
https://github.com/jhovold/linux/tree/wip/sc8280xp-v6.2-rc8
and @shawnguo2 patches on top of that
https://github.com/shawnguo2/linux/commits/devkit-dt-6.2-rc5
DTB
shawnguo2/linux@f3e0834

with this
clk_ignore_unused efi=novamap earlycon=efifb console=efifb
and I pluged in usbc ethernet so I have internet on that

I still have some problems

1. cpu is reporting 3GHz but in real it is runing on something really low as benchmarks are reporting same results on every frequency
2. I see kernel and some OS stuff booting on display connected with displayport but then it go dark. I am able to ssh as usb-c ethernet is working

[CriXYZ]

Greeting! I'm very novice in linux, help me , please instaley and a version as complete as possible for 7c s book go. I found the link with the sd image but I can not download to the end greenblue81@protonmail.com Regards!!!

[qzed]

@CriXYZ This is not the right place for this question, I don't think anyone here can help you with that device. Maybe people at https://github.com/aarch64-laptops can help you, but installing Linux on those devices is generally not something that I can recommend for a beginner.

[rgn]

I was able to boot with Johan's Hovald(@jhovold) git https://github.com/jhovold/linux/tree/wip/sc8280xp-v6.2-rc8 and @shawnguo2 patches on top of that https://github.com/shawnguo2/linux/commits/devkit-dt-6.2-rc5 DTB shawnguo2/linux@f3e0834

with this clk_ignore_unused efi=novamap earlycon=efifb console=efifb and I pluged in usbc ethernet so I have internet on that

I still have some problems

1. cpu is reporting 3GHz but in real it is runing on something really low as benchmarks are reporting same results on every frequency
2. I see kernel and some OS stuff booting on display connected with displayport but then it go dark. I am able to ssh as usb-c ethernet is working

@xlazom00 Awsome! Would you mind sharing the image/iso? Would love to test it as well.

[thelumberjhack]

I was able to boot with Johan's Hovald(@jhovold) git https://github.com/jhovold/linux/tree/wip/sc8280xp-v6.2-rc8 and @shawnguo2 patches on top of that https://github.com/shawnguo2/linux/commits/devkit-dt-6.2-rc5 DTB shawnguo2/linux@f3e0834
with this clk_ignore_unused efi=novamap earlycon=efifb console=efifb and I pluged in usbc ethernet so I have internet on that
I still have some problems

1. cpu is reporting 3GHz but in real it is runing on something really low as benchmarks are reporting same results on every frequency
2. I see kernel and some OS stuff booting on display connected with displayport but then it go dark. I am able to ssh as usb-c ethernet is working

@xlazom00 Awsome! Would you mind sharing the image/iso? Would love to test it as well.

I second that, but a quick write up on how you did it if you can would be preferred.
Thanks

[mjrc]

Is there any progress on this - by any luck?

[xlazom00]

Sorry :)

I made this branch of armbian
https://github.com/xlazom00/build/tree/qcom-sc8280xp

1. You will need to build image like this
   ./compile.sh EXPERT="yes" KERNEL_GIT=shallow BOARD=microsoft-devkit-2023 BRANCH=edge RELEASE=kinetic BUILD_MINIMAL=yes BUILD_DESKTOP=no KERNEL_CONFIGURE=no COMPRESS_OUTPUTIMAGE=sha,img EXTRAWIFI=no WIREGUARD=no PLYMOUTH=no PREFER_DOCKER=no

2. flash image to sd card or usb dongle with this for example https://www.balena.io/etcher

3. disable secure UEFI on device and change order to boot from usb first

Kernel will use ACPI to setup hw
USB ports working(ethernet port),
PCI also(wifi only need blobs from linux firmware)
cpufreq driver don't so device will run on some low freq
gpu driver don't

armbian will use https://github.com/jhovold/linux/tree/wip/sc8280xp-v6.3-rc4
and ACPI paches from here https://github.com/shawnguo2/linux/commits/wdk2023-acpi-6.2

Thank you for @jhovold and @shawnguo2 work!

Now it will be nice to have device tree version with all drivers
in DT version
cpufreq is working
missing one usb port(one with usb-ethernet)
display port dont
....

[merckhung]

Hi so I have booted DevKit 2023 to Ubuntu 23.04 desktop. But I am still using 6.0 kernel. I will try your kernel and see if I am lucky.

[merckhung]

So, I have switched to Steev’s 6.3-rc4 kernel with your ACPI patches. I feel the system (Ubuntu 23.04) runs smoother and faster than before (higher CPU freq?), but the GPU DRM is not detected, so it’s still software rendered.

I took a look at my X13s dmesg, I believe x13s’ GPU/DRM is detected through DTB (overlay?), instead of ACPI. Thinking about having a hack in the DRM msm_dpu driver.

[xlazom00]

@merckhung this one https://github.com/steev/linux/commits/lenovo-x13s-linux-v6.3-rc5 ?

[JeromeDeBretagne]

How about SSH?

SSH is working fine, over Wi-Fi or Ethernet, since my first message. That's how I am doing my follow up investigations and improvements on my Debian install running on the NVMe drive.

[tilator]

How about SSH?

SSH is working fine, over Wi-Fi or Ethernet, since my first message. That's how I am doing my follow up investigations and improvements on my Debian install running on the NVMe drive.

Is it just off by default? The problem is, I don't have any kind of control to the device after booting. No screen, no keyboard (I assume because I can't see if it works or not), no SSH access either.

Ethernet is up and running, but no means to access the device.

What do I do wrong or what should I do other way to get SSH access?

[jglathe]

How about SSH?

Fighting with my new image, not sure who's winning. Update tomorrow.

Update on this: The image won, I'm respawning from the (working) 6.6.0 image and do my mods there - again. Can't get it to do my bidding for unknown reasons, something systemd /snap related I'm afraid.

• extend root partition suddenly doesn't work anymore
• snap is really... unfriendly but comes up eventually
• (worst) this buggered user setup (ubiquity) decides to crash and you have to kinda do parts of it again
• sshd thinks it doesn't have to talk to me

All the firmware / driver setup works nicely, but what use is it without the GUI you're putting up with. So back to the drawing board I guess.

[jglathe]

Is it just off by default? The problem is, I don't have any kind of control to the device after booting. No screen, no keyboard (I assume because I can't see if it works or not), no SSH access either.

Ethernet is up and running, but no means to access the device.

What do I do wrong or what should I do other way to get SSH access?

@JeromeDeBretagne uses a different installation AFAICT. With the Ubuntu image, yes sshd is actually not only off, you have to install it. And you need a user that is already set up.

[tilator]

yes sshd is actually not only off, you have to install it. And you need a user that is already set up.

Which is all impossible without SSH or any other kind on UI.

[jglathe]

Which is all impossible without SSH or any other kind on UI.

yep. me wrestling with a new image and cursing.

[JeromeDeBretagne]

@JeromeDeBretagne uses a different installation AFAICT. With the Ubuntu image, yes sshd is actually not only off, you have to install it. And you need a user that is already set up.

Indeed, I'm using the official Debian netinst image(s) with my custom .dtb file so far, the built-in screen works during the whole installation. Then I boot the SP9 5G from the installed system and I can ssh into it (as I still don't get any screen working yet) as it keeps the Wi-Fi network parameters from the installation phase.

[jglathe]

Argh. This debugging work continues (found a few, still some to go). But, @tilator you can edit the image and do the installations you might need yourself.

What you need: a Raspberry pi 4 or some other aarch64 device with a console or tty. I have used a pi 4b for the initial images, worked nicely. And was a good fallback for the next try. An external SSD whith USB (c) to nvme or SATA adapter. This one will be written a few times. SSD is way faster than USB sticks, in my experience.

The easiest way would be to just burn the image to the SSD, mount it on the pi4, do the editing. Usually you need a console that is chroot'ed into the file system, and one that copies over things from outside.
If you want to do the edits in the image file, though, and burn it later, this is the way to do it:

loopf="$(sudo losetup -f)"
sudo losetup "$loopf" ./Ubuntu_Desktop_23.10_wdk2023_6.6.0.img 
sudo partprobe "$loopf"
sudo mount "$loopf"p2 /mnt/
sudo mount "$loopf"p1 /mnt/boot/efi/
for i in /dev /dev/pts /proc /sys /run; do sudo mount -B $i /mnt/$i; done 

This mounts the image file via loop and all its partitions. You can do a sudo chroot /mnt/ and do almost everything in there that doesn't require running services. apt works, snap not really. Installing ssh should be no problem, compromising security by setting a root password either. You can also do update-initramfs and update-grub. When you're done, you're exiting the file system and need to properly unmount it:

for i in /mnt/dev /mnt/dev/pts /mnt/proc /mnt/sys /mnt/run; do sudo umount -l $i; done
sudo umount "$loopf"p1
sudo umount "$loopf"p2
sudo losetup -d "$loopf"

The image can be written to an USB stick (16GB minimum) or SSD and is bootable. For writing to the device I use the disks utility.
You can also do this on an x86 machine but you need binfmt and qemu for this.

[jglathe]

The new image for Kernel 6.7.1, Ubuntu 23.10 can be found here. It's the "secure" image, no root pw changed, no sshd installed. It installs pretty smoothly, fetches the sc8280xp firmwares from the Windows partition, extends the rootfs, has cmdline option to see which firmware is loaded, has current WLAN firmware and a board file good for the WDK module, and uses bootmac to generate constant mac addresses. A suggested hack for the SP9 5G will follow. But not today.

[JeromeDeBretagne]

Quick update on my side, I have finally been able to get display working on the SP9 5G, re-using a mix of .dts / .dtb files. Display output is working for external screens plugged into one of the 2 USB-C ports at the moment, the bottom one to be specific.

Compared to my past attempts, I have realized that the 4 .jsn files were missing next to the .mbn firmware files extracted from the Windows partition (they were not in the same folder) so I've fixed it today and that's certainly what was missing previously.

Let's the fun continue!

[jglathe]

Display output is working for external screens plugged into one of the 2 USB-C ports at the moment, the bottom one to be specific.

Yay 🎆 Is it only the bottom one? Curious, I have seen a dependency here on the WDK, too. It's on the list. And I guess that would be USB-C 0. Do you have a display with an USB-C connector, or do you use an adapter?

the 4 .jsn files were missing next to the .mbn firmware files extracted from the Windows partition

I copied them because they looked interesting. These json files define service endpoints and qmi_instance_id for apparrently the user space side of things, what is running on those co-processors. The packages are alsa-ucm-conf protection-domain-mapper qrtr-tools Sound doesn't work to the DP devices yet. Those co-processors are part of what makes the USB-C multiplexer tick, though, so maybe they were not starting up properly without it.

Can you please push your dts so that I can use it as base for further tweaks? I would suggest to use a separate branch for this. According to Gus' repo the code name would be Arcata, so maybe use that name for the dts too?

[tilator]

Argh. This debugging work continues (found a few, still some to go). But, @tilator you can edit the image and do the installations you might need yourself.

Thanks. Unfortunately this Surface is the only proper ARM64 device I have at the moment. I have an old Nvidia Shield TV too, but it's too hard to use it.

I did try to edit some files by just mounting the installation stick to an X64 PC with Ubuntu, but it did not work. So I'm still without any UI.

[tilator]

Quick update on my side, I have finally been able to get display working on the SP9 5G, re-using a mix of .dts / .dtb files. Display output is working for external screens plugged into one of the 2 USB-C ports at the moment, the bottom one to be specific.

Sounds good. Is it somewhere available as a disk image to give it a try?

[jglathe]

@tilator that was quite a challenge, but its working now:

This is after the box boots up (twice) and leaving it untouched, trying to login from outside.

Image is compressing now, will upload soon.

[jglathe]

@tilator @JeromeDeBretagne
There it is, Image with open root access and ssh is available. Password: FsecuritY!

Would be good to get dmesg for a start.

[tilator]

Here you have a dmesg.
dmesg.txt

It seems to fail loading FW from Windows. Bitlocker is not on. I did double check it.

[tilator]

BTW: SSH is accessible while it boots up first time. After automatic reboot there is no SSH because network does not come up again.

It negotiates TCP address while booting first time, but not at the second time. I did try it with two different USB-RJ45 adapters and two different routers.

The dmesg was taken after first boot before it rebooted itself.

[jglathe]

Hmm the reboot is intended, its after extend-rootfs and firmware fetcher. hmm. Thanks for the dmesg, I'll take a look

[tilator]

Hmm the reboot is intended, its after extend-rootfs and firmware fetcher. hmm. Thanks for the dmesg, I'll take a look

Would it be any sense in disabling root fs extending to find out if it causes this network trouble? Second try would be disabling FW operation.

Making it boot properly with network and SSH might help us a lot at this point.

[jglathe]

First analysis: This really looks like the same mainboard. dmesg looks normal for a boot without firmware files... the r8152 messages look like from the internal USB adapter (strange as this is). Do you have an additional dock connector on this? Can't see any messages from any other USB ethernet adapters, though. The WLAN driver is good news: It's the same board as in the Windows Dev Kit, so the loaded board file should give you some pretty useful connectivity over WLAN and Bluetooth. @JeromeDeBretagne this could help you get better connectivity, drivers are in the wdk2023_syshacks repo.

Since I assume you just tried to "turn it off and on again" and didn't get further, what I could do:

• disable extend-rootfs
• disable reboot after fetching the firmwares.

The firmwares would be loaded at next reboot, but you could set up WiFi connection via command line, no prob. This will be remembered afair.

I will do a new image with these two tweaks, but the process takes at least an hour.

[tilator]

Would it be good idea to put the missing firmware files in the filesystem before trying to boot it at all? So that it can load those files properly at the first place. This might cut many errors making it boot further.

[jglathe]

With the new config this is something you get at second boot anyway. Won't publish these files, they are not mine. New image is compressing, upload soon.

[tilator]

the r8152 messages look like from the internal USB adapter (strange as this is). Do you have an additional dock connector on this?

This device has a combined connector for power supply and some kind of addition/dock. They seem to call it "Surface connector". Maybe the message comes from it.

But the USB-RJ45 adapter I used might have Realtek chip in it too. It's an other possibility.

[jglathe]

Well the surface dock has a separate driver. I guess not included yet...

[jglathe]

File is available here. Since this will take a while: Open a new discussion or issue in my repo?
If you want to grow the rootfs, its this: systemctl start growroot-almost.servicehere
I'll take a look at the surface dock driver, saw an option in the kernel config.

[tilator]

Thanks. It's up and running now.

Is it normal, that according to dmesg it does try to find the FW files from

/lib/firmware/...

but according to find -iname 'qccdsp8280.mbn' only copy seems to be in

/usr/lib/firmware/...

I did not dare to reboot it yet and we have to continue tomorrow anyways.

Well - I rebooted it now. No SSH any more and I suppose it's the missing network again. I don't have time now to evaluate it more. I can continue tomorrow.

[jglathe]

Oh that's good news. The firmware for WDK2023, like for the SP9 5G, is not in mainline yet, and may never be. I used the /usr/lib/firmware/updates/ path intentionally to ensure no collisions. It is checked before /lib/firmware/. Do try to reboot, I'm curious 😈

[tilator]

I'll open a new discussion now to leave this for DEVKIT_2023 as it is supposed to be.

[JeromeDeBretagne]

Can you please push your dts so that I can use it as base for further tweaks?

For the Microsoft Surface Pro 9 5G update, here you go! I've pushed my latest .dts and .dtb files in this commit :
arm64: dts: qcom: sc8280xp-sp9-5G: Enable external display

I have also cleaned up the .dts file in the previous commit to remove all the entries that don't apply/work on the Surface Pro 9 5G so I consider it the minimum .dts / .dtb files to boot and install Linux on the NVMe drive through Wi-Fi (or a USB Ethernet adapter) with external display support (working only on the bottom USB-C port). It's working fine using recent Debian netinst images based on Linux 6.7 and it is perfectly stable, yeah!

I have used the following path for the firmware files "qcom/sc8280xp/MICROSOFT/SurfacePro9/" by the way.

Sorry for the lag, I've spent a lot of time (way too much in fact) trying to get external display working on the second (top) USB-C port too, without success so far despite trying mdss0_mdp2, mdss0_dp3, etc. I wonder if the first "connector@0" section in the .dts file could apply to the Surface Connect port instead (as it seems based on USB using a different connector), but I don't have the right accessory to check this assumption.

If that's the case, getting the second (top) USB-C port to work would require to find the correct GPIOs driving the USB SBU (side band use pins) to create a new usbX-sbu-mux section I imagine. However I have no clue how to find that out :-( @merckhung @andersson @jhovold do you have any suggestions, either on Linux (I've tried gpiod, gpiomon, acpidumps, etc.) or through reverse engineering from the Windows installation? Thanks a lot if you have any tips you can share!