Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Need AArch64 / ARM64 versions #177

Open
Googulator opened this issue Oct 28, 2017 · 262 comments
Open

Need AArch64 / ARM64 versions #177

Googulator opened this issue Oct 28, 2017 · 262 comments
Assignees

Comments

@Googulator
Copy link

@Googulator Googulator commented Oct 28, 2017

Microsoft is now building Windows 10 and Windows Server for ARM64 machines. To successfully run these in qemu, virtio drivers are needed for the ARM64 platform.

@YanVugenfirer
Copy link
Collaborator

@YanVugenfirer YanVugenfirer commented Oct 28, 2017

It is on our roadmap.
In a meanwhile, could you please share how and what versions of Windows Server you try to run on QEMU, so we will align our efforts.

Thanks.

@vrozenfe
Copy link
Collaborator

@vrozenfe vrozenfe commented Oct 28, 2017

While it is not on our radmap, it is on our wishlist :)
I believe we can give it a try as soon as we can run ARM Windows VM image on QEMU.

Best,
Vadim.

@sameehj
Copy link
Contributor

@sameehj sameehj commented Nov 16, 2017

Hi @Googulator,

We are currently attempting to run Windows Server on top of qemu and didn't manage to fully boot Windows. In case you made any progress on that can you please share the it with us so we can port the drivers ASAP.

@Googulator
Copy link
Author

@Googulator Googulator commented Nov 17, 2017

In qemu v2.11-rc1, Windows 10 version 1709 (build 16299) for "ARM64", as Microsoft calls it, can now install and boot using this UEFI firmware image: https://github.com/Googulator/edk2/releases/tag/AHCI

Extract the contents of the ISO into a 5GB VHDX or other qemu-compatible image format, create a 2nd, empty VHDX of about 50GB, then run qemu like this (change "-smp 8" if you don't want to emulate that many cores):
qemu-system-aarch64 -M virt -m 8192 -cpu cortex-a57 -smp 8 -bios QEMU_EFI.fd -device VGA -device nec-usb-xhci -device usb-kbd -device usb-tablet -drive file=50gb.vhdx,id=install,if=none -netdev user,id=network -device virtio-net,netdev=network -device ahci,id=ahci -device ide-hd,drive=install,bus=ahci.0 -drive file=5gb.vhdx,id=dvd,if=none -device ahci,id=ahci2 -device ide-hd,drive=dvd,bus=ahci2.0

Walk through setup as you would on an x86 system.

Probably won't work under KVM on real ARM hardware, as it depends on the emulated VGA framebuffer, which is fundamentally incompatible with KVM. Unfortunately, virtio-gpu can't be used as an alternative because Windows requires runtime graphics capability through UEFI GOP after ExitBootServices(), which means it can't run on PixelBltOnly screens like the one provided by virtio-gpu.

EDIT: 2.11-rc1 is required, earlier versions have some bug related to memory management(?) that cause Windows to blackscreen early during boot.

@Googulator
Copy link
Author

@Googulator Googulator commented Nov 17, 2017

Unrelated issue, but the reason why you have to first extract the image to a VHDX is that simply using the ISO as an ide-cd device causes the qemu host process to crash when you press a key at the prompt to boot from the ISO.

@sameehj
Copy link
Contributor

@sameehj sameehj commented Nov 19, 2017

Hi @Googulator ,

Thank you very much for the instructions and the very detailed explanation. I am following the steps you provided in attempt to boot Windows from the vhdx image but I think I am not preparing the vhdx image from the iso appropriately. Can you please shed some light on how you did it? (tools, special configurations, etc..)

@oscarbg
Copy link

@oscarbg oscarbg commented Jan 11, 2018

@Googulator +1 can you post steps to prepare vhdx from ISO.. seems my virtual disk may be not bootable or badly created..

@huoqianyu
Copy link

@huoqianyu huoqianyu commented Jan 12, 2018

@sameehj , Hello friends, Today I found that a user called "LightBulbFun" in forum "BetaArchive" successfully installed Windows 10 ARM64 usinig an older firmware. You can see this link
https://www.betaarchive.com/forum/viewtopic.php?f=62&t=37172&start=50
for more information. By following his guide, I successfully installed Windows 10 build 16299.15.

2018-01-12 4

P.S. The screenshot shows a small "HELLO WORLD" program created by Visual Basic 6, and that may means x86 program can run in ARM64 win10.

@sameehj
Copy link
Contributor

@sameehj sameehj commented Jan 14, 2018

@huoqianyu,

That's great! We'll be trying to follow his instructions and update you on the progress =)

@falkor2k15
Copy link

@falkor2k15 falkor2k15 commented Apr 14, 2018

Any news on the KVM incompatibility issue and any efforts that are being taken to get around it? Is it something that will eventually be fixed in Qemu-system-aarch64?

What is the "emulated VGA framebuffer" exactly and why would this cause an incompatibility with KVM? Surely there must be a way around it?

@YanVugenfirer
Copy link
Collaborator

@YanVugenfirer YanVugenfirer commented Apr 16, 2018

As was mentioned in the previous comment by @huoqianyu it is possible to run Windows on QEMU now.

@falkor2k15
Copy link

@falkor2k15 falkor2k15 commented Apr 16, 2018

Yes, but it runs too slow, and the KVM acceleration problem is stopping it from running fast on ARM64 hardware. So my question is specifically about the KVM incompatibility: is it a problem with Qemu or KVM or Windows? Any efforts being made to resolve it? Is there a way around it?

@vrozenfe
Copy link
Collaborator

@vrozenfe vrozenfe commented Apr 17, 2018

Can you give a try to Linux arm64 guest and see if it performs better than Windows?

@falkor2k15
Copy link

@falkor2k15 falkor2k15 commented Apr 17, 2018

The ARM64 guests all have graphics turned off using the -nographic option - so I don't think they can be used with VGA output - for example see this:
https://wiki.ubuntu.com/ARM64/QEMU

KVM may also work with no graphics, hence the compatibility problem has something to do with the graphics, but it hasn't been discussed anywhere.

@vrozenfe
Copy link
Collaborator

@vrozenfe vrozenfe commented Apr 17, 2018

Headless configuration should work for Windows Server as well. But we might be able to build ARM64
version of virtio-gpu dod driver to give it a try.

@falkor2k15
Copy link

@falkor2k15 falkor2k15 commented Apr 17, 2018

I don't know where to find the Windows Server 2016 on ARM ISO - I think it was given to selected people only. If you could please build a new graphics device to get KVM to work in conjunction with Qemu-System-Aarch64 + Windows 10 on ARM then that would be great - I'm all ears and waiting day-by-day to test this!

@driver1998
Copy link

@driver1998 driver1998 commented Apr 25, 2018

Of cause you can boot Windows without VGA output, but that will only be a black screen, the OS should still be working though.
And if drivers for virtio-gpu is available, we may be able to use that.

@falkor2k15
Copy link

@falkor2k15 falkor2k15 commented Apr 25, 2018

There are currently no drivers available - they would have to be made for Windows on ARM; however, that would not be a solution, but a workaround to the problem of no VGA support in Qemu coupled with KVM compatibility issues. I can only assume this is some kind of deliberate joke to put out headless server scripts and hold back on allowing the public to visually experience modern hardware virtualization. A blind man should be grateful just to walk and talk - even though he cannot see where he's going?

@Bullseye300
Copy link

@Bullseye300 Bullseye300 commented Jun 11, 2018

For those who have problems by installing QEMU. I found a ready to run image with Windows pre-installed:
https://virtuallyfun.com/wordpress/2018/03/22/windows-10-arm64-on-qemu/

I think that needs to be added in the start script (Driver still missing):

-net nic,model=virtio ^
-net user ^

@YanVugenfirer
Copy link
Collaborator

@YanVugenfirer YanVugenfirer commented Nov 19, 2018

You are welcomed to check the branch: https://github.com/virtio-win/kvm-guest-drivers-windows/tree/arm64

Instructions for build machine (using community edition of VS2017):
https://github.com/virtio-win/kvm-guest-drivers-windows/wiki/Building-the-drivers#coming-soon-building-virtio-win-drivers-including-arm64-using-visual-studio-2017

Soon will be integrated to master and we will distribute the binaries.

Best regards,
Yan.

@falkor2k15
Copy link

@falkor2k15 falkor2k15 commented Nov 20, 2018

Thanks Yan! Is there a pre-built binary I could use to test with qemu-system-aarch64/KVM?

@ybendito
Copy link
Collaborator

@ybendito ybendito commented Nov 22, 2018

@falkor2k15
I can send built binaries (signed with test signature).
Which ones you want to try? (This is not an official build with complete tree, so I'll pick up ones you're interested in)

@falkor2k15
Copy link

@falkor2k15 falkor2k15 commented Nov 22, 2018

@ybendito, yes please!! :) Is there a VGA one you could send me as that's the main problem with the qemu-system-aarch64 and KVM. For example; is there a VGA QXL?
-vga qxl
And how about this one?
-net nic,model=virtio
Apparently, this ARM64 stuff is meant to be quite easy to build using the Latest Visual Studio (2017):
https://youtu.be/vdYIaUeZnqc?t=1047
However, I'm not a dev, so I wouldn't know.
Finally, if you do manage to build at least a VGA driver then how might we import them into a qemu-system-aarch64 script? Thanks and looking forward to it!

My email is gilius2k15@gmail.com

I'm not sure if you guys are aware of this new QEMU machine (xilinx-zynq-a9) and new ARM CPU (A72)?
https://www.xilinx.com/support/documentation/sw_manuals/xilinx2017_4/ug1169-xilinx-qemu.pdf

Would be good to test with a53, a57 and a72.

Edit: I don't think the QXL one is a virtio driver. I think it's just called -vga virtio or -virtio-vga or virtio-gpu

@ybendito
Copy link
Collaborator

@ybendito ybendito commented Nov 23, 2018

@falkor2k15 qxl-wddm-dod is not a part of virtio-win, we did not build it for ARMx and currently we do not have an estimation for that.
virtio-net driver is available for trial: https://drive.google.com/open?id=1fBBysiREXap4KSG2ImaY5IUp_yLIDt2x

@falkor2k15
Copy link

@falkor2k15 falkor2k15 commented Nov 23, 2018

Looking promising...
capture
Just need to try to install this windows so I can get to the desktop - might need to switch to Linux host... Qemu doesn't like my i7.

@ybendito
Copy link
Collaborator

@ybendito ybendito commented Nov 23, 2018

@falkor2k15 I think you play with some drivers set that is not related to our build, I've sent only netkvm.

@driver1998
Copy link

@driver1998 driver1998 commented Nov 23, 2018

I am testing Windows ARM64 on KVM, could you please send me the viostor bits? @ybendito

@ybendito
Copy link
Collaborator

@ybendito ybendito commented Nov 23, 2018

@driver1998 On which platform(chipset)?
I do not think you can use viostor at Windows setup, for that the driver shall be signed by Microsoft. But you can add virtio disk later if you enable test signature. https://drive.google.com/open?id=1FxOxFoONccJYdTaUnVs_YQ145O_FePZF

@falkor2k15
Copy link

@falkor2k15 falkor2k15 commented Sep 16, 2019

@WaseemAlkurdi, just a quick reply for now: what's the significance of the patch linked by @fcicq ? Is it only for the Jetson SBC he mentioned? What's it meant to do exactly?

About MBR: Jonathan mentions in the book that the first byte is technically MBR and the rest GPT and maybe some stuff about false positives; there's a reason for it anyhow. I am in the process of re-reading it again so will let you know.

@falkor2k15
Copy link

@falkor2k15 falkor2k15 commented Sep 16, 2019

The Samsung S8 kernels all compile successfully with KVM!

The official Kirin 970s failed - except for this unofficial one - compiles successfully albeit has hundreds of warnings/errors!
https://github.com/MZO9400/kernel_huawei_berkeley

@fcicq
Copy link

@fcicq fcicq commented Sep 17, 2019

@falkor2k15 if your kernel is older than 4.11, and there is some code in qemu >= 2.6 says

    if (kvm_enabled() && !kvm_arm_supports_user_irq()) {
        error_setg(errp, "KVM with user space irqchip only works when the "
                         "host kernel supports KVM_CAP_ARM_USER_IRQ");
        return;
    }

on this case / error message you will want to try my patch. otherwise that is not required. nvidia L4T kernel cant be upgraded and is currently stuck at 4.9.

@falkor2k15
Copy link

@falkor2k15 falkor2k15 commented Sep 17, 2019

@WaseemAlkurdi my understanding is that the XBL is the UEFI - not the ABL/Aboot. Jonathan mentioned in his book that most devs don't touch the bootloaders - working only at the Linux kernel level and above. I heard from various sources that the bootloaders are hard coded executables (probably the OBJ files I saw extracted). Therefore, in order to modify the bootloaders you first need the source code to recompile them in a modified state? However, Jon's article mentioned that aboot was partially open source (possibly now complete if fastboot.c is complete) - though he mentioned about "disassembling" it. All that is a bit of a contradiction to his other statement "anything is possible with an unlocked bootloader", and raises the question as to exactly what is within our power given the technical knowhow? Zhuowei (author of the Snapdragon article) mentioned this to me recently:

"Booting Windows 10 was my original goal, but after seeing the astonishing number of components in an UEFI image, I realized that it would be almost impossible to modify and repack one.

I did try creating a UEFI EDK2 port from source code instead, without using the original firmware: that didn't work since I don't know how to write drivers.

Others had more success: @NTAUTHORITY booted Windows on a Pixel 3XL by extracting drivers from the stock UEFI firmware. Unfortunately, their work isn't public, but if that's something you're interested in, you should contact them."

@WaseemAlkurdi
Copy link

@WaseemAlkurdi WaseemAlkurdi commented Sep 17, 2019

@falkor2k15,

my understanding is that the XBL is the UEFI - not the ABL/Aboot.

On Snapdragon 835 and above, ABL is a UEFI application that is loaded by the UEFI XBL.
On Snapdragon 821, UEFI XBL loads aboot as a legacy non-UEFI bootloader.

Jonathan mentioned in his book that most devs don't touch the bootloaders - working only at the Linux kernel level and above.

And there's a good reason for that. If a bad bootloader is flashed, which is almost definitely the case for a bootloader-in-development, and the device won't start because of that, the device is a sleek paperweight, and the eMMC would have to be desoldered, flashed with a known-good bootloader, and resoldered. And you have to have a debug interface, like an exposed UART pad on the phone's motherboard or a serial via headphone jack like on the Nexus and Pixel phones. Not worth it for 90% of people. If the second bootloader can be chainloaded without affecting the first one, however, things are different, and devs would take the plunge, like when U-Boot was ported to the Galaxy Nexus.

I heard from various sources that the bootloaders are hard coded executables (probably the OBJ files I saw extracted).

They are binary blobs, containing code that manufacturers won't release so that people won't catch them red-handed with stolen code because they are trade secrets.

However, Jon's article mentioned that aboot was partially open source (possibly now complete if fastboot.c is complete)

That's aboot. But what about the part that sits before this? PBL/SBL/XBL/HBOOT/S-BOOT/... ? And now Qualcomm seems to have gone one step ahead with its UEFI bootloaders, but two steps back when it replaced open-source aboot with a closed-source ABL.

"Booting Windows 10 was my original goal, but after seeing the astonishing number of components in an UEFI image, I realized that it would be almost impossible to modify and repack one."

Doesn't that feel a little non sequitur? We aren't going to lose tiny screws or something ... it's a software image. It wouldn't matter whether there is one or one million images/drivers/whatever ... it either could be, or could not be, unpacked and repacked.
Unless cryptographic verification is involved.

"I did try creating a UEFI EDK2 port from source code instead, without using the original firmware: that didn't work since I don't know how to write drivers."

That's very sensible, and I appreciate him for doing that. But why stop there? Why hadn't he added drivers from the original image, or at least tried to?

@NTAUTHORITY booted Windows on a Pixel 3XL by extracting drivers from the stock UEFI firmware. Unfortunately, their work isn't public, but if that's something you're interested in, you should contact them."

I had tried to contact NT AUTHORITY through various channels - my attempts would go unanswered.

But isn't it now clear? A UEFI port should be created and chainloaded from the stock bootloader so that one doesn't permanently lose the device.

@falkor2k15
Copy link

@falkor2k15 falkor2k15 commented Sep 17, 2019

@WaseemAlkurdi, regarding the first point - yeah you are right according to this article - so I stand corrected:
https://lineageos.org/engineering/Qualcomm-Firmware/

We can see how QC done away with the old system and then spearheaded UEFI!

@falkor2k15
Copy link

@falkor2k15 falkor2k15 commented Sep 17, 2019

@WaseemAlkurdi I finally managed to get the Realme 2 Pro kernel with Snapdragon 660 to compile with KVM enabled. It boots up, but there's no /dev/kvm present - probably because of this fix I applied...

The problem that was causing KVM to fail seems to be a bug* in /arch/arm64/kernel/cpu_errata.c that referenced a function named "bp_hardening".

*My phone was created using 4.4.78, but since then it's been updated to 4.4.193, and that whole "bp_hardening" code (and references to it) across /arch/arm64 have been completely removed! So I manually removed the main code and few references till I got a successful compile, but then I don't know if there's a new function that's replaced it, and without a substitute there's no guarantees KVM is in a working state.

@falkor2k15
Copy link

@falkor2k15 falkor2k15 commented Sep 23, 2019

Damn, I bricked my Realme 2 Pro because I had some problems downgrading Pie to Oreo; need to research why that's the case - had similar problems on Samsung S8 too.

With the S8+ I've now compiled KVM without a single error or warning - and can confirm it boots - but strangely there's no /dev/kvm again!?

I will still try to test it over the next week - but it's not looking promising. I wonder what the problem is this time as we are using an Exynos instead of Snapdragon. Also the maintainer of Limbo was the first person to announce KVM-enabled kernels for the S8 - but then his website got taken down. Was he really successful I wonder?

A lot will rest on Linux deploy and finding a distro with a counterpart KVM-enabled kernel.

Incidentally, although the Samsung S5 Neo was the first 64-bit phone, it's not until the 3.11 kernel that KVM was added to Android (ARM64); therefore, S5 Neo - happens to be based on 3.10 - has no Virtualization modules available whatsoever - at least not for ARM64 architecture. And not many phones seem to be getting ported to newer base kernels. Right now, makemenu config has 3-4 options under Virtualization for kernel version 4.4 - but for S7/S7 Edge there's only a single KVM option under kernel version 3.18 - and that also compiles quite well. I couldn't get S6,S9,S10 to compile - but there could well be clean kernel sources out there that I haven't come across.

@WaseemAlkurdi
Copy link

@WaseemAlkurdi WaseemAlkurdi commented Sep 23, 2019

Damn, I bricked my Realme 2 Pro because I had some problems downgrading Pie to Oreo; need to research why that's the case

It's almost universally because of incompatible bootloader versions, where Pie on a device would upgrade the bootloader to a version that won't boot Oreo. This is done to prevent a would-be attacker from downgrading the device to use exploits for older versions of Android, and would work in theory ... except that OEMs suddenly stop caring about users and old Android versions once the device is past the mandatory 18-month update schedule.
But beware of downgrading bootloaders ... most of them (but not all) have anti-rollback mechanisms that brick a device (for real this time, no lights, no fastboot) if a downgrade is attempted.

With the S8+ I've now compiled KVM without a single error or warning - and can confirm it boots - but strangely there's no /dev/kvm again!?

I will still try to test it over the next week - but it's not looking promising. I wonder what the problem is this time as we are using an Exynos instead of Snapdragon.

Now that is a cause for concern. Definitely odd. But might just be an anomaly. Did you check dmesg, especially for a line that says HYP not available? That's where I got the death blow in my case.

Also the maintainer of Limbo was the first person to announce KVM-enabled kernels for the S8 - but then his website got taken down. Was he really successful I wonder?

The website has vanished because it was a Weebly website done using Weebly's free tier. Perhaps @limboemu got tired of Weebly's banners all over their site?
Note what was taken down is the whole website - not the single announcement, and that the text for the announcement said that the author didn't test this kernel.

A lot will rest on Linux deploy and finding a distro with a counterpart KVM-enabled kernel.

Why would it matter that a distro has a counterpart KVM-enabled kernel, if said kernel is not being booted?
You can test using Linux Deploy and your own build of qemu-system-aarch64. Distro support doesn't matter (since you aren't booting that distro's kernel). KVM has to be enabled and a corresponding dmesg entry has to be found.

I couldn't get S6,S9,S10 to compile

The Galaxy S6's kernel build woes are understandable. I looked that device up since we happen to have a spare S6 Edge that keeps changing hands in our extended family, but I can't really root it, since rooting breaks either the fingerprint reader or the KNOX trigger - and people objected to breaking any of these.

but there could well be clean kernel sources out there that I haven't come across.

There is indeed a build guide that uses Samsung's official kernel source. (Link here: https://forum.xda-developers.com/galaxy-s6/general/guide-how-to-build-samsung-kernel-july-t3429355)

@falkor2k15
Copy link

@falkor2k15 falkor2k15 commented Sep 23, 2019

@WaseemAlkurdi if we have the original stock firmware then which parts of the restore does TWRP ommit when downgrading Pie to Oreo? It implies that some parts of the internal partitioning are never touched by TWRP despite restoring XBL and ABL - would make a good question for Jonathan too, I reckon. The follow up question would be: can we then use adb or fastboot for carrying out a complete "wipe and load" of a stock firmware?

What is dmesg? Is it just a command? I don't see any hyp mode listed here:
dmesg.txt

I realise there's no logic in testing Linux Deploy, but could the KVM still be enabled despite the lack of /dev/kvm or perhaps the android kernel has some strange quirks that we don't know about? When I compiled the kernel I selected all 3-4 options under KVM - but neither Realme nor S8 showed/shows this being enabled despite the kernel replacement being sound and, via uname -a, displaying today's date, etc. Presumably we don't have to build any additional library or module files? The KVM is meant to be built-in and there is no modular options available.

Screenshot from 2019-09-23 21-30-27
Screenshot from 2019-09-23 21-29-45

So that leaves the possibility that Android kernels are weird - or all smartphones chipsets block virtualization. I did read that all the Snapdragon devices have that EL2 problem - not just "some" or "most" as we suspected. In turn that just leaves the Kirin 970 as the sole untested SOC - but where to get a P20 Pro with an unlocked bootloader? And right now it's the wrong season for purchasing unlock codes.

@boerck
Copy link

@boerck boerck commented Sep 24, 2019

@falkor2k15 I have Exynos S8 and i builded KVM enabled kernel with limboemu’s guide. you can read KVM guide that limboemu didn’t test kernel on real devices. Kernel building was successful and i can insert it to my device. But i can’t find /dev/kvm too. I think other kernel config or boot loader of arm is problem because zenfone2 can enable KVM.

if you make kernel boot.img of SM-G950N, i can test it on my device.

@falkor2k15
Copy link

@falkor2k15 falkor2k15 commented Sep 24, 2019

@boerck yeah I was thinking about looking back at the Zenfone2 - but glad you've already confirmed. Of course my Linux Deploy test failed as per Waseem's comments on /dev/kvm being a prerequisite - even in the android kernel world.

Could it be that my SD660 device was running LineageOS 15 and my S8+ is running LineageOS 16? I will test using different Custom ROMs - but the boot.img still contains the same kernel and RAM Disk = root file system.

So it may be a while before we get to the bottom of this. I can compare kernel configs with, say, Pi4 Gentoo via the menuconfig - see if anything else looks out of place.

Where to report bugs/issues for all things KVM?

@WaseemAlkurdi
Copy link

@WaseemAlkurdi WaseemAlkurdi commented Sep 24, 2019

@falkor2k15,

if we have the original stock firmware then which parts of the restore does TWRP ommit when downgrading Pie to Oreo?

A regular TWRP backup only covers /system, /data, and on some devices, the modem partitions. Normally, TWRP doesn't even come near the bootloader, as it's far too dangerous to flash bootloaders using anything except fastboot or the device's official fastboot alternative (like Odin3 for Samsung).

It implies that some parts of the internal partitioning are never touched by TWRP despite restoring XBL and ABL - would make a good question for Jonathan too, I reckon.

As far as I know, TWRP never touches either XBL or ABL - again, far too dangerous.

The follow up question would be: can we then use adb or fastboot for carrying out a complete "wipe and load" of a stock firmware?

Yes ... that's the "official" way of upgrading and downgrading bootloaders. But beware, many (if not most) bootloaders have rollback protection.

So that leaves the possibility that Android kernels are weird - or all smartphones chipsets block virtualization.

Android kernels are Linux kernels that use a bunch of out-of-tree drivers. But otherwise they are still Linux kernels. They may be packaged in weird ways, but after all, they are regular ARM/aarch64 Linux kernels.

In turn that just leaves the Kirin 970 as the sole untested SOC

And MediaTek SoCs ... but does MTK have any aarch64 SoCs?

but where to get a P20 Pro with an unlocked bootloader? And right now it's the wrong season for purchasing unlock codes.

There are rumors (I haven't even checked, but I have heard them) that Huawei is considering allowing bootloader unlocks again.

Where to report bugs/issues for all things KVM?

The Linux Kernel mailing list - seeing that KVM is a Linux kernel component, and the Google git repository/mailing lists for the Android fork of the kernel.

One open question:
The Asus Zenfone family uses Intel Atom CPUs ... but not regular BIOS or (U)EFI.
Therefore, it must be using aboot.
Therefore, the manufacturer should've locked down aboot/LK to disallow KVM, like they do on Snapdragon.
Why isn't this the case?

@boerck
Copy link

@boerck boerck commented Sep 25, 2019

Virtualization is locked in Snapdragon and unlocked in Exynos is True? Where is reference of it?

You can change dmesg size limit by kernel config "CONFIG_LOG_BUF_SHIFT" to 21.
And check dmesg as soon as boot is over to check log from when boot is start (boot timer start - [0.000000]). Can you check it and upload dmesg that start in [0.000000]? I think HYP mode message is on [0.xxxxxx]

@thalyssra
Copy link

@thalyssra thalyssra commented Sep 25, 2019

For what it's worth, the Samsung Galaxybook2 boot with EL2 support, though as far as I know, there's no Linux support for it yet (it's one of the few Windows on ARM64 devices that supports Windows Subsystem for Linux version 2 which requires HyperV)

@WaseemAlkurdi
Copy link

@WaseemAlkurdi WaseemAlkurdi commented Sep 25, 2019

For what it's worth, the Samsung Galaxybook2 boot with EL2 support, though as far as I know, there's no Linux support for it yet (it's one of the few Windows on ARM64 devices that supports Windows Subsystem for Linux version 2 which requires HyperV)

That's good news indeed!
If I can find some time, I'll dig at its firmware with a pickaxe and see how it differs from the UEFI firmware for laptops that have the Snapdragon 855 but without EL2.
I'm optimistic about finding a difference. There is bound to be one.

@falkor2k15
Copy link

@falkor2k15 falkor2k15 commented Sep 25, 2019

Here's that dreaded message on my Samsung S8...
Screenshot from 2019-09-25 23-16-36
Capture

Is this still an issue on Snapdragon 855 phones I wonder!? MediaTek are on Helio P90 now - but refuse to release any kernel sources. There might be a kernel for the P20 though because that was used for a (pretty old) Samsung phone. I'm curious about the Kirin 970 now - and of course Zenfone 2 previously having HYP mode available in an x86 context. There's a Zenfone 6 out now using Snapdragon 855.

I still haven't tested outside of LineageOS - but presumably it wouldn't matter at the userland level? The RAM disk/rootfs starts out with a blank /dev sub-directory that gets filled up as the kernel presumably loads devices and modules. Incidentally, when I tried KVM on a 64-bit Pi4 kernel coupled with 32-bit userland it failed to work - but I don't recall whether KVM appeared under /dev or not. Lineage isn't ARM7 right with an ARM8 kernel? Just a thought anyhow because when HYP Mode was discussed before the word "firmware" was left deliberately vague (see older comments above).

@WaseemAlkurdi
Copy link

@WaseemAlkurdi WaseemAlkurdi commented Sep 26, 2019

Here's that dreaded message on my Samsung S8...

Yeah, there is it, sitting there peacefully like it had done nothing wrong ... :-(

Is this still an issue on Snapdragon 855 phones I wonder!?

The keyword here is phones. If it's Snapdragon 855 laptops we're talking about, then there is definitely no HYP Mode issue, as Windows is being allowed to take over and run the hypervisor.
But does this apply equally to phones? I don't think so. I tend to believe that it's UEFI component mix-and-match, and I'm going to find out when I squeeze just a little more free time and get two Snapdragon 855 UEFI images under my dissecting eye.

MediaTek are on Helio P90 now - but refuse to release any kernel sources.

Ah, right. Totally escaped me. :-(

There might be a kernel for the P20 though because that was used for a (pretty old) Samsung phone.

The one and only Samsung phone that used a MediaTek SoC was the Galaxy J1 Core (or J2 Core, not sure). Its performance is miserable running Android's user interface. What would it be like running Windows in a VM? Don't forget that even though the VM is accelerated, it still has to do the lifting for both CPU and GPU.

I'm curious about the Kirin 970 now

One, however, would have to pay $LOTS_OF_MONEY to get the bootloader unlocked. But let's see whether Huawei's stance on bootloader unlocks changes when the Mate 30 and other Android without Google phones hit the mass market.

and of course Zenfone 2 previously having HYP mode available in an x86 context. There's a Zenfone 6 out now using Snapdragon 855.

The Zenfone 2's bootloader did not opt-out of HYP mode (or perhaps more accurately "Intel VT-x", since this isn't ARM64). The bootloader could have with ease kept it disabled. New laptops ship with virtualization disabled in UEFI/BIOS, unless it's explicitly enabled there by the owner.

However, in no way does this necessitate that the Zenfone 6 should have it enabled. The only thing in common about the two devices is the nameplate and the engineers.

I still haven't tested outside of LineageOS - but presumably it wouldn't matter at the userland level?

It wouldn't change a thing, actually. Not with the same bootloader doing the same blocking.

The RAM disk/rootfs starts out with a blank /dev sub-directory that gets filled up as the kernel presumably loads devices and modules.

You answered your own question! It's the kernel (actually, helped by whatever is providing udev-like functionality in Android) which controls that.

Lineage isn't ARM7 right with an ARM8 kernel?

Not at all.

Just a thought anyhow because when HYP Mode was discussed before the word "firmware" was left deliberately vague (see older comments above).

This is making my firmware decompilation itch even worse! ;-P

@falkor2k15
Copy link

@falkor2k15 falkor2k15 commented Sep 27, 2019

@boerck
Copy link

@boerck boerck commented Sep 30, 2019

https://googleprojectzero.blogspot.com/2017/02/lifting-hyper-visor-bypassing-samsungs.html

In this article, samsung use HYP mode to use KNOX RKP. I think using KVM on S8 would be possible with some options.

@tarikkaya
Copy link

@tarikkaya tarikkaya commented Oct 30, 2019

please s8 kernel share. zip img I don't know how to compile.

@WaseemAlkurdi
Copy link

@WaseemAlkurdi WaseemAlkurdi commented Oct 30, 2019

@tarikkaya What for? If you want to test AArch64 KVM, then don't bother, because it isn't going to work, and you have this whole thread as proof.
If you need it for anything else, then go to XDA-Developers, where they might help you better.

@tarikkaya
Copy link

@tarikkaya tarikkaya commented Oct 30, 2019

@tarikkaya What for? If you want to test AArch64 KVM, then don't bother, because it isn't going to work, and you have this whole thread as proof.
If you need it for anything else, then go to XDA-Developers, where they might help you better.

I'm already looking at all the resources. I'm just investigating and trying. but I didn't learn to compile the kernel.HARD os version, java version, gcc version, other tools versions hard. this is not something I'm always interested in. just a hobby for me. in addition my english is not good.

@tarikkaya
Copy link

@tarikkaya tarikkaya commented Oct 30, 2019

Go to http://opensource.samsung.com/
search for:
s8 or g950F
Download the zip file and extract

Edit file exynos8895-dreamlte_defconfig and add these options:
CONFIG_VIRTUALIZATION=y
CONFIG_KVM=y
Maybe add these also?
CONFIG_KVM_MMIO=y
CONFIG_KVM_ARM_HOST=y

To build:
export ARCH=arm64
export CROSS_COMPILE=PATH_TO_NDK/android-ndk-NDK_VERSION/toolchains/aarch64-linux-android-TOOLCHAIN_VERSION/prebuilt/linux-x86_64/bin/aarch64-linux-android-
make clean
make mrproper
make exynos8895-dreamlte_defconfig
make menuconfig
make

I see but I can't.

@fxsheep
Copy link

@fxsheep fxsheep commented Jan 23, 2020

@WaseemAlkurdi @falkor2k15 I spent half an hour to look through all your comments.I'm from XDA btw.The reason why your post in XDA have 0 reply is that most custom ROM devs simply aren't familiar with low level stuffs like this, and they are already annoyed by some annoying noobs, so they simply ignored as expected.
I used to be interested in running KVM on Android phones too.But now it turns out that there is not much hope as of now.
On Snapdragons this is not possible as HYP is used for QHEE and more, as you can see there's a HYP partition contains the hypervisor (except the msm8916/17 lineup).So EL2 is mandatory for original Qualcomm firmware implementions to run.
And OEMs will of course not make a lot of hard work to workaround this on purpose.

Chainload u-boot to enable EL2 on Qcom? Not possible either, any code simply can't do this,as low ELs can't switch to high ELs, as LK/ABL already has EL1 only and passing control to u-boot does't give it EL2 either.
Modify the firmware to enable EL2?Not possible because of SecureBoot.(Unlocking the BL doesn't effectively bypass this because the unlock only disables boot image(kernel) sign check in LK/ABL)The only effective way to disable platform secureboot is to replace the SoC with a brand new one with secureboot eFUSE not blown.

So simply give up Snapdragons if intend to run KVM.
But what about others? So far there seems only Qcom uses EL2 in their firmware.But almost all phones start kernel in EL1, even Exynos.

However, afaik on (some) Exynos devices their Secure Monitor (firmware runs in EL3) supports SMC calls that switches other ELs,so kernel in EL1 might "request" EL2 with a specific SMC on Exynos.

And Mediatek , Nvidia and Amlogic have a powerful bootrom exploit respectively.This means that we can take full control of the CPU/have EL3 and bypass secureboot effectively, and thus it is possible to patch its firmware/compile from (maybe leaked, there's a lot, especially for mtk) sources to run kernel in EL2.

@tarikkaya
Copy link

@tarikkaya tarikkaya commented Jan 23, 2020

thanks for your message. I will follow the matter as long as I use the phone. But as you know, while buying a mobile phone, nobody plans to change the operating system on it and purchases it. visuality is more in the foreground. I bought a lot of cell phones. when i change the phone. I generally saw these kinds of updates.

@WaseemAlkurdi
Copy link

@WaseemAlkurdi WaseemAlkurdi commented Jan 24, 2020

Dear @fxsheep,

However, afaik on (some) Exynos devices their Secure Monitor (firmware runs in EL3) supports SMC calls that switches other ELs,so kernel in EL1 might "request" EL2 with a specific SMC on Exynos.

Pure gold. Absolutely brilliant.
The Exynos is a great platform because it's quite powerful, head to head with Qualcomm in terms of pure performance. And there's quite a lot of Samsung phones out there.

@pmsjt
Copy link

@pmsjt pmsjt commented May 21, 2020

ARM64 VHDX for creating virtual machines or for booting from VHDX finally available for download!!

https://t.co/RI5FAXIceJ?amp=1

@WaseemAlkurdi
Copy link

@WaseemAlkurdi WaseemAlkurdi commented May 22, 2020

ARM64 VHDX for creating virtual machines or for booting from VHDX finally available for download!!

https://t.co/RI5FAXIceJ?amp=1

Nice!
What advantages does it offer over the current approach of booting from an ISO extracted to a VHD image though?

@pmsjt
Copy link

@pmsjt pmsjt commented May 22, 2020

ARM64 VHDX for creating virtual machines or for booting from VHDX finally available for download!!
https://t.co/RI5FAXIceJ?amp=1

Nice!
What advantages does it offer over the current approach of booting from an ISO extracted to a VHD image though?

They are the same, both in solution and in problem.

Just like VHDXs, there were no officially released ISOs. I am aware that a couple of ISOs were leaked before this change, but these were exception, not the rule. So far, images were only officially distributed with a machine. This was the policy that changed, so you can start expecting images to start being distributed the same way as they are for x86-32 and x86-64.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.