Could you make this for Galaxy A53?

[kyeon-go]

I'm using A536N, but latest A536U (A536USQU2AVG2) kernel is compatiable.
So I modified arch/arm64/include/asm/virt.h like this

static inline bool is_hyp_mode_available(void)
{
    /*
     * If KVM protected mode is initialized, all CPUs must have been booted
     * in EL2. Avoid checking __boot_cpu_mode as CPUs now come up in EL1.
     */
    if (IS_ENABLED(CONFIG_KVM) &&
        static_branch_likely(&kvm_protected_mode_initialized))
        return true;

    /*
    return (__boot_cpu_mode[0] == BOOT_CPU_MODE_EL2 &&
        __boot_cpu_mode[1] == BOOT_CPU_MODE_EL2);*/
    return 1;
}

Also I removed the string "kvm-arm.mode=protected" from CONFIG_CMDLINE
After compiled, kvm showed in /dev, but it doesn't work properly...

To compile this kernel, you need two things:
Clang 12 : https://android.googlesource.com/platform//prebuilts/clang/host/linux-x86/+archive/b669748458572622ed716407611633c5415da25c/clang-r416183b.tar.gz

Build-tools : https://android.googlesource.com/platform/prebuilts/build-tools/+archive/refs/heads/master.tar.gz

1. Make directory named 'compiler' in kernel main folder, then extract those in compilers folder.
2. Open build_kernel.sh, edit like this

#!/bin/bash

export PATH=$(pwd)/compiler/clang/bin:$PATH
export PATH=$(pwd)/compiler/build-tools/path/linux-x86:$(pwd)/compiler/gas:$PATH

make PLATFORM_VERSION=12 ANDROID_MAJOR_VERSION=s LLVM=1 LLVM_IAS=1 ARCH=arm64 TARGET_SOC=s5e8825 CROSS_COMPILE=aarch64-linux-gnu- (defconfig_name)
make PLATFORM_VERSION=12 ANDROID_MAJOR_VERSION=s LLVM=1 LLVM_IAS=1 ARCH=arm64 TARGET_SOC=s5e8825 CROSS_COMPILE=aarch64-linux-gnu- -j32

3. Open terminal, run build_kernel.sh !

[raspiduino]

Don't use termux, it's proot (you can understand that it's emulating fake device, not the real KVM)
Instead, use some chroot things, like Linux Deploy (go to Google Play and search for that)

Thanks and hope it works for you!

P/S: You open this issue right the time I'm porting KVM kernel for M20 :))

[kyeon-go]

Sorry, because Stock ROM can't mount /system as rw, BusyBox can't be installed...
(I have to use stock ROM, because one of banking apps isn't compatible with any AOSP roms)

Before someone made /system rw patch, I have to wait...
(Current version of SystemRW Patch doesn't recongnize f2fs filesystem)

By the way, I didn't expected that modifying just 2 files enables kvm
If A53 Possible, I think S21 exynos and S22 exynos also may possible...

[raspiduino]

Sorry, because Stock ROM can't mount /system as rw, BusyBox can't be installed...

I don't have busybox and I can still run Linux Deploy

The only thing you need is ROOT YOUR DEVICE with Magisk

[kyeon-go]

I installed Linux Deploy and ubuntu

but KVM still has error

[raspiduino]

Hmm, have you disabled UH (RKP, TIMA or sth like that)?

Normally uh.elf or vmm.elf will be loaded into EL2 instead of the KVM module.

I don't know how it possible for you to get /dev/kvm with just modify virt.h and cmdline (as you described), but it won't work without KVM being loaded to EL2.

You should take a look at sleirsgoevy's work.

Thanks!

[kyeon-go]

Ah! I refered to commit of this source.
Sorry... I forgot that 😂😂😂
CONFIG_UH is not set
CONFIG_RKP is not set
CONFIG_KDP is not set
CONFIG_KDP_CRED is not set
CONFIG_KDP_NS is not set
and something like CONFIG_FIVE also made disable, removed RKP, UH related things from source code.
Also SELinux is permissive!

There's no TIMA, but I think we need to modify KVM logic

To sum it up, I applied commit changes of above source to my source
then, modified virt.h and CONFIG_CMDLINE

P.S ) I tried to apply your KVM code to my source code, but there's no KVM related folder in /arch/arm.
So I think we need to completely different patch for A53, S22, S21, etc...

[kyeon-go]

I just looked into M20 source code. It seems to similar to A6 for KVM related files/folders.
For A53, S21, S22, structure of KVM related folders/files is different, instead similar to Pixel 6 kernel source. For example there's no arch/arm/kvm folder
So I tried to port pKVM from Pixel 6 source code, but failed.
(Compiler complains about something, like ID_AA64DFR0_BRPS)
Compiling method of those two kernels seems to different...

Sorry for broken english

[raspiduino]

I also got difficulties in porting KVM to my M20. The vmm_goto_EL2 now changed to uh_goto_EL2 but AFAIK the interface stays the same. But when I tried that (as you can see in my m20lte-kvm repo), it just stucks at bootloader.

I also got this situation trying to port KVM to a Note 8.

The only explaination for this I can think of is somehow SBoot (Samsung's bootloader) verify the EL2 code's signature before allowing that code to jump into EL2. Not sure if that is real, but I'm RE sboot.bin to find out.

If that really is the case, then things got much harder. We need to "patch" sboot (which is REALLY REALLY dangerous and it might brick your device)

Hopefully it's not

[raspiduino]

For example there's no arch/arm/kvm folder

Yes, KVM support for ARM32 has been removed since Linux 5.x (if I remember correctly, 5.16)

[kyeon-go]

Result of porting pKVM to my A53 is boot-loop
I copied kvm-related files from Pixel 6
(arch/arm64/kvm folder
kvm-*.h files from arch/arm64/include/asm and /arm64/include/uapi/asm)
then added defines/functions to sysreg.h, cacheflush.h

I got no errors when compiling, but result is boot-loop

(I think the thing that pixel uses S2MPU is the reason for boot-loop.)

Wow, this may much harder task...
I decided to study of force transition to EL2

(Pixel 6's tensor chip and exynos 2*00, 1280 are similar, but implementation of KVM doesn't seem to compatible each other)

[raspiduino]

I found this, and he actually setup QEMU for playing with Sboot

[raspiduino]

Btw for some reason my bootloader (sboot.bin) doesn't contain ERET (aka exception return) instruction, which AFAIK is needed for returning from exception handler.

So how tf it can lift UH to EL2? I can't think of a way to do that without ERET!

[kyeon-go]

Could you check ERET in my bootloader?
https://mega.nz/file/c010iaTa#x7TEan7bAUZ8paV92dWBLZOY87Xg3pQ5RFCR2eRe81s
It was extracted from BL firmware file

I don't have any assembly related knowledge...
Sorry for that..

[kyeon-go]

May dmesg help for this...?
https://mega.nz/file/Y1l3BbKY#4JpvYbQxNMSGf0VFt1AsdyyY_OKZGdDjhUhBjdVWQaI

[raspiduino]

Sorry for late reply, i was busy with school works :((

I was about to ask you about dmesg, but you had already send it! Thanks!

I will check it

[raspiduino]

Ok I understanded why it doesn't have ERET in my sboot.bin

That is because the SMC call is handled in the chip itself (hardware) through SiP SMC call

[kyeon-go]

May my phone similar to yours?
How 'bout dmesg?

[raspiduino]

May my phone similar to yours?

Probably. I'm still surprised that your phone could get /dev/kvm without calling SMC. Probably new implementation on new phone

Wait, send me /proc/last_kmsg when /dev/kvm got internal error. Thanks!

[kyeon-go]

last_kmsg in my phone has no difference to dmesg
I checked it several times, but I can't get like this case

last_kmsg : kernmess2.log

P.S ) I also disabled TUI and TZDEV but it still shows error...

[raspiduino]

Alright I found this

[40549.940517]I[1:      swapper/1:    0] Call trace:
[40549.940534]I[1:      swapper/1:    0]  dump_backtrace+0x0/0x1ec
[40549.940546]I[1:      swapper/1:    0]  show_stack+0x18/0x24
[40549.940559]I[1:      swapper/1:    0]  dump_stack_lvl+0xc0/0x138
[40549.940568]I[1:      swapper/1:    0]  dump_stack+0x18/0x24
[40549.940699]I[1:      swapper/1:    0]  dbg_snapshot_save_context+0x22c/0x290 [dss]
[40549.940823]I[1:      swapper/1:    0]  dbg_snapshot_ipi_stop+0x1c/0x2c [dss]
[40549.940841]I[1:      swapper/1:    0]  __traceiter_android_vh_ipi_stop+0x30/0x50
[40549.940856]I[1:      swapper/1:    0]  trace_android_vh_ipi_stop_rcuidle+0xb8/0x110
[40549.940870]I[1:      swapper/1:    0]  ipi_handler+0x124/0x130
[40549.940884]I[1:      swapper/1:    0]  handle_percpu_devid_fasteoi_ipi+0x78/0x1e8
[40549.940897]I[1:      swapper/1:    0]  __handle_domain_irq+0x7c/0xd4
[40549.940910]I[1:      swapper/1:    0]  gic_handle_irq+0x5c/0x14c
[40549.940924]I[1:      swapper/1:    0]  el1_irq+0xe4/0x1c0
[40549.940938]I[1:      swapper/1:    0]  cpuidle_enter_state+0x1bc/0x5c8
[40549.940950]I[1:      swapper/1:    0]  cpuidle_enter+0x38/0x50
[40549.940964]I[1:      swapper/1:    0]  do_idle.llvm.972246581917381849+0x1e8/0x2c4
[40549.940977]I[1:      swapper/1:    0]  cpu_startup_entry+0x24/0x28
[40549.940989]I[1:      swapper/1:    0]  secondary_start_kernel+0x1b0/0x1fc
[40549.941011]I[7:      swapper/7:    0] debug-snapshot dss: core register saved(CPU:7)
[40549.941043]I[7:      swapper/7:    0] DSU: NO Error: [NUM:0][ERXSTATUS_EL1:0x00000000000000]
[40549.941061]I[7:      swapper/7:    0]  L1: NO Error: [NUM:1][ERXSTATUS_EL1:0x00000000000000]
[40549.941074]I[7:      swapper/7:    0] debug-snapshot dss: context saved(CPU:7)

For some reason, kvm init successfully on CPU 0 (the startup CPU), but not on other CPU (which is said to copy the state of the first CPU)

Which surprise me most is that this kernel still run UH without the need of a SMC call to EL3. Maybe the kernel is already boot at EL2 by S-Boot? Can you restore the original function is_hyp_mode_available to check if it's put in EL2 by S-Boot?

I also find no reference to vmm.elf, uh.elf. There is also no vmm_goto_EL2 or uh_goto_EL2. Probably the uh code is intergrated in the kernel source with a different name, or it's located at some location in the memory.

OR maybe, maybe the UH is intergrated within EL3 itself (either it's in EL3, or it's dropped to EL2 by S-Boot). I only see "call" to UH, but to uh init code in kernel.

BTW I will look at your sboot to search for RKP strings.

Log message from S-Boot. It seems that UH is loaded from S-Boot directly. So I don't think we will be able to access EL2 unless some SMC hack is spotted.

It's possible for patching sboot.bin then flash it, but it is far more harm than good.

I think you should try something else, like porting EDK2 to your phone, then run port your OS (write drivers or sth) to it.
Poor you :((

[raspiduino]

WAIT
There is a partition called EL2?

Can you check your firmware to see if their is anything like uh.img.lz4?

[kyeon-go]

Ah, yes~ That file included in BL
uh.bin.lz4

I think that structure of KVM, Virtualization-related things hugely changed from year 2021

Also UH related partition shows in /dev/block

[raspiduino]

If we are lucky, we can patch uh, then upload it to uh partition.

So we patch uh to load kvm stubs from kernel, or even load the whole kernel to EL2, then boot the kernel.

But I'm not sure if S-Boot will continue to boot if uh partition has been changed (This will need further RE). If it refuse to boot, you might brick your phone unless you have access to phone's hardware and restore uh.

If S-Boot continues to boot even if we change uh, then it's good to go!

BTW, I tried reproducing the way uh_goto_EL2 works on my kernel, but failed to do so, always with memory access fault.
I think this might be the reason why they actually implement this SMC in silicon and no in S-Boot.

As you can see from the image, I tried changing SPSR_EL3 (and I even changed that register in a near ERET location), but still no luck. It failed at ISB in _uh_goto_EL2 after being returned with memory access fault.

I know it's hard, but maybe I can try patching uh to memcpy the kernel, then start kernel from EL2 again.

[raspiduino]

I think that structure of KVM, Virtualization-related things hugely changed from year 2021

Hmm, can I ask how do you know? Like you inspect every Samsung phone or sth? Or maybe you work at Samsung? (probably not, but I still want to ask) :)

[kyeon-go]

I don't work at Samsung, but I'm just interested on KVM, native speed VM on phone.

After I started to be interested in KVM, I inspected(?) S7, S20+ kernel which are my previous phone.

S7, S20+ kernel are seems to be similar to A6, S8
But I feeled S22, A53 kernel very differ from previous models...

P.S ) Sometimes I feel to sorry for that I don't have much programming knowledge...

[raspiduino]

I think I will work on this later, since I currently have exam at school, sorry for that :(

[kyeon-go]

That's OK!

[raspiduino]

Wait, can you check the max size of the uh partition?

[kyeon-go]

You mean.... this?

Welcome to fdisk (util-linux 2.31.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): p
Disk /dev/block/sda: 119.1 GiB, 127930466304 bytes, 31233024 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 524288 bytes / 524288 bytes
Disklabel type: gpt
Disk identifier: 52444E41-494F-2044-4D4D-43204449534B

Device             Start      End  Sectors   Size Type
/dev/block/sda1     1024     5119     4096    16M Microsoft basic data
/dev/block/sda2     5120     9215     4096    16M Microsoft basic data
/dev/block/sda3     9216    10239     1024     4M Microsoft basic data
/dev/block/sda4    10240    12287     2048     8M Microsoft basic data
/dev/block/sda5    12288    14335     2048     8M Microsoft basic data
/dev/block/sda6    14336    16383     2048     8M Microsoft basic data
/dev/block/sda7    16384    16511      128   512K Microsoft basic data
/dev/block/sda8    16512    26751    10240    40M Microsoft basic data
/dev/block/sda9    26752    27519      768     3M Microsoft basic data
/dev/block/sda10   27520    28031      512     2M Microsoft basic data
/dev/block/sda11   28032    29055     1024     4M Microsoft basic data
/dev/block/sda12   29056    31103     2048     8M Microsoft basic data
/dev/block/sda13   31104    31119       16    64K Microsoft basic data
/dev/block/sda14   31120    47503    16384    64M Microsoft basic data
/dev/block/sda15   47504    55695     8192    32M Microsoft basic data
/dev/block/sda16   55696    80271    24576    96M Microsoft basic data
/dev/block/sda17   80272    98191    17920    70M Microsoft basic data
/dev/block/sda18   98192   100111     1920   7.5M Microsoft basic data
/dev/block/sda19  100112   100367      256     1M Microsoft basic data
/dev/block/sda20  100368   100495      128   512K Microsoft basic data
/dev/block/sda21  100496   100751      256     1M Microsoft basic data
/dev/block/sda22  100752   114063    13312    52M Microsoft basic data
/dev/block/sda23  114064   118159     4096    16M Microsoft basic data
/dev/block/sda24  118160   118175       16    64K Microsoft basic data
/dev/block/sda25  118176   118191       16    64K Microsoft basic data
/dev/block/sda26  118192   126383     8192    32M Microsoft basic data
/dev/block/sda27  126384   126384        1     4K Microsoft basic data
/dev/block/sda28  126385   159152    32768   128M Microsoft basic data
/dev/block/sda29  159153   159231       79   316K Microsoft basic data
/dev/block/sda30  159232  3026431  2867200    11G Microsoft basic data
/dev/block/sda31 3026432  3384831   358400   1.4G Microsoft basic data
/dev/block/sda32 3384832  3392511     7680    30M Microsoft basic data
/dev/block/sda33 3392512  3546111   153600   600M Microsoft basic data
/dev/block/sda34 3546112  3558911    12800    50M Microsoft basic data
/dev/block/sda35 3558912  3571711    12800    50M Microsoft basic data
/dev/block/sda36 3571712 31231999 27660288 105.5G Microsoft basic data

Partition 14 ~ 29 does not start on physical sector boundary

[raspiduino]

Yes
Thank you!

So we got about 2 MB limit of uh code.

Btw I just figure out that uh.bin is loaded as a plugin to S-Boot, which means at first it got access to EL3 level, then it use SMC call (again, to hardware handler), then it will be taken to EL2

And wtf this uh is written in Rust

[kyeon-go]

LOL... Written in Rust... which made me surprise also...
Is there any taken-to-EL1-related code?

[raspiduino]

Is there any taken-to-EL1-related code?

Again, there is no ERET, but I found just one SMC call, so probably that is how they return to EL1 (using silicon implementation)

I looked at your phone's kernel source and found that they still use HVC to call to EL2, so RKP must somehow thrown from EL3 to EL2

[kyeon-go]

I updated to Android 13, and applied custom kernel (based on opensource's android 13 kernel)

I edited virt.h, and removed UH, RKP, "kvm.mode=protected", ... from config file. /dev/kvm still shows!

P.S ) System partition is now RW!

[raspiduino]

Wow, but does it work?

[kyeon-go]

Nope...

[kyeon-go]

But this is more better than my previous phone, S20 (Snapdragon)
because current phone can show /dev/kvm at least

--I'll buy latest Pixel series as my next or second phone--

[raspiduino]

But this is more better than my previous phone, S20 (Snapdragon) because current phone can show /dev/kvm at least

--I'll buy latest Pixel series as my next or second phone--

Snapdragon SoC cannot be exploited to get KVM

[Marietto2008]

Can I exploit my tablet samsung nexus 10 to get KVM enabled ?

[raspiduino]

It depends on your luck

Can I exploit my tablet samsung nexus 10 to get KVM enabled ?

[Marietto2008]

I could try,which patch can I use ? which method ? your is ok ?

[raspiduino]

I could try,which patch can I use ? which method ? your is ok ?

I'm not sure. My appoarch works for some people but not for others.

You might want to follow sleirsgoevy's patches instead. He also explains the working principle of this.

[raspiduino]

@kyeon-go they use the same method of calling SMC, but with different number

load_plugin is just

[Marietto2008]

Hello to everyone,

it's again me. I would like to know if,in your opinion,I can apply your patch to enable kvm on the old Samsung / Google / Chromebook based on arm32,model xe303c12 based on Exynos 5250,this one :

https://github.com/quarkscript2/xe303c12_arm_linux

I ask it because I'm working on the project to enable kvm on a 5.4 (main line) kernel since 1 year or so and I haven't still be able to do that. Really it has been already explained how to do that,for example on this tutorial :

http://www.virtualopensystems.com/en/solutions/guides/kvm-on-chromebook/#setting-up-the-boot-medium

but they used a very old kernel version,3.13 ! I want to use a newer kernel version,the last version which support kvm on arm32,that should be 5.4 and something. It makes no sense to use kernel 3.13 on a recent version of ubuntu,even because it does not work. kernel 3.13 is too old if a modern userland is used. And the patch proposed by virtual open system is the only one I know. Im forced to use it if I don't find another method. And that patch forces me to use their old and custom u-boot. Your patch if worked could be the game changer.

[raspiduino]

I cannot see the patch, all i see is that they give us a repo pre-patched, with long commit history that i can't find which one contains (or related to) the patch, since that history also contains pulled mainstream commits.

So I think you should clone their repo (https://github.com/virtualopensystems/linux-kvm-arm), diff with the original linux source (of ver 3.14, which they currently use) to see what exactly changed.

Then you can pull the latest kernel source and try to apply the same mechanism on it.

Good luck

[Marietto2008]

I'm not able to find the link to downlad the main line official kernel vers 3.14-rc2. Can you point me there ? thanks. Anyway I think that your plan can't work because I'm sure that the modifications I found can't be applicable to a such new kernel version like the 5.14.

[Marietto2008]

I have old kernels on kernel.org website.

https://mirrors.edge.kernel.org/pub/linux/kernel/v3.x/

I can's see RC kernels, there is only 3.13, 3.14, 3.14.1, etc.

So,your patch can't be applicable in any way ?

[kyeon-go]

@raspiduino So, is it okay if I change value of VMM(UH)_64BIT_CALL_SMC_MAGIC to 0xC2000480?

[raspiduino]

@raspiduino So, is it okay if I change value of VMM(UH)_64BIT_CALL_SMC_MAGIC to 0xC2000480?

Probably we have to remove things in the UH partition first (to not load Samsung Knox)

WAIT: the magic code 0xC2000480 you see in the picture is from another phone's firmware (Galaxy A57), so I might need to recheck if the your phone's magic code is the same