This repository has been archived by the owner. It is now read-only.

Device qemu #56

Closed
wants to merge 12 commits into
base: master
from

Conversation

Projects
None yet
5 participants
@mmaret
Contributor

mmaret commented Jun 4, 2017

So it will be easier to try the project and/or develop userspace.

Status:

  • build a img that you can mount (with kpartx) and use the binaries on the boot partitions to start qemu (qemu-system-arm -M vexpress-a9 -kernel vmlinuz-qemu-armhf -dtb vexpress-v2p-ca9.dtb -initrd initramfs-qemu-armhf)

  • This need a bootloader to give to qemu to look for the kernel, dtb, initramfs on the boot partition

  • Mounting rootfs

  • More testing ...

@lawl

This comment has been minimized.

Show comment
Hide comment
@lawl

lawl Jun 4, 2017

Contributor

Wow, that's some cool stuff, one issue I noticed. If you disable LUKS//dm-crypt, the current initrd will not find the root partition anymore, because it simply looks for the first partition with a LUKS header.

Not that that's necessarily the best thing to do, but that's currently how it works, see: https://github.com/postmarketOS/pmbootstrap/blob/master/aports/postmarketos-mkinitfs/init_functions.sh#L23

So I believe there would need to be some more changes.

But man, testing non device specific userspace stuff will be NICE.

I think this might actually deserve a pmbootstrap command that instantly spawns the instance for you.

Contributor

lawl commented Jun 4, 2017

Wow, that's some cool stuff, one issue I noticed. If you disable LUKS//dm-crypt, the current initrd will not find the root partition anymore, because it simply looks for the first partition with a LUKS header.

Not that that's necessarily the best thing to do, but that's currently how it works, see: https://github.com/postmarketOS/pmbootstrap/blob/master/aports/postmarketos-mkinitfs/init_functions.sh#L23

So I believe there would need to be some more changes.

But man, testing non device specific userspace stuff will be NICE.

I think this might actually deserve a pmbootstrap command that instantly spawns the instance for you.

@mmaret

This comment has been minimized.

Show comment
Hide comment
@mmaret

mmaret Jun 4, 2017

Contributor

You are right, in the final PR, the LUKS stuff should not be present.
I've use it as a convenient way to inspect rootfs but I expect to get the kind of issue you are describing :)

Contributor

mmaret commented Jun 4, 2017

You are right, in the final PR, the LUKS stuff should not be present.
I've use it as a convenient way to inspect rootfs but I expect to get the kind of issue you are describing :)

@ollieparanoid

This comment has been minimized.

Show comment
Hide comment
@ollieparanoid

ollieparanoid Jun 4, 2017

Member

Cool idea, this could help a lot with debugging!
A few change requests, in addition to the luks functionallity:

  • the mainline kernel is already packaged in Alpine Linux (linux-vanilla), so I'd suggest using that one :)
  • keyboard=false in the deviceinfo <- well, most of our computers do have a keyboard, so I'd change that.
  • the xwayland workarounds copied from lg-mako should not be necessary. these demos only get started, because the postmarketos-demos menu doesn't work there yet

Thank you for this PR 👍

Member

ollieparanoid commented Jun 4, 2017

Cool idea, this could help a lot with debugging!
A few change requests, in addition to the luks functionallity:

  • the mainline kernel is already packaged in Alpine Linux (linux-vanilla), so I'd suggest using that one :)
  • keyboard=false in the deviceinfo <- well, most of our computers do have a keyboard, so I'd change that.
  • the xwayland workarounds copied from lg-mako should not be necessary. these demos only get started, because the postmarketos-demos menu doesn't work there yet

Thank you for this PR 👍

@mmaret

This comment has been minimized.

Show comment
Hide comment
@mmaret

mmaret Jun 4, 2017

Contributor

Keyboard and wayland workaround fixed.

About using linux-vanilla, I'm not sure... I'm not an expert here, but I think that even if the goal is to support all device with the same kernel, it will be a bit hard to have a kernel config that work for all of them. I'm not sure it's even cool to have the same "big" kernel for all of them with all dtb in the same .apk.
I thinks that's partially why there is a linux-rpi and linux-rpi2 package in your link.
Anyway, It's easy this way for the dev, and as I got some hard time with uboot and the BS this PR will not be ready soon, so we have plenty of time for the discussion about that :)

Contributor

mmaret commented Jun 4, 2017

Keyboard and wayland workaround fixed.

About using linux-vanilla, I'm not sure... I'm not an expert here, but I think that even if the goal is to support all device with the same kernel, it will be a bit hard to have a kernel config that work for all of them. I'm not sure it's even cool to have the same "big" kernel for all of them with all dtb in the same .apk.
I thinks that's partially why there is a linux-rpi and linux-rpi2 package in your link.
Anyway, It's easy this way for the dev, and as I got some hard time with uboot and the BS this PR will not be ready soon, so we have plenty of time for the discussion about that :)

@ollieparanoid

This comment has been minimized.

Show comment
Hide comment
@ollieparanoid

ollieparanoid Jun 4, 2017

Member

The "big" kernel is what I'd want to have. Probably a (automatically generated from linux-vanilla) linux-postmarketos package in the long term, which is mainline linux + patches for different devices, that are not upstream yet + different config options enabled compared to the upstream Alpine Linux linux-vanilla package.

linux-vanilla also has a lot of dtb files, they don't seem to be that big:
https://pkgs.alpinelinux.org/contents?branch=edge&name=linux-vanilla&arch=armhf&repo=main

If that becomes a problem at one point, we could group them together and only install the relevant group.

Finally, I think the raspberry pi has an extra package, because it doesn't use the mainline kernel (don't they have a boot.txt file and some other hacks, like displaying on screen when the voltage is low etc?).

Member

ollieparanoid commented Jun 4, 2017

The "big" kernel is what I'd want to have. Probably a (automatically generated from linux-vanilla) linux-postmarketos package in the long term, which is mainline linux + patches for different devices, that are not upstream yet + different config options enabled compared to the upstream Alpine Linux linux-vanilla package.

linux-vanilla also has a lot of dtb files, they don't seem to be that big:
https://pkgs.alpinelinux.org/contents?branch=edge&name=linux-vanilla&arch=armhf&repo=main

If that becomes a problem at one point, we could group them together and only install the relevant group.

Finally, I think the raspberry pi has an extra package, because it doesn't use the mainline kernel (don't they have a boot.txt file and some other hacks, like displaying on screen when the voltage is low etc?).

@MartijnBraam

This comment has been minimized.

Show comment
Hide comment
@MartijnBraam

MartijnBraam Jun 6, 2017

Member

Raspberry pi does use a mainline kernel, but it also has a bunch of DTB overlays that you can enable for hardware extensions and a bunch of firmware from the broadcom team.

Member

MartijnBraam commented Jun 6, 2017

Raspberry pi does use a mainline kernel, but it also has a bunch of DTB overlays that you can enable for hardware extensions and a bunch of firmware from the broadcom team.

@MartijnBraam

This comment has been minimized.

Show comment
Hide comment
@MartijnBraam

MartijnBraam Jun 9, 2017

Member

After some research I found this: https://wiki.alpinelinux.org/wiki/Alpine_on_ARM#Using_QEMU

Currently the Alpine's kernel does not boot with Qemu

Member

MartijnBraam commented Jun 9, 2017

After some research I found this: https://wiki.alpinelinux.org/wiki/Alpine_on_ARM#Using_QEMU

Currently the Alpine's kernel does not boot with Qemu

@mmaret

This comment has been minimized.

Show comment
Hide comment
@mmaret

mmaret Jun 9, 2017

Contributor

Just to keep you updated on what I'm stuck.
I'm still using the kernel from this PR and I can boot it with the associated initramfs. But the main rootfs cannot be detected.
It does not look like a huge problem but I got a lot of issue with the buildsystem to investigate this.
I've few hack to have network on the qemu.
I've build a uboot it order to simplify the boot:

  • currently, to run qemu, you need to mount the 1st partition of the image image (to get the kernel, initramfs, dtb) and launch this kind of command : qemu-system-arm -M vexpress-a9 -kernel zImage -initrd initramfs-qemu-armhf -dtb vexpress-v2p-ca9.dtb -hda qemu-armhf.img
  • having a prebuild uboot will remove the need of mounting the 1st partition of disk image. Uboot will look for kernel with a bit of builtin configuration
    qemu-system-arm -M vexpress-a9 -kernel uboot -sd qemu-armhf.img
Contributor

mmaret commented Jun 9, 2017

Just to keep you updated on what I'm stuck.
I'm still using the kernel from this PR and I can boot it with the associated initramfs. But the main rootfs cannot be detected.
It does not look like a huge problem but I got a lot of issue with the buildsystem to investigate this.
I've few hack to have network on the qemu.
I've build a uboot it order to simplify the boot:

  • currently, to run qemu, you need to mount the 1st partition of the image image (to get the kernel, initramfs, dtb) and launch this kind of command : qemu-system-arm -M vexpress-a9 -kernel zImage -initrd initramfs-qemu-armhf -dtb vexpress-v2p-ca9.dtb -hda qemu-armhf.img
  • having a prebuild uboot will remove the need of mounting the 1st partition of disk image. Uboot will look for kernel with a bit of builtin configuration
    qemu-system-arm -M vexpress-a9 -kernel uboot -sd qemu-armhf.img
@ollieparanoid

This comment has been minimized.

Show comment
Hide comment
@ollieparanoid

ollieparanoid Jun 9, 2017

Member

@MartijnBraam: good find, but that info is from "16:40, 26 November 2015‎". Judging from what @mmaret tells us, it is outdated. I will update it in the Alpine wiki.

@mmaret: glad to see, that you're still working on this. Regarding the root partition: @lawl has explained it above - it must be encrypted with cryptsetup, otherwise the init script will not find it (we may change this in the future to a detection by partition label or something similar - this was just an easy hack to get it working fast on most devices).

What issues do you have with the build system? I'm improving the initramfs situation, as mentioned in #69 already (big improvements incoming, stay tuned :) ) - is there more?

You've said, that it works best with a pre-built u-boot. How about packaging that pre-built u-boot in the pmbootstrap/aports folder? You could start with Alpine's u-boot package (source), rename it, put it in pmbootstrap/aports and adjust it to your needs.

Member

ollieparanoid commented Jun 9, 2017

@MartijnBraam: good find, but that info is from "16:40, 26 November 2015‎". Judging from what @mmaret tells us, it is outdated. I will update it in the Alpine wiki.

@mmaret: glad to see, that you're still working on this. Regarding the root partition: @lawl has explained it above - it must be encrypted with cryptsetup, otherwise the init script will not find it (we may change this in the future to a detection by partition label or something similar - this was just an easy hack to get it working fast on most devices).

What issues do you have with the build system? I'm improving the initramfs situation, as mentioned in #69 already (big improvements incoming, stay tuned :) ) - is there more?

You've said, that it works best with a pre-built u-boot. How about packaging that pre-built u-boot in the pmbootstrap/aports folder? You could start with Alpine's u-boot package (source), rename it, put it in pmbootstrap/aports and adjust it to your needs.

@ollieparanoid

This comment has been minimized.

Show comment
Hide comment
@ollieparanoid

ollieparanoid Jun 9, 2017

Member

BTW, for other devices, we have merged them early during their WIP phase into an extra "device-vendor-name" branch. I would like to do that with your qemu "device" too, if you don't mind.

Member

ollieparanoid commented Jun 9, 2017

BTW, for other devices, we have merged them early during their WIP phase into an extra "device-vendor-name" branch. I would like to do that with your qemu "device" too, if you don't mind.

@mmaret

This comment has been minimized.

Show comment
Hide comment
@mmaret

mmaret Jun 10, 2017

Contributor

Sorry that it advance slowly but I've got few spare time...
About finding the main partition, I only add the option to disable LUKS, it was not using it :)

Of course, I've been using alpine package as a start for uboot and I push my commit about it. It still miss a patch to change default environment.

You can merge the branch in master but we will not be able to change the git history. For example, I just remove the LUKS options from the commits, rebase the branch on master ...
But it is as you want !

Contributor

mmaret commented Jun 10, 2017

Sorry that it advance slowly but I've got few spare time...
About finding the main partition, I only add the option to disable LUKS, it was not using it :)

Of course, I've been using alpine package as a start for uboot and I push my commit about it. It still miss a patch to change default environment.

You can merge the branch in master but we will not be able to change the git history. For example, I just remove the LUKS options from the commits, rebase the branch on master ...
But it is as you want !

@ollieparanoid

This comment has been minimized.

Show comment
Hide comment
@ollieparanoid

ollieparanoid Jun 11, 2017

Member

Sorry that it advance slowly but I've got few spare time...

That's perfectly fine, no worries ;)

You can merge the branch in master but we will not be able to change the git history. For example, I just remove the LUKS options from the commits, rebase the branch on master ...

I didn't mean to merge it to master, but to a new branch. But if you are unsure, I'll keep it around as PR for now.

You have mentioned issues in the build system - is there anything we/I can improve?

Member

ollieparanoid commented Jun 11, 2017

Sorry that it advance slowly but I've got few spare time...

That's perfectly fine, no worries ;)

You can merge the branch in master but we will not be able to change the git history. For example, I just remove the LUKS options from the commits, rebase the branch on master ...

I didn't mean to merge it to master, but to a new branch. But if you are unsure, I'll keep it around as PR for now.

You have mentioned issues in the build system - is there anything we/I can improve?

@ollieparanoid

This comment has been minimized.

Show comment
Hide comment
@ollieparanoid

ollieparanoid Jun 12, 2017

Member

I was reading through the init scripts while fixing #76 and noted, that it expects the root partition to be in /dev/mmcblkp* or /dev/mapper/*: init_functions.sh.

When you have time again to work on this, you could try to find out the location of the root partition (maybe by adjusting init.sh and dropping to a shell before the first splash? simply add sh in the line above) - then we should be able to adjust the init_functions.sh script.

Member

ollieparanoid commented Jun 12, 2017

I was reading through the init scripts while fixing #76 and noted, that it expects the root partition to be in /dev/mmcblkp* or /dev/mapper/*: init_functions.sh.

When you have time again to work on this, you could try to find out the location of the root partition (maybe by adjusting init.sh and dropping to a shell before the first splash? simply add sh in the line above) - then we should be able to adjust the init_functions.sh script.

@mmaret

This comment has been minimized.

Show comment
Hide comment
@mmaret

mmaret Jun 13, 2017

Contributor

This have advance. After compiling qemu-armhf target, you can now run qemu with this cmd:
sudo qemu-system-arm -M vexpress-a9 -m 1G -kernel PMB_SRC/aports/device-qemu-armhf/u-boot -sd PMB_BUILD_PATH/chroot_native/home/user/rootfs/qemu-armhf.img

It stop to the point where it ask for you password and, sadly, decryption fail. This still have to be investigated.

Contributor

mmaret commented Jun 13, 2017

This have advance. After compiling qemu-armhf target, you can now run qemu with this cmd:
sudo qemu-system-arm -M vexpress-a9 -m 1G -kernel PMB_SRC/aports/device-qemu-armhf/u-boot -sd PMB_BUILD_PATH/chroot_native/home/user/rootfs/qemu-armhf.img

It stop to the point where it ask for you password and, sadly, decryption fail. This still have to be investigated.

@mmaret mmaret changed the title from WIP: Device qemu to Device qemu Jun 14, 2017

@mmaret

This comment has been minimized.

Show comment
Hide comment
@mmaret

mmaret Jun 14, 2017

Contributor

There is still things to improve (like network, boot time ...) but it's now usable

Contributor

mmaret commented Jun 14, 2017

There is still things to improve (like network, boot time ...) but it's now usable

@ollieparanoid

Thank you very much for your great work! There are only a few things I'd like to see changed, could you have a look at them?

Show outdated Hide outdated aports/device-qemu-armhf/deviceinfo
@@ -0,0 +1,24 @@
pkgname=device-qemu-armhf

This comment has been minimized.

@ollieparanoid

ollieparanoid Jun 14, 2017

Member

Please remove the uboot blob from the device-qemu-armhf aport folder. It's great, that you're building uboot from source instead!

@ollieparanoid

ollieparanoid Jun 14, 2017

Member

Please remove the uboot blob from the device-qemu-armhf aport folder. It's great, that you're building uboot from source instead!

This comment has been minimized.

@mmaret

mmaret Jun 15, 2017

Contributor

The build uboot will be in .apk, and that's not convenient for running qemu with this. You will first have to manually extract the file.
Any idea on this?

@mmaret

mmaret Jun 15, 2017

Contributor

The build uboot will be in .apk, and that's not convenient for running qemu with this. You will first have to manually extract the file.
Any idea on this?

This comment has been minimized.

@ollieparanoid

ollieparanoid Jun 15, 2017

Member

Alpine's .apk format is basically a .tar.gz file, that can be extracted with tar -xf uboot.apk.

I suggest listing your QEMU port as device (just like the other devices) in the wiki, and creating a wiki page with detailed instructions how to actually start qemu with the port (pmbootstrap init, select the qemu device, pmbootstrap install, extract the uboot apk, run the qemu launch command?).
That page would then contain the extraction step with tar (the apk file is in ~/.local/var/pmbootstrap/packages/armhf after it has been built).

At a later point, we could automatize this with pmbootstrap (something like: pmbootstrap qemu --arch=...), just like @lawl proposed and adjust the wiki page accordingly.

@ollieparanoid

ollieparanoid Jun 15, 2017

Member

Alpine's .apk format is basically a .tar.gz file, that can be extracted with tar -xf uboot.apk.

I suggest listing your QEMU port as device (just like the other devices) in the wiki, and creating a wiki page with detailed instructions how to actually start qemu with the port (pmbootstrap init, select the qemu device, pmbootstrap install, extract the uboot apk, run the qemu launch command?).
That page would then contain the extraction step with tar (the apk file is in ~/.local/var/pmbootstrap/packages/armhf after it has been built).

At a later point, we could automatize this with pmbootstrap (something like: pmbootstrap qemu --arch=...), just like @lawl proposed and adjust the wiki page accordingly.

@@ -0,0 +1,101 @@
# APKBUILD based on linux-vanilla aport. Changes:

This comment has been minimized.

@ollieparanoid

ollieparanoid Jun 14, 2017

Member

Can we use the linux-vanilla kernel from Alpine instead? I'd really prefer this, because the goal is to have as few (ideally only one) kernel as possible, and Alpine automatically updates its linux-vanilla kernel.

@ollieparanoid

ollieparanoid Jun 14, 2017

Member

Can we use the linux-vanilla kernel from Alpine instead? I'd really prefer this, because the goal is to have as few (ideally only one) kernel as possible, and Alpine automatically updates its linux-vanilla kernel.

This comment has been minimized.

@mmaret

mmaret Jun 15, 2017

Contributor

As @MartijnBraam state before, this kernel is not supposed to work on qemu, so I did not really try it.
The vanilla-kernel come will a lot of feature build as modules. Is there any way to embedded them inside the initfs (particularly the crypto modules)?
The vanilla kernel install the dtb in the /usr/lib/linux-KERNVER/ folder instead of /boot, so we will need to either adapt the APKBUILD or change the buildsystem.
A last little point, the current vanilla-kernel is 4.9.31, last kernel is 4.11.5.

@mmaret

mmaret Jun 15, 2017

Contributor

As @MartijnBraam state before, this kernel is not supposed to work on qemu, so I did not really try it.
The vanilla-kernel come will a lot of feature build as modules. Is there any way to embedded them inside the initfs (particularly the crypto modules)?
The vanilla kernel install the dtb in the /usr/lib/linux-KERNVER/ folder instead of /boot, so we will need to either adapt the APKBUILD or change the buildsystem.
A last little point, the current vanilla-kernel is 4.9.31, last kernel is 4.11.5.

This comment has been minimized.

@ollieparanoid

ollieparanoid Jun 15, 2017

Member
  • That info was from November 2015, I think would be worth trying it again.
  • You can specify modules in your deviceinfo, they will then get added to the initramfs. Crypto and ext4 modules are getting added by default.
  • When you specify the dtb in your deviceinfo, it will get appended by postmarketos-mkinitfs to the vmlinuz image, so that should work too.
  • You are right, Alpine's kernel isn't as new as your kernel. I think we really need to have the latest kernel for phones, where we want to use mainline, because right now we need more compatibility (which should be given in newer kernels) than stability.

With all that being said, I think it is good enough for now if we keep the extra kernel for qemu right now, and make a detailed plan for mainlining kernels. I think, that we will need a linux-postmarketos kernel package, which is really based on the latest kernel, and enables all the device specific options we need for postmarketOS (which are certainly not enabled in Alpine, and I'm not sure if they would even want to have them). I will create an extra ticket for that.

@ollieparanoid

ollieparanoid Jun 15, 2017

Member
  • That info was from November 2015, I think would be worth trying it again.
  • You can specify modules in your deviceinfo, they will then get added to the initramfs. Crypto and ext4 modules are getting added by default.
  • When you specify the dtb in your deviceinfo, it will get appended by postmarketos-mkinitfs to the vmlinuz image, so that should work too.
  • You are right, Alpine's kernel isn't as new as your kernel. I think we really need to have the latest kernel for phones, where we want to use mainline, because right now we need more compatibility (which should be given in newer kernels) than stability.

With all that being said, I think it is good enough for now if we keep the extra kernel for qemu right now, and make a detailed plan for mainlining kernels. I think, that we will need a linux-postmarketos kernel package, which is really based on the latest kernel, and enables all the device specific options we need for postmarketOS (which are certainly not enabled in Alpine, and I'm not sure if they would even want to have them). I will create an extra ticket for that.

@ollieparanoid

This comment has been minimized.

Show comment
Hide comment
@ollieparanoid

ollieparanoid Jun 15, 2017

Member

Thanks for cleaning up the deviceinfo! Could you remove the uboot blob, and create the device specific page with usage instructions (as commented above)? I'd be happy to merge it then.

(Of course this is still open for discussion if you don't agree with that.)

Member

ollieparanoid commented Jun 15, 2017

Thanks for cleaning up the deviceinfo! Could you remove the uboot blob, and create the device specific page with usage instructions (as commented above)? I'd be happy to merge it then.

(Of course this is still open for discussion if you don't agree with that.)

@ollieparanoid ollieparanoid referenced this pull request Jun 15, 2017

Closed

"The Mainline Kernel" #91

5 of 5 tasks complete
@ollieparanoid

This comment has been minimized.

Show comment
Hide comment
@ollieparanoid

ollieparanoid Jul 24, 2017

Member

Thank you again for all your work on this, and especially for having this great idea.
@MartijnBraam has taken a new shot on implementing this (together with a mainline kernel) in #228
Looks like we'll have pmOS running in qemu soon 👍

Member

ollieparanoid commented Jul 24, 2017

Thank you again for all your work on this, and especially for having this great idea.
@MartijnBraam has taken a new shot on implementing this (together with a mainline kernel) in #228
Looks like we'll have pmOS running in qemu soon 👍

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.