Prototype ARM64 EFI build (also Ten64) (#7) #8
Conversation
This adds support for ARM64 Embedded & Server Boot systems which use grub2 and EFI to boot. As OpenSuSE also embeds a copy of U-Boot for certain targets it may be possible to use this for the Raspberry Pi 3 and 4 with slight adjustments. This image also include drivers for the Traverse Ten64 board.
|
@mcbridematt Thanks for this much anticipated pull request. Much appreciated. All looks to be good but I'm currently just modifying our back-end buildbot image builder / up-loader mechanism to account for the unexpected profile name and the image type change. I'd originally build this mechanism assuming the "Ten64" machine type profile element would be used and that it's image type would be raw, as per for the Pi4 profile; which we then, via a contextual buildbot step compress via: to gain the more manageable/transport-friendly and generally expected (in Pi land) *.raw.xz But given your qcow2 preference here I'm just having to tricolate (tweak) the buildbot mechanism I've got in place prior to being able to test our build/upload mechanisms with this pr pre-applied prior to actually merging. So just to let you know I'm currently 'working on it'. And will be able to test this pr using our normal systems shortly there after. Thanks again and great to finally have this profile in place. And as you say more generaly relevant than specifically for the Ten64 which is great. And again as you say, in time we can hopefully remove the Ten64 platform specific packages currently required that may worry some folks after a 'vanilla' arm64 EFI image. But apparently this is just a matter of awaiting the appropriate drivers being up-streamed. Sorry for the delay and I'll update on my related progress here in due course. |
|
@mcbridematt Our existing back-end compresses the end raw image (for the Pi4) and the interim raw ARM64EFI image as per this pr thus: The resulting xz file can then be expanded using 4 threads is my understanding. Anyway, by way of a test for generic images such as your proposed ARM64EFI profile we transfer the raw image to a pre-prepared KVW raw drive image created via: and we then transit the minimally sized raw image, via dd, to the much larger raw disk via: See: https://osinside.github.io/kiwi/building_images/build_oem_disk.html In this scenario your profile works a treat and we get a self expanding image with around 10% btrfs usage on the 'ROOT' pool: Where as when instantiating a KVM instance from the intended qcow2 image we see only the minimum size and no auto expansion: Much as we would see if we instantiated a KVM instance from the original raw image in fact. I believe this is due to qcow2 incorporating a disk image size component: And from looking at vanilla openSUSE Leap 15.2 qcow2 images they look to be set to 24GB So I propose that for this profile we add a size element to set our overall disk size to be a minimum of 16GB via say: from: https://osinside.github.io/kiwi/building_images/build_vmx_disk.html#modifying-the-size-of-the-image Although 20GB or even 24 GB might be nice. But I'm unsure of your requirements here and you may have a need to trim this down. It's a shame we don't have auto-expansion such as we see in the raw and iso oem variants but I'm planning on also distributing the raw.xz of this profile as well given it does auto expand to it's 'discovered' device size. So in brief, I think that the qcow2 target image, given it's implicit size specification, presents a difficulty with regard to self expanding images. And our work around may be to simply specify our minimum desired size (16-20-24GB as above). I've not tried adding the suggested size element here as it's now late but given I've seen no auto expansion at all, even after: where the resulting image is then: Our installed instance, once booted again, shows the expected larger disk size but no auto expansion of the btrfs partition: And I've also tried kicking off our one shot systemd elements again but no joy. So do you fancy adding the size element, fit for our purposes (I'm unsure of your Ten64 limitations here) or do we move/resort to the raw format, which I will be distributing in xz format anyway to keep the size down. But with my existing back-end changes it looks like we can have both for a single build. And with target qcow2 image size change we may be sorted, though without auto expansion. Just one more request for a change here: Given, if I understand correctly, the mcbridematt repo is concerned solely with Ten64 drivers. And that we are expecting in time for those drivers to be up-streamed, making this repo redundant. How do you feel about changing this repo Alias/Name to be say Ten64 as at least then, when folks list it, they can look that up and see it's relevance. I think I'd rather see that myself and even better "Ten64_drivers" say. Saves worrying folks with what is otherwise a nondescript repo label. Thanks again for submitting this and I think if we get the size element in and change the repo name we can merge this and have our 'first draft' we can always make changes as we go and folks wont have imminent failure on via no disk space after just a few package updates. As you see we default to enabling boot to snap irrespective of 'ROOT' pool size and so we have keep an eye on the deployed size, and we are likely to increase our minimum system requirements to say 16 GB min 32 GB preferred once we have our new installer published. So if you could add the size element and the repo name change and test, I can test here via our buildbot also and hopefully we can merge this and move one. And please let me know if I've miss-understood your use/requirement of the qcwo2 format here with maybe an example of how I might test the resulting qcow2 to KVM image method, if that is the equivalent of what you are doing, as we need to ensure we do get image expansion if it's expected to work to test against regressions as we have seen that 'feature' fail in earlier kiwi-ng versions and configs of ours. I've only ever seen qcow2 used to boot from directly, but with initial adequate space set as per the JeOS example above. |
|
@mcbridematt An example of the JeOS qcow2 format begin accompanied by a size element is available in the kiwi template for Leap 15.2 JeOS here: However the openstack cloud qcow2 variant doesn't, but I think they use cloud-init to do their post install expansion and use only a single XFS partition: |
|
Yes, I haven't quite worked out why autoexpanding doesn't always happen, I think something must be regenerating the initramfs (dracut) on first boot. I think 4GB is too small given the image comes out as 2GB swap + 2GB btrfs, 8GB might be a good minimum, especially if you do most of your storage on a separate disk drive to the Rockstor install itself. My 'two cents' on the minimum image size is that should be that you have enough space to run a OS/zypper update some point after install.. on many of the cloud images from distributions I've seen - the default image size will often run out of space when you do this. qcow2 is preferable for certain situations:
Similarly for muvirt, images are downloaded to tmpfs (/tmp) and written to their final destination with qemu-img convert.
Since the .raw is built by kiwi-ng anyway, I see no reason why that should not be distributed as well, it's a bit like .raw vs .vmdk/.vhd for VMware/HyperV. As I indicated via email I will likely transfer across all Ten64 related packages to a 'traverse' user on build.opensuse.org soon and will update the pull request accordingly. |
|
@mcbridematt Unfortunately we've had some other required changes with regard to repo persistence. I think we just need to add the discussed size line for this qcow2 profile only and rebase this pr on master and we can pop this one in. Let me know if your happy to do these. As per previously discussion I think the min size for the qcow should be 16G, you didn't indicate that this would be a problem so I think we are safe on that front hopefully. That way if folk use the qcow2 image directly (highly likely given that is their more common use) we don't have them fail on the first update. Nice utility by the way and thanks for the pointers/schooling re qcow2 as transport qemu-nbd trick. I'll try and incorporate such a test for this image within our buildbot backend. |
|
@mcbridematt No worries on the rebase and size line, the merge was fairly simple and I've added the required size to this profile while I was there. I'll do a final test of what we have thus far and if all is well, or well enough, I'll merge as-is to avoid further potential merge conflicts. We can always do follow up pull requests as needed or as we find stuff. |
|
@mcbridematt I'm going to merge this as-is with the caveat that it's a preliminary profile. But then aren't they all currently given the repos recent release. This way it's easier for folks to contribute further to this profile once it's merged. The resulting qcow2 image boots in a Pi4 hosted KVM instance with the expected 16 GB drive size in evidence: The ROOT pool shows 9.1% usage. And given the other outstanding issues in this repo, subscribes to testing and updates thereafter successfully to the 4.0.1-0 rpm. Other pending upstream updates are also install-able via the Web-UI and the consequent install is able to reboot successfully. There is an outstanding issue/discrepancy in that the added profile resources an x86_64 specified repo, the rockstor:home Leap 15.2 one on OBS, but this doesn't currently cause an issue as the only package required for there is our intended provider of systemd-presets-branding: Which is noarch, so it still gets install OK from that repo. But I'd rather we use the proper AArch64 variant of this repo used in the Pi4 profile. I know this is for Factory but I think matching arch is better than matching base OS and it's our best option currently. But it's not currently a show stopper given the noarch nature of this singe package. But we may well add arch specific packages to that x86_64 specified repo which would not work out so well for existing installs. I'll open a GitHub issue for this specifically so we can reference it going forward given it involves the caveat of not honouring the OS base match re Leap 15.2 vs Factory. And having now also specified the size, our interim .raw file is also 16 GB. Cumbersome but our intention is to have a 16GB minimum anyway, and those attempting to use the raw.gz I'm hoping to distribute as well will then at least be informed, upon attempted write, if their target device is inadequately sized. I have also confirmed that this interim raw image, post image size specified, does (as before size was added) successfully self expand when dd'ed to a larger device: with the resulting: The ROOT pool show 6.42% usage.
Cheers, and as I'm merging this as-is shortly, to bring it into the fold as it were, the referenced repo name change can go in as it's own dedicated pull request at the appropriate time. Thanks a bunch for helping us up-our-game on the AArch64 front and for helping to support Rockstor's development in this important direction. Delighted to be merging our very first contributed profile. |
This is to support 64-bit ARM systems that implement the Embedded Boot or Server boot standard - the basic gist is that these systems follow a PC style boot flow using EFI and (typically) Grub2.
With some tweaks it might be possible to use this for the Raspberry Pi 3 and 4 as openSuSE embeds a copy of U-Boot to implement EFI - in which case we can have just one aarch64 image.
I haven't named it after my system (Ten64) as there are wider applications such as virtual machines -in fact all the systems I am running in "production" at the moment are running as VMs (on top of Ten64 using μVirt) - but I have added the drivers for the Ten64 and have tested 'bare metal' as well. (Better naming suggestions welcome!)
Note that I have set the output format as qcow2 as this avoids transport issues with sparse images (e.g non-sparse aware tools inflating the image to full size). Ten64 at least is able to write qcow2 images using it's Recovery feature - for other boards we could convert to compressed raw post-build.