Add QCOW2 build mechanism #349
Conversation
|
Looks alright at a glance, thanks. It might be a while before I can go through it properly. |
Thanks! Take your time! |
Merge upstream
Merge upstream into qcow2
Force fs sync on umount and reverse kpartx/qemu-nbd unloading
Merge upstream
Remove info regarding manual NBD_DEV setting. Not possible anymore.
|
Sorry for the additional commits. I try to stay up to date with the main repo while adding some features and make the whole patch more robust... Commit 4ea3675 is the biggest so far, because I thought about the export stages again. As you know, when I opened this pull request I thought it wouldn't make any big difference to change the export stages. But I found out that if I would create the base qcow2 image with the 2-partition layout (vfat/ext4) right from the beginning, I could use simple calls to |
|
For anyone who is wondering about the many syncs: I am running the whole thing from within a Ubuntu 18.04 VM under VMware Workstation on a laptop and on a DebianARM64 VM under Proxmox, respectively. Both systems can get under a heavy load and if the overall load gets too high, the image operations inside the VM fail from time to time. Sync makes it work... |
Merge upstream.
configure unprotected wifi when WPA_PASSWORD is unset (RPi-Distro#357)
Merge upstream
Merge upstream
Merge upstream.
|
Rework upstream merge to stay up-to-date. |
|
This seems a useful addition to improve speed and space! Any chance to see it merged in upstream repository? |
|
As I am using this in all my derived pi-gen projects and never went back to traditional build mechanism, as it simply is too slow and uses too much space, I can assure you that this speeds up things like hell :-) - I don't understand why there is so little to no reaction but who knows... I tried to keep up with the main repo the last months, resolve conflicts and stuff, but for now, from my side, I will not fix the mentioned conflicts at the moment, as they are simply of no interest to me. |
|
@XECDesign Any chance this can get merged soon? |
|
Yeah, it's a worthwhile change, but I can't spend time checking it. Given that there's now verification that it works from another user, I am a bit more comfortable with accepting it as is. |
|
If I find time the next days I can add the last changes from master branch to my branch to resolve the conflicts if you wish... |
|
That would be much appreciated. |
|
I'm currently using pi-gen to build a custom image and the build needs half an hour on GitLab CI. I'm definitely interested in testing this PR if the conflicts have been resolved. That will give you some extra verification, @XECDesign. @pandel How much of a speedup can I expect from a half-hour build in the traditional way? |
|
@koenvervloesem I simply can't say. It really depends on the underlying hw or vm you are using, cached debs or not, network speed, whatnot. If you have a very fast hdd, then it might not be that much, because the file copying mechanism in the "old" way is fast enough. For me, creating a base qemu and then only derived ones for every next step is much faster. EDIT: |
|
Image size looks okay, but alignment doesn't seem right - 4 sectors rather than 4MiB. |
|
Partition id should be 0c, not 0b. Not sure if it makes a difference, but I remember having obscure issues in some cases, so I'd like to keep changes to a minimum. |
|
Many thanks! I look into it as soon as possible. The change should alteady have corrected alignment but I will find the reason why it didn‘t! |
|
Sorry, sorry, sorry! No time for nothing... Alignment and partition type should be correct now. I also added the bootable flag to the FAT partition, just in case. Please re-test a last time, hopefully... |
The bootable flag is ignored on the Raspberry Pi. |
As I said, just in case ;-) - and it doesn't hurt anyway. |
|
@XECDesign My last tests are working fine. Any recommendations? Is the partition layout now right for you? |
|
@XECDesign Any news regarding this PR? Does it work for you? |
|
Thank you for keeping this up to date and sorry about delaying this again. One more bug release image release and then I'll circle around to PRs again. |
|
I'll just merge this now to avoid further delays and address any issues with it if they come up. Thanks |
This reverts commit bf8c9f5.
This reverts commit bf8c9f5.
(cherry picked from commit bf8c9f5)
This reverts commit bf8c9f5.
This reverts commit bf8c9f5.
(cherry picked from commit bf8c9f5)
(cherry picked from commit bf8c9f5)
(cherry picked from commit bf8c9f5)
What's this?
This PR integrates a build variant utilizing qcow2 based images. It is mainly focused on reducing build time and space. Some additional changes have been made to the export stages for this as well, but are only activated if you use the qcow2 mechanism.
How does it work?
This mechanism replaces the copy_previous style build by creating a base qcow2 image in WORK_DIR at the beginning of the first stage and adding additional so called backing images stage by stage on top of it. These backing images record only the changes of the stage they belong to. That leads to reduced space consumption and faster stage creation times.
Only the two export stages (image and noobs) are created "the old fashioned" way, because it would need a more in depth rework of both export stages to use qcow2 for them as well. I think that this is not really necessary.Qow2-based export-image stage has been added with with 4ea3675, see my comment below.I dont' like it! How to deactivate?
In this PR, qcow2 based building is activated by default. If you want to activate the normal build variant, set
USE_QCOW2=0inside yourconfigfile.Further information
You will find further descriptions inside the updated README.md as well.
I developed and tested it under Kubuntu 18.04.03 LTS, with and without Docker and it works for me (not taking the current qemu/i386 problem into account, as this is all about image building).