With what Linux distro can you actually build this in 2021?

[sktpin]

I found this issue, telling me I need a 32bit Linux.

The last 32bit Ubuntu LTS I found was 16.04.7, so that's what I put on a VirtualBox VM.
Did apt-get update, upgrade, and attempted install the list of dependencies mentioned in the readme.md here.
One of them refuses to install:

The following packages have unmet dependencies:
libarchive-tools : Depends: libarchive13 (= 3.2.1-2~ubuntu16.04.1) but 3.1.2-11ubuntu0.16.04.8 is to be installed
E: Unable to correct problems, you have held broken packages.

Does this mean that that particular Linux distro, as is, is broken w.r.t. that library, or the way the packages are set up rather?
Can that be fixed?

Or is there, by chance, some Linux VM image floating around somewhere that's known to allow one to build this project?
I read Debian is also supposed to work, I have no experience with Debian, though, and probably it won't be easier to get a 32bit version that still allows me to install dependency packages, for desktop PC i.e. x86, anyway.

What are you guys actually using?

[XECDesign]

Official builds are currently built on an arm64 machine running Ubuntu 20.04.2 LTS. You should be able to build on a pi running Raspberry Pi OS too, but I haven't tried that in a while.

qemu is in a bit of a broken state right now, which makes building things on non-arm hosts tricky. Once this issue is fixed and makes it into stable releases, I can take a look at improving amd64 support: https://gitlab.com/qemu-project/qemu/-/issues/385

If you build your own i386 qemu binary from the commit just before the one referenced in the issue above, you should be able to get it to work with newer versions of Ubuntu. There are some instructions included in the comments.

I've seen reports that the amd64 issue (that makes readdir quietly return nothing on 32-bit guests, IIRC) may have been resolved upstream, so maybe there's a right combination of software to make builds work without relying on i386 qemu builds. However, last time I tried it, I didn't get far and didn't have time to investigate how the fix was meant to work or why it hasn't.

[sktpin]

I also just installed Debian 11 - there was a 32bit one.
There I'm getting this:

Required dependencies not installed
This can be resolved on Debian/Raspbian systems by installing:
parted debootstrap zerofree dosfstools libcap2-bin kpartx

But if I apt install parted debootstrap zerofree dosfstools libcap2-bin kpartx , it says for every single thing that the newest version is already installed, ...
Some of those were in sbin and "which" did not find it, I copied them to bin. But dosfstools and libcap2-bin are nowhere to be found, even if I remove and reinstall them.

I tried to build it on 64bit Ubuntu 21 before, got a bunch of errors, and when I found the issue about 64bit systems linked above, I did not look further into it.

I realize one can do this on a Raspi itself, but that seems rather impractical. Alone because of storage. If I had a big and fast USB3 external drive laying around, ok... but I don't... Might get one if all else fails, but it obviously is much preferable to build this on a VM.

[XECDesign]

Alright, thanks for giving Debian 11 a go. It'll have to work there at some point, so I'll check what's going on.

[sktpin]

Since I wrecked the desktop by apt remove dosfstools libcap2-bin with a lot of dependencies that were also removed, I rebooted, this time without desktop. For the heck of it, I ran build.sh, and now it is running still... weird. No idea whether just a reboot would have sufficed, or any of my manual copying from sbin to bin did any of it.
While that is running, I'll create another VM with Debian 11, and if that doesn't work there, I might try 10, as that apparently was used before.
Edit: it just aborted, with a "Buffer I/O error on dev nbd0, logicval block 1, async page read" and "Dev nbd0: unable to read RDB block 0"

[XECDesign]

I currently have builds running on debian 11 xfce arm64 and i386 in virtualbox. Not seeing the issues you've mentioned so far. Not seeing any of the issues you've mentioned so far. I suspect both builds will result in broken images due to qemu, but I'll provide a workaround once I've confirmed it works.

[XECDesign]

On the i686 host, I just hit #454

[sktpin]

With Debian 10.10 i386, it's running much further now. Although I do see the occasional "no such file or directory", but it does continue...

[sktpin]

Now it ended with:

Applying patch /home/sk/src/PROJ/pi-gen/stage2/01-sys-tweaks/00-patches/02-swap.diff
can't find file to patch at input line 5
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|Index: jessie-stage2/rootfs/etc/dphys-swapfile
|===================================================================
|--- jessie-stage2.orig/rootfs/etc/dphys-swapfile
|+++ jessie-stage2/rootfs/etc/dphys-swapfile
--------------------------
No file to patch.  Skipping patch.
1 out of 1 hunk ignored
Patch /home/sk/src/PROJ/pi-gen/stage2/01-sys-tweaks/00-patches/02-swap.diff does not apply (enforce with -f)
[15:31:52] Unloading image
/home/sk/src/PROJ/pi-gen/work/2021-09-01-raspi-triplein-custom/stage2/rootfs: 10.4 GiB (11148898304 bytes) trimmed
/home/sk/src/PROJ/pi-gen/work/2021-09-01-raspi-triplein-custom/stage2/rootfs/sys
/home/sk/src/PROJ/pi-gen/work/2021-09-01-raspi-triplein-custom/stage2/rootfs/proc
/home/sk/src/PROJ/pi-gen/work/2021-09-01-raspi-triplein-custom/stage2/rootfs/dev/pts
/home/sk/src/PROJ/pi-gen/work/2021-09-01-raspi-triplein-custom/stage2/rootfs/dev
/home/sk/src/PROJ/pi-gen/work/2021-09-01-raspi-triplein-custom/stage2/rootfs/boot
/home/sk/src/PROJ/pi-gen/work/2021-09-01-raspi-triplein-custom/stage2/rootfs
/dev/nbd0 disconnected

Not sure those complaints were fatal or not. Never built this before.
I set on the configure that the image should not be put in a .zip file.
What is the actual image format then?
In the work folder I got a bunch of "image-stage*-qcow2" files. But there are states from 0 to 2, does that map to 1 to 3 of the "stage" name folders that start with 1, or is 0 something else here? I put a SKIP in stage4 and 5 folders, 3 was supposed to be built also, albeit heavily pruned w.r.t unneeded stuff.

[XECDesign]

Yes, very much fatal. The qcow images aren't what you're after. Successful builds end up in deploy/

Have you made any local changes like removing dphys-swapfile from stage2/01-sys-tweaks/00-packages?

[sktpin]

Oh, it looks like it! My aim was to make an image that contains only stuff my device needs. While I attempted to make sense of the short descriptions of the standard packages in some list I found, I might have erred there. Hopefully not much more.

Edit: Fixed that, now I also got thte "Directory not empty" / issue #454

[sktpin]

Building on the Raspi4 now... Seems the "... not empty" error is "fixed" by deleting the work folder, which means, after every change, all is done all over. Ok, but that at least works.
Getting another problem now.

/bin/bash: warning: setlocale: LC_ALL: cannot change locale (de_DE.UTF-8)
/bin/bash: line 1: setupcon: command not found

As for the locale thing, before that line, I get some dozen times "perl: warning: Setting locale failed", I wonder why it tries so often. I can't remember changing anything about locale in the config file etc, weird.
But it created the stage2 folder now, that's a step further than yesterday ^^ If only on the RasPi instead of the VM.

---

Edit: Ok, I didn't get that this somehow uses things of the image to do stuff with it during assembling the image.
I saw that the console-setup thing is another one I removed from stage2 packages.
Is there a list somewhere in the source tree that shows everything that's absolutely mandatory to leave in, in order to build a basic image at all?

[sktpin]

It now put something into deploy. The lite image is 1.6 GB. Can I, realistically, get that significanly smaller, like half or so? (I really need a rather bare bones image, basically a Linux that can run C++ apps I put on it, with USB gadget / network capability, but that's about it - no installing of new packages on the running device, only selfmade binaries of GCC-compiled programs)

What is weird is that it complains about stage4.qcow2 missing. It's not in the work folder either - but a stage5 folder is there.
Even though neither of them should (?), as I have "SKIP" and "SKIP_IMAGE" files in both, stage4 and stage5 folders in the source dir.

Would be cool if someone got this to work with e.g. Debian11, I must say this takes quite a while on the RasPi4 with some lame SDcard in it ;)

[sktpin]

Edit:
ok, not so fast... I was not able to repeat a build on Ubuntu. All I did was add one stage2 package (gdbserver), but this seems not related, so IDK what's going on: The build aborted around the time of also giving "warning: Deprecated use of backing file ... (detected format of qcow2)", it then shortly after gets llseek error and read error.
_
After cleaning the directories and a reboot it built again on Ubuntu21-64bit without any fixing of QEMU, this time also the "full" image (that, for me, only goes to stage3). I'm only getting complaints at the end that it can't find stage4 image - which is also not supposed to build.
But I have an image now that, from superficial inspection with 7zip GUI, in places like some config files, looks good.

---

That's interesting.
While, after fixing what I broke w.r.t. pruning packages,
this now builds until the lite image on Ubuntu21-64bit ... but not on Debian11-32bit due to the "stage0/rootfs/debootstrap': Directory not empty" error.
On Ubuntu, it writes into deploy the info file for the full image, but not the full image. I have not seen an error message, but I'll dig through this more thoroughly later.
What's weird is that it did not unmount /dev/loop0, so that it wouldn't let me delete the work wolder, because of some temp image folder in it. The build script had stopped and the end looked, superficially, ok, though.

[XECDesign]

My findings so far are that everything works if you use the right version of QEMU. The version of QEMU in every current distro is broken.

I build an i686 qemu-arm-static binary from 7abc8cabad, then register it with binfmt by running:

echo ":pi-gen-armhf:M::\x7f\x45\x4c\x46\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/home/shift/qemu-arm-static:OCF" | sudo tee /proc/sys/fs/binfmt_misc/register > /dev/null

The change is not persistent and will be lost after reboot.

I also needed to set USE_QCOW2=0, which I think I should make the default, since that's how official images are built and I'm finding qcow handling unreliable.

[marcone]

For what it's worth I've successfully built several images on 64-bit Ubuntu 20 ("Ubuntu 20.04.1 LTS (GNU/Linux 5.4.0-54-generic x86_64)")
The only problem I've noticed is #478, but I don't know if that's because I'm building on 64-bit or something else. I also only built "lite" images, so maybe there are problems in later stages I haven't run in to.

[XECDesign]

Yes, that's the issue that occurs when you use 64-bit qemu. Desktop images would also be missing icons when booted. If you switched to Ubuntu's version of qemu-user-static:i386 you'd run into a different set of issues.

For everything to work right now on a non-arm host, you'd have to build the binary yourself, as mentioned in the previous comment.

I could bundle it in with pi-gen, but I don't think people should run mystery binaries from github repos.

[sktpin]

Btw., what part of the build uses qemu, and why was this path of doing things chosen? I assume something from a running system that's the same as the target is needed, which somehow cannot (with reasonable effort) be done otherwise?

[XECDesign]

something from a running system that's the same as the target is needed

Having some trouble parsing what you mean there.

qemu (in user, not system mode) is used to run arm binaries inside the chroot, which we have to be able to do to set everything up. Otherwise things would fall over right after the base packages are extracted. The next step is to configure all those packages, which will run maintainer scripts that tend to call other binaries on the system. Without qemu, there's no dpkg, apt or anything else.

[sktpin]

I just have not seen things done this way. Linux image generation I saw before seems to rely on cross compilation - although I don't know how the "set everything up" part works there.

Where does the info come from what GCC version to incorporate in the image? I've only seen GCC mentioned in general in a packages file, but not a version. I wonder whether it's possible to use a newer version.

[XECDesign]

The image isn't built like you'd see with yocto of buildroot. Things aren't built from source, nor does GCC come into it. debootstrap pulls in core .deb files (which already contain binaries), extracts them, runs their maintainer scripts and then you get a small rootfs to work with which contains all the tools necessary to build up the rest of the system.

[sktpin]

Ah, thanks. W.r.t. GCC, I mean the GCC that will be present in the resulting image - can this be made a newer version than 8?

[XECDesign]

It depends on what's default and what's available in the distro being built. In our current releases based on Buster, that's GCC 8 and I'm not sure if a newer version is available. In the next release (bullseye), it will be GCC 10.

[sktpin]

Ah... right, as Debian11 is out... I haven't tracked this at all before, usual cycles of new Debian vs. new Raspberry OS, is there a rough estimate when the new one will be out? Then maybe I'll just wait for that one.

[XECDesign]

I'm a bit cautious about giving estimates, since sometimes people take it as a promise and complain if it takes longer.

[XECDesign]

To sum up the answer to the original question. None, unless you can work around the qemu issues (#541 (comment)), If you have the workaround in place, Ubuntu 20.04 LTS, Debian Buster and Bullseye have been tested to work.

[starbasessd]

Yes, that's the issue that occurs when you use 64-bit qemu. Desktop images would also be missing icons when booted. If you switched to Ubuntu's version of qemu-user-static:i386 you'd run into a different set of issues.

For everything to work right now on a non-arm host, you'd have to build the binary yourself, as mentioned in the previous comment.

I could bundle it in with pi-gen, but I don't think people should run mystery binaries from github repos.

I, for one, would be interested in the set of sub-instructions to use to set this up, and generate my own copy of qemu, or download it from a github repo, even if you decide not to include the qemu in the bundle. I have my own set of added in scripts, that work quite well, but using one of my Pi-4Bs to do the build takes them out of service for most of a day, and tossing up a VM using Debian Buster or Ubuntu would take a lot off of their plates.