Podman farm support for emulated builds

[kcrocombe]

Feature request description

Existing farm functionality will only schedule a build for a particular architecture on a member of the farm that is native to that architecture (i.e it will only perform, say, arm64 builds on arm64 based machines). Any podman servers with on-board qemu emulation capability (which theoretically could also perform the builds), get ignored.

While leveraging the performance advantage of native hardware is the primary design intent of the farm, there are still situations where the ability to perform farm out builds to emulators serves a useful purpose. Even integration testing of farm functionality itself would easier if the bulk of it could be done without recourse to secondary machines.

From examination of the code base, it seems likely that what is described above was the original design intent for 'Farm'; builds could be performed by emulation, but builds on native hardware would be preferred. Indeed, much of the code to support this behaviour is already in place, albeit not production-ready.

Suggest potential solution

From the commit history, its not clear to why the above functionality never made it into production. Hopefully, it was just resource constraints, in which case I'm happy to try and complete. However, before I do so I just wanted to make sure there was no later design re-think on this, and it was decided that this was NOT the desired behaviour of the product.

Hence this issue, I suppose.

Have you considered any alternatives?

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

[tangentsoft]

I can think of only one reason you might not want this done without asking first: you previously set your farm up with all-native nodes, so it should never fall back to emulation. In that case, you either want it to wait and try again when the complete set of native builders do belatedly come up, or you want it to result in a ping to the sysadmin to investigate the reason why the native builders aren't coming up reliably. You don't want the build to suddenly become 10x slower just to get it to complete.

My use case for wanting the option of emulated architectures during farm builds is that it's the simplest way to get edge-case architecture support, as for i386, MIPS64, etc. The rarer or more niche the target platform, the less realistic it is to expect to have a fast native builder for these, disqualifying farm when it would otherwise be a perfect match.

I therefore suggest adding a flag to podman farm build as part of this to signal the tooling that falling back to emulation is acceptable, like --allow-emulation.

[Luap99]

This sounds reasonable, an extra option for this makes sense to me.

[kcrocombe]

I agree we need to control whether emulation is active or not, but maybe for different reasons, and maybe I'd suggest a slightly a different way...

At the moment, in the event of a node in the farm being unavailable there is no danger of any falling back to emulation because unless ALL the nodes in the farm are available then the build will just fail completely. As the current product is envisaged, that makes good sense; a common use-case (podman farm build --farm myfarm --cleanup -t name .) relies on interrogating the nodes in order to determine which architectures it has to build for. Without all the nodes being present, it doesn't really know what it's being asked to build, so it can't make alternative arrangements. You could change it and build some fault tolerance into the product, though, personally, I'm not sure you'd want to; there are plenty of other solutions already out there that will crack that particular nut.

My bigger concern would also be around inadvertently introducing unwanted behaviour into something like the podman farm build --farm myfarm --cleanup -t name . use case, though. If myfarm, is a two node farm of amd64 and arm64 architectures, with the existing product you know its going to build you a manifest of just those images. However, if the product started allowing uncontrolled emulation, then this particular use case may very well start building manifests of 6 or 7 architectures, all but two of which are unwanted.

You could control the behaviour with a new --allow-emulation flag as you suggest. However, I think that the behaviour could be regulated pretty well by the pre-existing --platforms flag, which is currently used to restrict builds to a subset of what is available on the farm.

Emulation would be off unless the --platforms flag is explicitly provided. If the --platform specifies a list of architectures, and one or more of them is not provided natively by the nodes in the farm, then by implication, you are asking for that particular architecture to be built by emulation.

Does that make sense?

[tangentsoft]

I like it! My use case is 32- and 64-bit ARM and Intel, of which I have three of the four in hardware. All I need do then is add a linux/i386 emulated builder to my farm, and things go on as before.

[Luap99]

Emulation would be off unless the --platforms flag is explicitly provided. If the --platform specifies a list of architectures, and one or more of them is not provided natively by the nodes in the farm, then by implication, you are asking for that particular architecture to be built by emulation.

That sounds good with me. One question though how should we decide on what node to schedule the emulation build? I assume pick a unused node from the farm (which has the right emulation arch setup of course) and if there is no free node left add an extra build to one that already does a native build and so on?

[kcrocombe]

No great sophistication to it, I'm afraid. The existing algorithm would schedule it on the first node it finds that has the capability to do the build. So if you had a node capable of emulating 3 archs, it would be tasked with doing all 3 builds (in parallel) even if there was another node in the farm capable of sharing the work. However, you can, of course, design your farm with 3 different emulator nodes each capable of just one type of emulation. Then the load is naturally balanced.

Alternatively, there is a trivially easy tweak to the algorithm that could be made. The above 'first-one-found' strategy would still apply, but would be over-ridden in one specific instance. Should a connection in the farm be actually named after a particular architecture, then all emulated builds for that particular architecture will be sent its way. So, a real example:

  [kevin@fed1 ~]$ podman farm list
  Name                    Connections                               Default     ReadWrite
  emulationFarm  [linux/arm64 linux/ppc64 linux/s390x anythingElse]  true        true

Now, the user can elect to take complete control over where the emulated builds happen, or just continue to use the first-one-found strategy.

[kcrocombe]

/assign

[github-actions]

A friendly reminder that this issue had no activity for 30 days.