Question relating to nested Targets

[JustArion]

Warning

Long Post. Verbose, but not written by AI lol

I had an idea to make myself some build servers for various open-source programs, then to build them via NUKE / Fallout. I've ran into some issues that hopefully someone more experienced can share some guidance on how I should go about structuring this.

Ideally I'd like for something like ./build.ps1 mpv or ./build.ps1 all --target-os 'windows' where you can target multiple operating systems,

How it currently works is via the Main method I'd do multiple Execute<MPV_Linux>(x => x.Build), Execute<MPV_Windows>(x => x.Build) which is very tedious, requires editing the source for changing targets.

How I see NUKE / Fallout set up with the expression dependencies it uses reflection within the context of the current : NukeBuild / : FalloutBuild class to detect dependent targets. This is problematic when you have nested targets, eg.

public class MPVServer : NukeBuild
{
    readonly MPVServer_Native NativeBuildServer = new();
    readonly MPVServer_Linux LinuxBuildServer = new();

    public Target BuildAll => _ => _
        .DependsOn(LinuxBuildServer.Build)
        .DependsOn(NativeBuildServer.Build);
}

Then inside your main class, you'd have something like:

class BuildManager : NukeBuild
{
    readonly MPVServer MPV = new();
    readonly VSCodeServer VSCode = new();

    public static int Main() => Execute<BuildManager>(x => ...);

    Target BuildAll => _ => _ ...
}

Due to Targets having dependencies locked to their own respective instance, this approach is impossible.

Another approach I've considered is executing nested Execute<ProjectServer>(x => ...) calls within each-other but this is also impossible due to NUKE locking the build.log file during each top-level Execute call.

What NUKE / Fallout recommends in this approach from what I've read is using Build Components. But I've hit a snag on this too. Interfaces restrict the use of fields and non-public members, which means persisting state is problematic.

I started looking into how the original NUKE build chain deals with the issue of persisting state and it just seems absolutely horrible. Their original approach is making the main build class partial, and daisy-chaining on state as they go (example found here). This approach also has an issue of being unable to have duplicate member names. So you can only have 1 "Restore" member through the entire c;ass (And no, C#'s file keyword can't be used on member instances, only top-level types)

Another approach I've considered is isolating each build server to their own respective project, then controlling everything from the NUKE build project via DotnetTasks.DotnetRun, but this (probably) has the same issue as the Execute<ProjectServer>(x => ...) calls, where the build.log will get locked by the top build project.

Would love some feedback on how I can approach nested build servers using NUKE / Fallout

- arion

[ChrisonSimtian]

tldr; yes the original NUKE approach IS horrible and we are working on fixing this with an architecture piece that probably spans at least the next 1-2 major versions. So long story short is, you probably have to put up with it for now :-(

I had an idea to make myself some build servers for various open-source programs

Awesome, this is exactly my vision. Fallout at some point should be its standalone build server and deployment agent. Just give us some time, we're slowly but surely getting there :-) I too have a homelab and most of my projects can happily build there, instead of in MS cloud somewhere in a US DC ;-)

Their original approach is making the main build class partial, and daisy-chaining on state as they go

Iirc then we introduced context with todays release, but at least the main branch has it. We're slowly building it up so that you can have a shared context instead of limping this along in static partial classes. Basically real OOP :-)

Conceptual for your problem:
I would probably just build up a matrix

[Parameter] string[] TargetOs = [ "windows", "linux", "macos"]

and then have the one execute method that iterates over all those TargetOs and runs your build definition.
Make the OS an input parameter if you have to (i.e. you might wanna pass it down to cli calls or other stuff)
Its a poor mans matrix fanout, but it should already work today.

As for running one central build to build all the other builds, not quite in scope unfortunately.
Assuming your SCM sits in github you can just trigger other workflows via API, or you have some sort of master repo that defines your builds as a shape, similar to what I do with my homelab (its basically the matrix param approach from above)

As I said, we're reworking the underlying architecture because in the future those things should just work out of the box and natively

---

What I would suggest as a realistic setup (i.e. I run this at home):
If you wanna have a proper build server, I would suggest running Fallout per repo and having your own Forgejo server with a bunch of runners in the back.
I do that at work as a pre CI/CD stage (to bypass queues in Github and Azure Devops) via Docker and at home I got my lab running runners (both gh and forgejo) to do the dirty work for me.
Mind you one day it would be great if Fallout had its own runner, its own orchestrator and release manager etc. but we're still a long way away from that.

[JustArion]

We're slowly building it up so that you can have a shared context instead of limping this along in static partial classes.

Could you explain a bit more for me on how this would work, or a possible example of how it would work? I know this is under the context of the FalloutBuild class and reducing extensions via partial classes, but are there plans for anything to expand the contexts that a Target can access? Like the code example for nested targets:

public class MPVServer : NukeBuild
{
    readonly MPVServer_Native NativeBuildServer = new();
    readonly MPVServer_Linux LinuxBuildServer = new();

    public Target BuildAll => _ => _
        .DependsOn(LinuxBuildServer.Build)
        .DependsOn(NativeBuildServer.Build);
}

To my knowledge this throws an exception since it tries to look for the target called "Build" in the current context of the class and doesn't look deeper into the actual reference.

I tend to agree that a per-project build is probably better for my case, but I personally dislike the idea of a fragmented base for this kind of project. That and I have never done anything with Forgejo before, which is probably the solution to the fragmentation issue.

[ChrisonSimtian]

Well currently the whole NUKE architecture is basically a one-shot commandl-ine run. Nothing in it is designed to let you run in an endless loop as a daemon with different context (plural)

So lets take the LinuxBuildServer.Build for example, it simply boils down to Build as thats the name. Now the NativeBuildServer.Build one shares the same name.
In your architecture you want them both to co-exist in parallel, while as NUKE would want to have them as 2 separate builds (they are one-offs, remember?)
It doesnt really like the orchestration approach you picked and at the moment, mostly due to its static architecture, cant have multiple instances of the same/similar thing running side by side.

We introduced a context class to hold context-data with the latest release, this eases this a bit up and builds more towards "define once, run multiple instances of it in parallel" but its only halfway there, you know?

So as I said at the moment you'd have to do a per-project and work around the OS thing with a matrix for the OS passing this through as a parameter into the one build (and then have different paths for different OS in the same build OR have a per OS build with a unique name)

Btw. do you wanna share a repo here so that we can go a bit more into detail? Pretty sure we can figure out a good solution together and Im happy to have some of the feedback flow back into our backlog for the next version(s) ;-) As I said, my vision is to enable something in that direction

[JustArion]

I've published a temp repo here. The Main method is inside ./Build/BuildRunner.cs The server is in ./Build/Servers/Seanime/SeanimeServer.cs BuildRunner's kinda barren right now since the main focus was first building the actual implementation, then focusing on multi-builds when the time arrives, like now. The server uses nested contexts to build platform specific builds (all with various targets, with various dependencies and skip conditions). Of course, none of the nested contexts actually work due to the error of Sequence contains no matching element due to reflection failing to find the Target within the class' own context. This error will persist from BuildRunner -> SeanimeServer -> SeanimeServer.Linux or SeanimeServer.Native (Native here is just a placeholder name for host OS that isn't Linux)

[ChrisonSimtian]

hey fyi, i raised a PR against your repo with what I believe could give you an idea on how to solve your problem.
Let me know if you need more guidance