Hypothetical new toolchain build flow

[dmcilvaney]

I'm investigating a new wrapper around the toolchain bootstrap scripts, I'd like some input on the various modes the toolchain can operate in.

The "build" mode can be one of Auto, Fast, Force, Never. Generally auto will try and get a toolchain as fast as possible and ideally should be the default going forward.

The archive is a path to a toolchain.tar.gz, and will take precedence over locally built/downloaded packages.

A new feature will allow automatic updating of the manifest files. This is future looking and likely won't be in V1.

There is also a global cache flag. It doesn't affect the overall behavior of the tools, just short-circuits certain build steps using pre-built parts available locally in the dev environment.

Background:
The prototype of the wrapper tool currently lives inside a new toolchainV2.mk which can fully replace the existing toolchain.mk. It calls a go tool which in turns calls the existing toolchain scripts.

Currently my intent is to leave the toolchain scripts unmodified. Future iterations of the tool can move logic out of the shell scripts into the go tool to simplify the process. The tool will examine the local environment and decide the optimal set of operations needed to generate a toolchain: downloading, building the raw bootstrap, and building he official rpms.

The go tool as it currently exists is not suitable for direct calling, since its a wrapper for the scripts it still needs to pass the dozens of parameters through to them, meaning it has a lot of parameters currently. Instead it provides a make recipe for producing the toolchain .rpms the same way the existing toolchain.mk does.

Does the table make sense? Any suggestions for changing the behavior? 

Old table version

| Mode:Auto | Mode:Fast | Mode:Force | Mode:Never | Archive | Update Manifest | | Behavior | |----------- |----------- |------------ |------------ |--------- |----------------- |--- |---------------------------------------------------------------------------------------------------------------------------------- | | ✅ | ⛔ | ⛔ | ⛔ | ✅ | ✅ | | Manifest: Update from Archive
Download: NO
Rebuild: NO | | ✅ | ⛔ | ⛔ | ⛔ | ✅ | ⛔ | | Manifest: Toolkit
Download: NO
Rebuild: NO | | ✅ | ⛔ | ⛔ | ⛔ | ⛔ | ✅ | | Manifest: If './SPECS' available, use local, otherwise latest on PMC
Download: Any without a './SPEC'
Rebuild: Any missing | | ✅ | ⛔ | ⛔ | ⛔ | ⛔ | ⛔ | | Manifest: Toolkit
Download: All matching
Rebuild: Any missing | | ⛔ | ✅ | ⛔ | ⛔ | ✅ | ✅ | | INVALID (can't build & use archive) | | ⛔ | ✅ | ⛔ | ⛔ | ✅ | ⛔ | | INVALID (can't build & use archive) | | ⛔ | ✅ | ⛔ | ⛔ | ⛔ | ✅ | | Manifest: Update from './SPECS', never PMC
Download: Any available from PMC
Rebuild: Any missing after download | | ⛔ | ✅ | ⛔ | ⛔ | ⛔ | ⛔ | | Manifest: Toolkit
Download: All matching
Rebuild: Any missing | | Mode:Auto | Mode:Fast | Mode:Force | Mode:Never | Archive | Update Manifest | | Behavior | | ⛔ | ⛔ | ✅ | ⛔ | ✅ | ✅ | | INVALID (can't build & use archive) | | ⛔ | ⛔ | ✅ | ⛔ | ✅ | ⛔ | | INVALID (can't build & use archive) | | ⛔ | ⛔ | ✅ | ⛔ | ⛔ | ✅ | | Manifest: Update from './SPECS', never PMC
Download: NO
Rebuild: Everything | | ⛔ | ⛔ | ✅ | ⛔ | ⛔ | ⛔ | | Manifest: Toolkit
Download: NO
Rebuild: Everything | | ⛔ | ⛔ | ⛔ | ✅ | ✅ | ✅ | | Manifest: Update from Archive
Download: NO
Rebuild: NO | | ⛔ | ⛔ | ⛔ | ✅ | ✅ | ⛔ | | Manifest: Toolkit
Download: NO
Rebuild: NO | | ⛔ | ⛔ | ⛔ | ✅ | ⛔ | ✅ | | Manifest: Latest on PMC
Download: YES
Rebuild: NO | | ⛔ | ⛔ | ⛔ | ✅ | ⛔ | ⛔ | | Manifest: Toolkit
Download: YES
Rebuild: NO | | ⛔ | ⛔ | ⛔ | ⛔ | 🎲 | 🎲 | | INVALID |

Mode	Archive	Update Manifest		Manifest	Download	Rebuild
Auto	✅	✅		🟢 aka never 1	🔴 aka never 1	🔴 aka never 1
Auto	✅	⛔		🟢 aka never 2	🔴 aka never 2	🔴 aka never 2
Auto	⛔	✅		🟢 aka fast 3	🟡 aka fast 3	🟡 aka fast 3
Auto	⛔	⛔		🔴 aka fast 4	🟢 aka fast 4	🟡 aka fast 4
Fast	✅	✅		🚫 Invalid, can't build and use archive	🚫	🚫
Fast	✅	⛔		🚫 Invalid, can't build and use archive	🚫	🚫
Fast	⛔	✅		🟢 Use files in ./SPECS, otherwise PMC **	🟡 Yes (missing *.spec version only) *	🟡 Yes (local *.spec only) *
Fast	⛔	⛔		🔴 Use manifest from toolkit	🟢 Yes	🟢 Yes, "delta build"
Force	✅	✅		🚫 Invalid, can't build and use archive	🚫	🚫
Force	✅	⛔		🚫 Invalid, can't build and use archive	🚫	🚫
Force	⛔	✅		🟢 Use files in ./SPECS,	🔴 NO	🟢 Yes, "full build"
Force	⛔	⛔		🔴 Use manifest from toolkit	🔴 NO	🟢 Yes, "full build"
Never	✅	✅		🟢 Use contents of archive	🔴 NO	🔴 NO
Never	✅	⛔		🔴 Use manifest from toolkit	🔴 NO	🔴 NO
Never	⛔	✅		🟢 Use latest on PMC	🟢 Yes	🔴 NO
Never	⛔	⛔		🔴 Use manifest from toolkit	🟢 Yes	🔴 NO

*

Download from PMC anything that is either missing locally, or has a version missmatch. However, update manifest is on so local .spec versions should match... Is this desired behavior? Or should we force rebuilding local copies somehow?

**

Manifest will prioritize versions found in ./SPECS over PMC, but will fill in any gaps on PMC. This feels like it could break?

[christopherco]

Tagging a few folks @PawelWMS @anphel31 @jslobodzian for their thoughts on this new proposal.

I'm investigating a new wrapper around the toolchain bootstrap scripts, I'd like some input on the various modes the toolchain can operate in.

I will need a lot more context to better understand what this proposal is trying to accomplish and where you envision this wrapper to live.

• Which scripts are you targeting to wrap with this new wrapper? raw toolchain stage 1, rpmbuild toolchain stage 2, both, portions of one, etc.?
• Are you expecting this new wrapper interface to be exposed to the user directly so the user can directly build the toolchain? Or are you more looking to replace current toolchain build operational behavior driven through the Makefile?

The "build" mode can be one of Auto, Fast, Force, Never. Generally auto will try and get a toolchain as fast as possible and ideally should be the default going forward.

Can't tell if these build modes are your proposal? I think these 4 modes are actually your shorthand names for some of our current toolchain behavior?
Specifically to the modes:

• Never and Force make sense. Force always builds the toolchain from scratch and Never never builds the toolchain
• Auto and Fast seem redundant. What is the difference? Can we get rid of one?

The archive is a path to a toolchain.tar.gz, and will take precedence over locally built/downloaded packages.

Are you referring to an archive that is user-supplied during invocation or the tar.gz's that are constructed as intermediate build artifacts when building the toolchain stages? Thoughts on simplifying by always operating off of the intermediate tar.gz's and updating those tar.gz's when user is iterating on toolchain packages?

A new feature will allow automatic updating of the manifest files. This is future looking and likely won't be in V1.

V1 of what exactly?

There is also a global cache flag. It doesn't affect the overall behavior of the tools, just short-circuits certain build steps using pre-built parts available locally in the dev environment.

Where can I get more info about the global cache flag? Not sure I have much to comment on yet without really knowing which steps are short-circuited and how and replaced by what.

Does the table make sense? Any suggestions for changing the behavior? 

I think the "Behavior" portion of the table is too vague to be of real use. Also with the way this table renders, you might be better served condensing "Mode" into its 4 values so the table renders with more room on the "Behavior" column which should make it easier to read. (btw I do like your use of emojis ✅ )

No real suggestions on altering behavior other than what I wrote above. Will need much more info to really start thinking about what to alter with a new wrapper.

[dmcilvaney]

Tagging a few folks @PawelWMS @anphel31 @jslobodzian for their thoughts on this new proposal.

...

I've added some background.

Can't tell if these build modes are your proposal?

I would like to simplify the existing set of different flags into one single argument. Currently you need to set various combinations of DELTA_BUILD, INCREMENTAL_TOOLCHAIN, ALLOW_TOOLCHAIN_DOWNLOAD_FAIL, REBUILD_TOOLCHAIN, TOOLCHAIN_ARCHIVE (and probably more...) to get different behaviors. My proposal is to reduce all those combinations into a single variable which can be one of never, force, auto, fast.

fast vs auto: Open to simplifying, I was thinking auto will select between fast and never depending on the environment. For a derivative I'm not sure fast ever makes sense, never is really the only option.
-fast: Build packages using delta build, other optimizations where possible
-never: Refuse to build packages, only download

Are you referring to an archive that is user-supplied during...

It's unchanged from the current argument of TOOLCHAIN_ARCHIVE=..., a lot of the pipelines rely on using that argument, so I don't want to change the behavior. I'm not sure I follow the suggestion for iteration though.

Where can I get more info about the global cache flag?
Its a super early stubbed out thing I wrote just to see what it would bring performance wise. Will need to have a separate discussion about caching later. (prototype just hashes a set of input files and if the inputs are the same it will give the cached file back out, VERY simple and probably not 100% correct).

I'll have a look at reworking the table. I tried a few different layouts and none of them really worked well...

[dmcilvaney]

Ok, table is updated. Old table is hidden in the spoiler tag if you need it.