Add basic support for hotfix builds

[jlebon]

ART needs the ability to perform hotfix builds, i.e. builds outside the
regular stream for a specific purpose. These builds must not pollute
production stream references.

Support this via the new hotfix section of the pipecfg. Require a
name subkey which is then used to modify the S3 subpath we upload to
and the oscontainer tags we push to. Add new hotfix parameters to the
build, build-arch, and release jobs which will be used to pass the
repo containing the hotfix pipecfg.

Don't support hotfix cloud uploads since it's not yet needed and
requires plumbing of the hotfix prefix.

Requires: #739

[jlebon]

Still need to test all this!

[dustymabe]

I'm not the biggest fan of the first commit I'm not sure the benefit is worth us losing the default list that shows up in the cases where a human is typing parameters in manually.

2nd commit LGTM

[dustymabe]

we've actually been using this (overriding the default) today. Right now the pipeline is configured to build for ppc64le too but we tell ourselves when we go to make official builds to remove the ppc64le from the list.

Now there will be no list to remove an item from, but rather an empty field. It's not a big deal but I kind of prefer the other UX.

[jlebon]

I'm not the biggest fan of the first commit I'm not sure the benefit is worth us losing the default list that shows up in the cases where a human is typing parameters in manually.

IMO we should avoid humans typing parameters when possible. Ideally, all the information needed is in git in the pipecfg. Having parameters is still valuable of course when on-the-fly overriding is necessary. But then we should make that semantic really clear, which is what the patch aims to achieve. The relationship was already confusing before between the parameter and the pipecfg key, but with hotfix pipecfgs it becomes even more so.

For the ppc64le situation right now in FCOS, my suggestion is to either support an additional_arches override or a skip_arches per-stream knob to avoid building it in the prod stream. Then we can git revert it when we're actually ready to build ppc64le.

[dustymabe]

I'm not the biggest fan of the first commit I'm not sure the benefit is worth us losing the default list that shows up in the cases where a human is typing parameters in manually.

IMO we should avoid humans typing parameters when possible. Ideally, all the information needed is in git in the pipecfg. Having parameters is still valuable of course when on-the-fly overriding is necessary. But then we should make that semantic really clear, which is what the patch aims to achieve. The relationship was already confusing before between the parameter and the pipecfg key, but with hotfix pipecfgs it becomes even more so.

I agree. Typing in things isn't ideal, but it is really nice for when you need it to have the default values there in front of you to edit. How about we compromise and still show the values (not sourced from the pipecfg) and then validate them after the hotfix pipecfg is processed? The following is just draft code but conveys the idea I think:

diff --git a/jobs/build.Jenkinsfile b/jobs/build.Jenkinsfile
index 6108cac..26e48b9 100644
--- a/jobs/build.Jenkinsfile
+++ b/jobs/build.Jenkinsfile
@@ -24,7 +24,7 @@ properties([
              trim: true),
       string(name: 'ADDITIONAL_ARCHES',
              description: 'Override default additional target architectures (space-separated)',
-             defaultValue: "",
+             defaultValue: pipeutils.get_supported_additional_arches().join(' '),
              trim: true),
       booleanParam(name: 'FORCE',
                    defaultValue: false,
@@ -66,6 +66,13 @@ if (params.PIPECFG_HOTFIX_REPO || params.PIPECFG_HOTFIX_REF) {
 def cosa_img = params.COREOS_ASSEMBLER_IMAGE
 cosa_img = cosa_img ?: pipeutils.get_cosa_img(pipecfg, params.STREAM)
 def additional_arches = params.ADDITIONAL_ARCHES.split()
+
+for (arch in additional_arches) {
+    if (!pipecfg.additional_arches.contains(arch)) {
+        error('requested architecture disallowed by pipecfg')
+    }
+}
+
 additional_arches = additional_arches ?: pipecfg.additional_arches
 
 def stream_info = pipecfg.streams[params.STREAM]

For the ppc64le situation right now in FCOS, my suggestion is to either support an additional_arches override or a skip_arches per-stream knob to avoid building it in the prod stream. Then we can git revert it when we're actually ready to build ppc64le.

I think we should do that too. The ppc64le situation has gone on long enough that it warrants a knob if we can afford to implement that.

[dustymabe]

I think with the suggestions above we get the best of both worlds (reached through collaboration and compromise).

[jlebon]

How about we compromise and still show the values (not sourced from the pipecfg) and then validate them after the hotfix pipecfg is processed?

Hmm, in the case of a hotfix build for x86_64 only (or with stream-level knobs, any stream build where the arch set doesn't match the Jenkins set), that would require the operator to both edit additional_arches in pipecfg and trim down the list from the default set in the parameter. The UX there isn't ideal. I'm also still not a fan of the "off by one run" semantic that can cause confusion here. It's not like we're turning arches on and off very often but it's also such a subtle issue that it's bound to trip someone up down the line.

Two suggestions:

1. Instead of modifying the default value, we put the list of supported arches in the description. E.g. tweaking your patch slightly:

   diff --git a/jobs/build.Jenkinsfile b/jobs/build.Jenkinsfile
   index 6108cac..26e48b9 100644
   --- a/jobs/build.Jenkinsfile
   +++ b/jobs/build.Jenkinsfile
   @@ -24,7 +24,7 @@ properties([
                 trim: true),
          string(name: 'ADDITIONAL_ARCHES',
   -             description: 'Override default additional target architectures (space-separated)',
   +             description: "Override default additional target architectures (space-separated). Supported arches: ${pipeutils.get_supported_additional_arches().join(' ')}",
                 defaultValue: "",
                 trim: true),
          booleanParam(name: 'FORCE',
                       defaultValue: false,

   So then if someone wants to manually insert arches, they can copy/paste or retype from that list and modify as needed.

2. It sounds like what you're getting at is the ability to skip arches defined in the pipecfg via parameters. We could instead add a SKIP_ARCHES that would be a better fit for this.

WDYT?

[dustymabe]

Two suggestions:

1. Instead of modifying the default value, we put the list of supported arches in the description. E.g. tweaking your patch slightly:

Yeah. Let's go with 1. 👍

[jlebon]

Updated!

[jlebon]

For the ppc64le situation right now in FCOS, my suggestion is to either support an additional_arches override or a skip_arches per-stream knob to avoid building it in the prod stream. Then we can git revert it when we're actually ready to build ppc64le.

I think we should do that too. The ppc64le situation has gone on long enough that it warrants a knob if we can afford to implement that.

Follow-up for that in #747.

[dustymabe]

LGTM