Build promotion pipeline YAML

[JohnTortugo]

Closes: #4601

Changes:

• Most of the changes are to plumb parameters fetched from BAR/Maestro to the YAML stages/jobs.
• Other set of changes is to patch DownloadBuildArtifacts tasks to be able to download from the current build or from a pointed build.
• Renamed some files to make more clear the distinction of promote build logic and assign build to channel logic.

I've been testing the Build Promotion in this pipeline.

I also tested if the post-build.yml templates are still working when not used for promoting a build. See test here.

[riarenas]

Is there a reason why most of this is in eng/common? There's going to be a single build promotion pipeline that should take the yml from Arcade.

I'd like us to decouple the release variables that are really common from whatever is only needed for channel assignment after the fact. The pipeline can still reference the stages in eng/common, but unless there's something I'm missing, I don't think we need to distribute all of these changes to all repos.

[riarenas]

The logic seems sound, but I'd like us to not add more files to eng/common unless it really makes sense for every repo to have this.

[JohnTortugo]

@riarenas - I addressed the comments we discussed offline: no new files in eng\common

Note that some variables are used only in build promotion and others only at build creation time. Other variables are used in both cases. The major change here is because we need to be able to download artifacts from the current build or from another [completed] build.

I use the same Job in setup-maestro-vars.yml for all vars because that reduces the number of changes to the other files. Otherwise I'd have to add a coalesce(xxxx, yyy) in every place we refer to a variable that can be declared in different jobs.

Testing the build promotion: https://dnceng.visualstudio.com/internal/_build/results?buildId=488127&view=results
Testing the post-build YAMLs for build creation: https://dnceng.visualstudio.com/internal/_build/results?buildId=488352&view=results
Testing the post-build YAMLs for build creation + activating a channel: https://dnceng.visualstudio.com/internal/_build/results?buildId=488415&view=results

[JohnTortugo]

Ping

[riarenas]

Sorry I missed the latest update. Looking now.

[riarenas]

The test build at https://dnceng.visualstudio.com/internal/_build/results?buildId=479710&view=results failed. Maybe not caused by the changes, but the errors happened in one of the publishing tasks, so I'd like to have a successful build + publish with these changes before merge.

[JohnTortugo]

Re-run again the tests:

• Build promotion pipeline
• Build creation pipeline

[riarenas]

Build promotion pipeline

This build didn't publish to any channels

[JohnTortugo]

Tested it again here, now publishing to some channel:

• https://dnceng.visualstudio.com/internal/_build/results?buildId=496071&view=results
• https://dnceng.visualstudio.com/internal/_build/results?buildId=496027&view=results

Also added the check to validate that the channel exist.

[riarenas]

Looks good to me, but I started thinking that it's not easy to tell what the pipeline is doing just from looking at the build.

I think we can look at what Davis did in this PR https://github.com/dotnet/arcade/pull/4664/files so that when looking at a build for that pipeline, it's clear which build was promoted, and to which channel. What do you think?

[JohnTortugo]

Looks good to me, but I started thinking that it's not easy to tell what the pipeline is doing just from looking at the build.

I think we can look at what Davis did in this PR https://github.com/dotnet/arcade/pull/4664/files so that when looking at a build for that pipeline, it's clear which build was promoted, and to which channel. What do you think?

Yeah, looks simple and useful enough to include in this PR.

[JohnTortugo]

Test build with build tags: https://dnceng.visualstudio.com/internal/_build/results?buildId=496298&view=results

[JohnTortugo]

New test build here: https://dnceng.visualstudio.com/internal/_build/results?buildId=496588&view=results

[JohnTortugo]

Fixed a bug while enabling/disabling conditions. Ran new tests to test that part specifically:

• build promotion
• build creation