teamcity: excessive number of full builds during each CI run

[knz]

I just noticed that every CI runs compiles the entire crdb binary, including all web assets etc, a minimum of 7 times:

• The acceptance CI target runs teamcity-acceptance.sh which in turn runs pkg/acceptance/prepare.sh which in turn runs builder.sh mkrelease linux-gnu

  In other words, the acceptance CI target builds its own crdb executable.

• Meanwhile, the Build test binary CI target runs teamcity-build-test-binary.sh which in turn runs builder.sh mkrelease linux-gnu

  In other words, the "build test binary" CI target builds its own linux crdb executable.

• Meanwhile, the roachtest CI target runs teamcity-local-roachtest.sh which in turn runs builder.sh make build

  In other words, the roachtest CI target builds its own linux crdb executable.

• Meanwhile, the Compile builds CI target runs teamcity-compile-builds.sh which in turn runs pkg/cmd/compile-builds which in turn runs release.MakeReleases with "all targets" as arguments, which in turns run builder.sh mkrelease on each supported target.

  In other words, the "Compile builds" CI target builds 3+ different crdb executables, including the linux-gnu one.

• Meanwhile, the verify archive CI target runs teamcity-verify-archive.sh which in turns create a tar archive of the source directory, then extracts it, then checks that make clean && make install works on it.

  In other words, the "verify archive" CI target builds its own linux crdb executable.

We could shave some time on the entire CI run (as well as disk space, time collecting artifacts etc) by only building the linux executable just one time as pre-requisite for other CI targets.

I would recommend using verify archive as a starting point, since we also want to assert that the code can be built from a downloadable source archive.

cc @jlinder for triage

Epic CRDB-10439

Jira issue: CRDB-6973

[knz]

Another reason why it would be desirable to have a build target as prereq for the other targets: in case the source doesn't build, we would only hog 1 agent to report that error and abort the entire CI pipeline, instead of spawning 8 agents upfront.

[knz]

Here's a sketch of a dep tree:

1. new variant of "verify archive" which extracts the source then runs builder.sh mkrelease linux-gnu (instead of make install), then stores the produced binary somewhere where the other targets can find it. Possibly also runs make install separately to ensure the binary produced by mkrelease can be installed.
2. depending on (1): "compile builds" but exclude the linux binary, since it was already built in (1)
3. depending on (1): roachtest, and remove the "make build" step from the script
4. depending on (1): acceptance, and remove the "mkrelease" step from the script

Then we can remove the "Build test binary" step altogether.

Since the CI config change and the script updates are not atomic, we'll need to have a "transition phase" where the scripts work in both configurations. For this we can modify the scripts to expect a pre-built executable in a fixed path, and use if ! test -e path/to/executable; then ...build...; fi to build it on demand if it's not there yet.

[github-actions]

We have marked this issue as stale because it has been inactive for
18 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it in
10 days to keep the issue queue tidy. Thank you for your contribution
to CockroachDB!