Improve staging-test-with-rebase performance

[eloquence]

staging-test-with-rebase is our slowest CI job; it currently typically clocks in at just under 1 hour. With parallel job execution, that makes it our leading CI performance optimization candidate. In cases where it flakes out, it also incurs the highest time penalty on code review/integration.

The scope of this issue is to systematically examine each part of a typical job run, and target low-hanging fruit improvements to reduce the job execution time.

[eloquence]

@conorsch and @emkll have expressed interest in working on this during the 5/5-5/19 sprint. Conor mentioned package updates as one potential area of optimization.

[conorsch]

Update: I have not investigated here, although I remain optimistic that simply updating the boxes (and thereby their apt lists) will shave at least a few minutes off each staging run. @maeve-fpf mentioned she'd be interested in tackling this one, so I'll recommend we carry forward onto next sprint, and I can advise and/or pair as necessary.

[maeve-fpf]

I've built a new image; ID is here: https://github.com/freedomofpress/securedrop/compare/update-ci-nested-virt-buster

[conorsch]

Good news, that new image is already passing in CI: https://app.circleci.com/pipelines/github/freedomofpress/securedrop?branch=update-ci-nested-virt-buster @maeve-fpf Would you mind opening a PR? I'll handle review (which is mostly confirming that CI passes).

[cfm]

With staging-test-with-rebase now running only nightly after #6063, is further optimization of the job itself still a priority?

[eloquence]

Closing this for now, running it nightly has definitely greatly reduced the friction. We can revisit but currently resilience of this job (#6112, #6085, #6052, etc.) and possibly restoring a custom branch filter to ensure it runs on all PRs that target a release branch (but whose branch may or may not begin with stg-) seem higher priority.