CI: Add caching and cleanup all workflows

[sieniven]

Summary

1. Caching:

• Add caching of build tree for depends, rust and cpp builds when building defid binaries using pre-warmed container (ain-builder) and using docker build.
• Dev build workflow will save the docker build tree caches. Dev build will attempt to find a matching cache key scoped to the current branch, and if a cache miss occur, it will default to getting the base branch's cache (master) and cache the build trees scoped to that branch.
• test-ethlibs workflow will save the native build tree caches (workaround for known issues) in build job. All other test workflows will restore the caches saved if cache hit.
• Restoring of the cached build trees is implemented into every CI pipeline that requires defi binaries.
• Implementation of ccache into incremental cpp builds.
• Note that test-sync pipeline will not introduce caching as we will want to do a full clean build of the defid binaries natively.

2. Docker builds

• Enable docker packaging of build directories for caching. (see known issue as to why we store docker build caches)
• Docker incremental builds are now enabled with multistage build by separating target build stages in our build dev pipeline.
• builder target image: building defi binaries and deploying the release directory and build caches.
• defi target image: specific for linux-x64, for deployment of defid image to be pushed into the registry.

2. Cleanup CI workflows

• Refactoring to implement matrices for build strategies when running multi-arch builds for dev, staging and release to clean up repeated code blocks.
• All test workflows to use pre-warmed container (ain-builder).
• Fixes to rust lint workflow - no more lint failures with CI reflecting a pass. (note that this has been cherry picked into PR clippy: Throw error for warnings #2152)
• Implementation and refactoring of signing release binary packages into make.sh from CI workflows into sign() function.
• Refactor getting package name into package() function in make.sh.
• Packaging includes the default zip pipeline for win-x64 binaries.

Pending future work

• Currently ccache does not cache the rust builds. We cache our rust build directory which helps with caching dependency packages quickly but there are no incremental builds for our rust source code. a To do in the future will be to upgrade ccache to sccache, which will be able to provide incremental builds to the rust side as well.

Known issues

1. Caches are unable to be generate cache hits when running in the pre-warmed builder container. Refer to the e2e test CI using ain builder in run (https://github.com/DeFiCh/ain/actions/runs/5385145166/jobs/9773854907) vs running natively in https://github.com/DeFiCh/ain/actions/runs/5385121002).
2. Build caches saved in docker builds are unable to build when restored natively. The cpp build throws a target error due to the incorrect file paths. /work/* (docker builds) vs /home/runner/work/* (native)
3. Implementation of docker builds into test-sync CI workflow is discontinued due to as built docker images on self-hosted CI runners are not pruned after checking out.

Implications

• Storage

  • Database reindex required
  • Database reindex optional
  • Database reindex not required
  • None

• Consensus

  • Network upgrade required
  • Includes backward compatible changes
  • Includes consensus workarounds
  • Includes consensus refactors
  • None

[sieniven]

Caching the cpp build directory poses issues since git cloning the repository do not keep track of timestamping, thus re-building all of the cpp files. However, caching the depends and rust build directories improves the build workflow time significantly, from ~1hour to 30mins.

[sieniven]

Closing the dangling PR. Fixes from #2718 and #2721 resolves the CI improvements that this PR is trying to achieve.