Broader CI bundler cache

[marcotc]

Depends on #821

This PR adds an extra caching layer to the local gem cache, /usr/local/bundle, that is able to be re-utilized across different CI builds.

This PR does not affect the behavior of bundle install or appraisal install, or the gems they select to install. This only pre-populates the local cache.

The caching key needs to only depends on information available before bundle install and appraisal install are run. This rules out the checksum of gem lock files.

With this in mind, this caching key was chosen:

bundle-{{ .Environment.CIRCLE_CACHE_VERSION }}-<<parameters.ruby_version>>-{{ checksum "lib/ddtrace/version.rb" }}-{{ .Branch }}-{{ checksum "Gemfile" }}-{{ checksum "Appraisals" }}-{{ checksum "ddtrace.gemspec" }}

It varies by: Ruby version, dd-trace-rb version, git branch name, contents of Gemfile, Appraisals, and ddtrace.gemspec.

It can also be arbitrarily invalidated at any point by setting a different CIRCLE_CACHE_VERSION environment variable in our CI pipeline.

When restoring this cache key, we try a few combinations:

- restore_cache:
 keys:
  - bundle-{{ .Environment.CIRCLE_CACHE_VERSION }}-<<parameters.ruby_version>>-{{ checksum "lib/ddtrace/version.rb" }}-{{ .Branch }}-{{ checksum "Gemfile" }}-{{ checksum "Appraisals" }}-{{ checksum "ddtrace.gemspec" }}
  - bundle-{{ .Environment.CIRCLE_CACHE_VERSION }}-<<parameters.ruby_version>>-{{ checksum "lib/ddtrace/version.rb" }}-{{ .Branch }}-
  - bundle-{{ .Environment.CIRCLE_CACHE_VERSION }}-<<parameters.ruby_version>>-{{ checksum "lib/ddtrace/version.rb" }}

1. With every parameter matching.
   1. Speeds up workflow reruns.
2. With the same Ruby version, dd-trace-rb version, and git branch name.
   1. Speeds up successive runs on branches that change Gemfile and Appraisals files. Caching content is still good, as all unchanged dependencies are likely the same.
3. With the same Ruby version and dd-trace-rb version.
   1. Speeds up first build of a branch. Caching content should be valuable, as we still ensure we use the latest cache for the current dd-trace-rb version.

This fallback strategy guarantees the only build that runs without any gem caching is a version bump (or manual cache invalidation through CIRCLE_CACHE_VERSION).

For reference, the current cache persistency layer we have is used to propagate installed dependencies to downstream jobs (e.g CI test steps,) and speeds up CI workflow reruns.
It is very precisely matched to the current build environment, depending on the checksum of all gemlock files.
Given it depends on gemlocks files, that are only available after gem installation, it cannot be used to speed up such operation.

Some results

Before:

After:

(~ 1-2 minute reduction)

[delner]

Thanks for this @marcotc! I'll give this a review in a bit, but those results look promising!

[marcotc]

I added checksum ".circleci/bundle_checksum" to the caching key, to allow for cache busting when gem lock files change.

[delner]

👍