Cache AWS CLI install in CI (#7472)

### Problem
We freshly install the AWS CLI program on nearly every CI shard, even though the program does not change between runs. This adds about 30 seconds to most shards.

### Solution
Modify the `install_aws_cli_for_ci.sh` script to first check if the program is installed, and only install if it is not. Also add the install folder to Travis's cache.

Note that Travis does not cache symlinks, only the actual files. So, we must re-symlink the bin every time we run the script.

### Result
Overall CI time reduced by ~13.5 minutes. Each individual shard is now an average of ~19.4 seconds faster.

See https://docs.google.com/spreadsheets/d/1KWzpHnJzUpEZwx7_m-Kt_hZ-QNYFnzoFRPbi3dX0t6c/edit#gid=60516952 for a per-shard analysis of the old time and the new time with this cache.

.travis.yml

@@ -32,6 +32,7 @@ env:
     # PYENV_ROOT on those shards, or their Python will no longer work.
     - PYENV_ROOT="${PYENV_ROOT:-${HOME}/.pants_pyenv}"
     - PATH="${PYENV_ROOT}/shims:${PATH}"
+    - AWS_CLI_ROOT="${HOME}/.aws_cli"
 
 # Stages are documented here: https://docs.travis-ci.com/user/build-stages
 stages:
@@ -66,6 +67,7 @@ native_engine_cache_config: &native_engine_cache_config
     # TODO: Figure out why we have such large caches (2-7GB) and try to trim them.
     timeout: 500
     directories:
+      - ${AWS_CLI_ROOT}
       - ${PYENV_ROOT}
       - ${HOME}/.cache/pants/rust/cargo
       - build-support/pants_dev_deps.py27.venv
@@ -97,6 +99,7 @@ pants_run_cache_config: &pants_run_cache_config
     # TODO: Figure out why we have such large caches (2-7GB) and try to trim them.
     timeout: 500
     directories:
+      - ${AWS_CLI_ROOT}
       - ${PYENV_ROOT}
       - ${HOME}/.cache/pants/tools
       - ${HOME}/.cache/pants/zinc

build-support/bin/install_aws_cli_for_ci.sh

@@ -9,15 +9,31 @@ set -euo pipefail
 # to use, which slows down CI jobs significantly. This is also the installation method recommended
 # by AWS, see https://docs.aws.amazon.com/cli/latest/userguide/install-bundle.html.
 
-TMPDIR=$(mktemp -d)
+source build-support/common.sh
 
-pushd ${TMPDIR}
+if [[ -z "${AWS_CLI_ROOT:+''}" ]]; then
+  die "Caller of the script must set the env var AWS_CLI_ROOT."
+fi
+AWS_CLI_BIN="${AWS_CLI_ROOT}/bin/aws"
 
-curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
-unzip awscli-bundle.zip
-sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws
+# We first check if AWS CLI is already installed thanks to Travis's cache.
+if [[ ! -x "${AWS_CLI_BIN}" ]]; then
 
-popd
+  TMPDIR=$(mktemp -d)
+
+  pushd ${TMPDIR}
+
+  curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
+  unzip awscli-bundle.zip
+  ./awscli-bundle/install --install-dir "${AWS_CLI_ROOT}"
+
+  popd
+
+fi
+
+# Travis does not cache symlinks (https://docs.travis-ci.com/user/caching/), so we create
+# the symlink ourselves everytime to ensure `aws` is discoverable globally.
+sudo ln -s "${AWS_CLI_BIN}" /usr/local/bin/aws
 
 # Multipart operations aren't supported for anonymous users, so we set the
 # threshold high to avoid them being used automatically by the aws cli.

build-support/travis/travis.yml.mustache

@@ -25,6 +25,7 @@ env:
     # PYENV_ROOT on those shards, or their Python will no longer work.
     - PYENV_ROOT="${PYENV_ROOT:-${HOME}/.pants_pyenv}"
     - PATH="${PYENV_ROOT}/shims:${PATH}"
+    - AWS_CLI_ROOT="${HOME}/.aws_cli"
 
 # Stages are documented here: https://docs.travis-ci.com/user/build-stages
 stages:
@@ -59,6 +60,7 @@ native_engine_cache_config: &native_engine_cache_config
     # TODO: Figure out why we have such large caches (2-7GB) and try to trim them.
     timeout: 500
     directories:
+      - ${AWS_CLI_ROOT}
       - ${PYENV_ROOT}
       - ${HOME}/.cache/pants/rust/cargo
       - build-support/pants_dev_deps.py27.venv
@@ -90,6 +92,7 @@ pants_run_cache_config: &pants_run_cache_config
     # TODO: Figure out why we have such large caches (2-7GB) and try to trim them.
     timeout: 500
     directories:
+      - ${AWS_CLI_ROOT}
       - ${PYENV_ROOT}
       - ${HOME}/.cache/pants/tools
       - ${HOME}/.cache/pants/zinc