Equinix

[johnalotoski]

Improvements

• Nomad is bumped to v1.4.3 -- see release notes here

• Adds capability for an "AWS Extended" (awsExt) cluster -- so that the bitte cluster can add, as a first type, Equinix bare metal machines.

• This PR provides TF Equinix integration and AWS extended machine cluster integration with Equinix bare metal resources.

• Network communication between the two cloud networks for a unified cloud network would happen via a network overlay such as Ziti, wireguard, etc, which is outside the scope of this PR.

• For clusters not interested in using this feature, this PR should have no impact on operations and no special migration is required (other than the Nomad version bump, notes below).

Migration

• The bump to Nomad 1.4.3 in this PR will require a metal deployment to all bitte machines to update the Nomad version. See Nomad upgrade docs for further details.

• If utilizing an AWS extended cluster type for bare metal provider integration from Equinix is not desired, no further migration is required.

Utilizing an AWS Extended cluster type

• For those bitte clusters which are "aws" clusterType and wish to utilize additional Equinix prem machines in their bitte cluster, the following is an outline of migration steps:

• Ensure that upon updating your cluster's bitte pin to this PRs' commit, that all migration steps from previous PRs have been completed successfully.

• Make sure that all TF workspaces are up to date and applied and there is no diff remaining. This is because changing from "aws" to "awsExt" cluster type will cause a number of TF changes in the next plan/apply cycle and separating them from any diff which may already exist that you don't want applied will be tedious.

• Modify the cluster type to "awsExt" for AWS extended. The declaration is typically found in nix/metal/bitteProfile/default.nix:

  cluster.infraType = "awsExt";`

• Run nix run .#clusters.$CLUSTER.tf.equinix.plan and requirements for equinix TF resources will be printed; read them.

• Setup a sops encrypted equinix project json file per the requirement instructions seen above:

  • Create $relEncryptedFolder/equinix.json with decrypted form of:

    {
      "<project1>": "<project1Id>",
      "<project2>": "<project2Id>",
      ...
    }

• Initialize a new TF workspace for equinix resources:

  nix run .#clusters.$CLUSTER.tf.<TF_WORKSPACE>.initLocal

• Create secrets/equinix-secret.sh which is NOT git committed and contains the project Equinix API metal token
  • Currently, this PR will allow for 1 equinix project per world repo:

export METAL_AUTH_TOKEN=$PROJECT_API_TOKEN

• Modify your .envrc to add the following lines:

# Equinix deployment key
source_env_if_exists secrets/equinix-secret.sh
env_vars_required METAL_AUTH_TOKEN

• TF plan apply the core workspace for new and modified IAM resources

• bitte deploy core-1 in order to execute a vault-setup systemd service modification to the bound_iam_principal_arn of the auth/aws/role/$CLUSTER-client role

• Verify with an admin vault token that an awsExt-bitte-system bound principal now shows for the cluster:

> vault read auth/aws/role/$CLUSTER-client -format=json | jq '.data.bound_iam_principal_arn'
[
  "arn:aws:iam::$ACC:role/core-$CLUSTER-client",
  "arn:aws:iam::$ACC:user/awsExt-bitte-system"
]

• Declare equinix resources

• Run a TF equinix workspace plan apply

• Apply network overlays as needed

Testing

• Tested and deployed to bitte-world, ci-world