Sandbox

Table of Contents

• Build
• sandbox-api
  • Installation
  • Setup local development environment
• sandbox-replicate
  • Push lambda
• sandbox-metrics
  • Deploy Metrics Prometheus
• Create AWS sandboxes
• Conan - Sandbox Cleanup Daemon
• Add a new OCP shared cluster for OcpSandbox

This repository is a consolidated codebase for everything related to sandboxes for the Red Hat Demo Platform. It contains:

• sandbox-api: The Backend API for all things sandbox.

• sandbox-issue-jwt: CLI to generate a new signed JWT login token

• sandbox-list: Interact with the AWS Sandbox DB in a read-only way.

• sandbox-metrics: Prometheus Endpoint exposing metrics.

• sandbox-replicate: AWS Lambda function to replicate changes from DynamoDB to a Postgres database.

• sandbox-rotate-vault: Program to reencrypt the IAM keys using a new secret key (ansible-vault, AES256)

Build

Build all binaries

You will need to have 'go' setup on your machine. Please use version 1.22 onwards

make

Build separately

make sandbox-api
make sandbox-list
make sandbox-metrics
make sandbox-replicate
make sandbox-issue-jwt
make sandbox-rotate-vault

sandbox-api

Installation

Create a secret file and run helm.

secrets.yaml

sandbox_api_secrets:
  database:
    url: postgres://...
    dynamodb_aws_access_key_id: ...
    dynamodb_aws_secret_access_key: ...
  # Secret to generate and validate JWT tokens
  auth:
    jwt_auth_secret: ...
  # ansible-vault (AES256) key to encrypt the secret in the DB
  vault:
    vault_secret: ...
  # AWS usser that can assume role into the accounts
  aws:
    assume_aws_access_key_id: ...
    assume_aws_secret_access_key: ...

Install chart

helm install  -f secrets.yaml sandbox-api deploy/helm-api/

Upgrade chart

helm upgrade  -f secrets.yaml sandbox-api deploy/helm-api/

To initialize or update the postgresql schema, use the following:

Run the DB migration

oc run admin-$$  \
--image=quay.io/rhpds/sandbox-admin:latest -i -t \
--restart=Never --rm -- /bin/bash (1)

DATABASE_URL=postgres://postgres:PASSWORD@RDS_ADDRESS.us-west-2.rds.amazonaws.com:5432/sandbox_api_dev?sslmode=require

# git clone https://github.com/rhpds/sandbox.git
# cd sandbox
# migrate github://rhpds/sandbox/db/migrations#VERSION -database $DATABASE_URL up

# For example
migrate \
  -source github://rhpds/sandbox/db/migrations#main \
  -database $DATABASE_URL up

1. Use the rhpds/sandbox-admin image which contains all the necessary binaries and tools.

Bootstrap an admin login token

oc run admin-$$ --image=quay.io/rhpds/sandbox-admin:latest -i -t --restart=Never --rm -- /bin/bash

export DATABASE_URL=postgres://postgres:PASSWORD@RDS_ADDRESS.us-west-2.rds.amazonaws.com:5432/sandbox_api_dev?sslmode=require

./sandbox-issue-jwt
[root@admin sandbox]# ./sandbox-issue-jwt
JWT Auth secret: Enter Claims in the JSON format:
for example: {"kind": "login", "name": "gucore", "role": "admin"}
{"kind": "login", "name": "gucore", "role": "admin"}
token:
[TOKEN HERE]

Create an access token

oc run admin-$$ --image=quay.io/rhpds/sandbox-admin:latest -i -t --restart=Never --rm -- /bin/bash

logintoken=[TOKEN]

curl -H "Authorization: Bearer ${logintoken}" sandbox-api:8080/api/v1/login

[ACCESS TOKEN]

token=[ACCESS TOKEN]
# check access
curl -H "Authorization: Bearer ${token}" sandbox-api:8080/api/v1/health

Setup local development environment

All filed used for the local development environment are prefixed by .dev and are ignored by Git, see .gitignore

make run-local-pg  # run postgresql locally in a Container
make migrate  # Run the DB migrations to setup the db schema
# Set the following secrets, notice the heading space ' ' to avoid shell history
# IAM secrets to access AWS sandboxes
 export ASSUMEROLE_AWS_SECRET_ACCESS_KEY=...
 export ASSUMEROLE_AWS_ACCESS_KEY_ID=...
# IAM secrets to access dynamodb table that contains info of the AWS sandboxes
 export AWS_ACCESS_KEY_ID=...
 export AWS_SECRET_ACCESS_KEY=...
# AES key to encrypt sensible data in the different databases
# If you're using the dynamoDB dev database for AWS sandboxes (which you probably are)
# Then this needs to match the one in use on the DEV environment
 export VAULT_SECRET=...

make tokens # issue some JWT token for access
make run-api # (1)
air # (2)

1. When iterating, you will be stopping and relaunching this step

2. You can use cosmtrek/air instead. That will watch local files and rebuild + launch the API automatically if any changes are made.

sandbox-replicate

The role of the lambda function is to replicate any changes made to the dynamoDB table into a postgresql database.

Push lambda

export AWS_PROFILE=infra-dev
make push-lambda

That will:

1. Create a role, a policy and a lambda function

2. Attach the policy to the role and the role to the lambda function

3. Push the updated 'build/sandbox-replicate' binary to the lambda function

sandbox-metrics

Deploy Metrics Prometheus

1. clone this repository

   git clone --depth 1 https://github.com/rhpds/sandbox sandbox

2. If it doesn’t exist yet, create an IAM user in AWS to read-only access to dynamoDB

3. Create the secret file containing the key for the IAM user that has read-only access to DynamoDB

   aws_sandbox_readonly.yaml

   aws_sandbox_metrics_secrets:
     readonly:
       aws_access_key_id: ...
       aws_secret_access_key: ...

4. Install the helm chart

   helm install  sandbox-metrics sandbox/deploy/helm-metrics/ -f aws_sandbox_readonly.yaml

   Output should look like:

   NAME: sandbox-metrics
   LAST DEPLOYED: Thu Jun 17 09:30:04 2021
   NAMESPACE: user-gucore-redhat-com
   STATUS: deployed
   REVISION: 1
   TEST SUITE: None

Create AWS sandboxes

Use ansible playbooks.

Conan - Sandbox Cleanup Daemon

See conan.

Add a new OCP shared cluster for OcpSandbox

In order for the sandbox API to be able to talk to the OCP shared cluster, you need to create a new OcpSharedClusterConfiguration.

That configuration holds the information needed to authenticate to the cluster and the additional variables that will be passed to the deployer when a sandbox is scheduled on that cluster.

There is 2 ways to authenticate to the cluster: . Using a service Bearer Token . Using a kubeconfig file

Here we’ll be describing the first method.

On the cluster, create an admin service account

oc create serviceaccount sandbox-api-admin -n kube-system
oc create clusterrolebinding sandbox-api-admin -n kube-system --clusterrole=cluster-admin --serviceaccount=kube-system:sandbox-api-admin
oc create token sandbox-api-admin -n kube-system --duration=4294967296s

Then create a JSON file describing the OcpSharedClusterConfiguration. For example:

clustername.json

{
    "name": "clustername", (1)
    "api_url": "https://api...:6443", (2)
    "ingress_domain": "apps...",
    "additional_vars": {
        ... (3)
    },
    "annotations":
    {
        "virt":"no",
        "cloud":"aws",
        "purpose":"dev"
    }, (4)
    "token":"..."  (5)
}

1. The name of the cluster as it will be saved in the database

2. The API URL of the cluster

3. Additional variables that will be passed to the deployer if that cluster is elected when scheduling a sandbox

4. Annotations are used to filter the clusters when ordering. For example, if you want to deploy a sandbox on a cluster that is not in production, you can use the purpose annotation to filter out the production clusters. That is done in agnosticV using the meta.sandboxes[].cloud_selector key

5. The token is the token created in the previous step. It is used to authenticate the Sandbox API to the cluster.

Then use hurl and ./tools/ocp_shared_cluster_configuration_create.hurl

hurl --variable login_token_admin=$admintoken \
--file-root . \
--variable host=SANDBOX_API_ADRESS \  (1)
--variable ocp_cluster_def=clustername.json \
./tools/ocp_shared_cluster_configuration_create.hurl


# or with curl directly
accesstoken=$(curl -s --header "Authorization: Bearer $admintoken"     --header 'Content-Type: application/json'     'https://SANDBOX_API_ADDRESS/api/v1/login'|jq -r .access_token)

curl --header "Authorization: Bearer $accesstoken" \
  --header 'Content-Type: application/json' \
  --data-binary '@./clustername.json' \
  'https://SANDBOX_API_ADDRESS/api/v1/ocp-shared-cluster-configurations'

1. Replace SANDBOX_API_ADDRESS with the address of the Sandbox API