jabos

Jabos attempts to be a fully automated K8s GitOps framework.

This is WIP - any comments, requests or issues would be welcome! please use this link

TL;DR - What does that mean?

What you need to do?

1. Installing Jabos into your K8s cluster using
2. Setting up the K8s objects for your
   • Git Repository
   • Docker images
   • Manifest folder

What happens next?

1. Any new commits would be picked up from Git automatically
2. Docker images would get build from new commits and pushed automatically
3. New manifest versions would be deployed automatically

Goals

• Automate all steps to deploy from Git repositories into a K8s cluster
• No GUI requiring manual human intervention
• Git as a single source of truth
• Pull only model from within runtime environments.
• Isolation of build environment from runtime environment
• Idempotent builds
• Preview environment + Pre-deploy integration testing

Guidelines

• Minimal set of tools/technologies as pre-requisites/installs
• Stay as tech-stack agnostic as possible
• Minimal steps to install or setup development environment. Automated as possible.

Installation

Video version:

1. Minikube setup
2. Jabos and prerequisites

Instructions:

1. Make sure Metacontroller is installed on your cluster. Find instructions here
2. Optionally install kube-prometheus-stack to expose metrics from jabos
3. Optionally install grafana-dashboard-operator to setup grafana dashboards for jabos
4. Create a namespace for jabos to use. E.g. kubectl create namespace jabos. Use the same namespace with the next command.
5. Run kubectl apply -n <NAMESPACE> -f https://github.com/srfrnk/jabos/releases/latest/download/jabos-manifests.yaml

Usage

Jabos uses CRDs in order for users to define a codebase and how to build and deploy that.

See API Docs here

Video version:

1. Configure CRDs
2. Push changes and trigger builds

Resource Status

All resources show current status using Status Sub-resource and Event Resources. These can be viewed as with any K8s resource. i.e. kubectl describe git-repositories.jabos.io

GitRepository resources have a Syncing condition in the status. If it becomes False an Event will describe the error. They also have a Latest Commit(latestCommit) status containing the latest git commit id found.

DockerImage and ***Manifest resources have a Synced condition in the status. If it becomes False an Event will describe the error. They also have a Latest Commit(latestCommit) status containing the latest git commit id found and a Built Commit(builtCommit) status containing the git commit id last built.

Jsonnet example

Create a file example.jsonnet:

function(latestCommitId) {
  apiVersion: 'apps/v1',
  kind: 'Deployment',
  metadata: {
    name: 'test-deployment',
    labels: {
      app: 'test-deployment',
    },
  },
  spec: {
    replicas: 1,
    selector: {
      matchLabels: {
        app: 'test-deployment',
      },
    },
    template: {
      metadata: {
        labels: {
          app: 'test-deployment',
        },
      },
      spec: {
        containers: [
          {
            name: 'test-deployment',
            image: 'registry.kube-system:80/example-image:' + latestCommitId,
          },
        ],
      },
    },
  },
}

Git Repository Authentication

GitHub

Using SSH Keys

1. Create an SSH key and add it to GitHub - Optionally skip the "adding it to the ssh-agent" section.
2. Create a secret with the passphrase and key created in the previous step. (i.e. kubectl create secret generic -n example-env first-repo-private --from-file=git_ssh_passphrase=./build/passphrase --from-file=git_ssh_key=./build/key)
3. Add an ssh property to each applicable GitRepository object to point to the secret.

Image Registry Authentication

Docker Hub

1. Obtain your Docker Hub username.
2. Obtain your Docker Hub password or access token.
3. Create secret with the credentials. (i.e. kubectl create secret generic -n example-env docker-hub --from-file=docker_hub_username=./build/docker_hub_username --from-file=docker_hub_password=./build/docker_hub_password)
4. Add a dockerHub property to any applicable DockerImage object to point to the secret.

GCP (GCR and Artifact Registry)

1. Obtain a Service Account with the required permissions.
2. Obtain the Service Account JSON key.
3. Create secret with the JSON key. (i.e. kubectl create secret generic -n example-env gcp --from-file=gcp_service_account.json=./build/gcp_service_account.json)
4. Add a gcp property to any applicable DockerImage object to point to the secret.

AWS (ECR)

1. Obtain an Access Key with the required permissions.
2. Obtain the Access key ID and Secret Access Key.
3. Create secret with these credentials. (i.e. kubectl create secret generic -n example-env aws --from-file=aws_access_key_id=./build/aws_access_key_id --from-file=aws_secret_access_key=./build/aws_secret_access_key)
4. Add a aws property to any applicable DockerImage object to point to the secret.

Note: You can use instance roles instead when pushing to ECR from a EC2 instance or from EKS, by configuring the instance role permissions.

Metrics

All metrics are exported into Prometheus using the ServiceMonitor API by kube-prometheus-stack. To otherwise configure Prometheus to collect the metrics you need to point it to 'OPERATOR_POD_IP:3000/metrics'.

All metrics exported are prefixed with jabos_operator_. Numerous metrics are exported most of them describe nodsjs and expresjs operations. Removed due to security audit fails

Important metrics for the operation of Jabos are:

# HELP jabos_operator_latest_commit_changed new "latest commit" detected for git repository
# TYPE jabos_operator_latest_commit_changed counter

# HELP jabos_operator_docker_image_build_trigger new build triggered for a docker image
# TYPE jabos_operator_docker_image_build_trigger counter

# HELP jabos_operator_jsonnet_manifests_build_trigger new build triggered for jsonnet manifests
# TYPE jabos_operator_jsonnet_manifests_build_trigger counter

# HELP jabos_operator_git_repository_updater_start GitRepositoryUpdater start
# TYPE jabos_operator_git_repository_updater_start counter

# HELP jabos_operator_git_repository_updater_end GitRepositoryUpdater end
# TYPE jabos_operator_git_repository_updater_end counter

# HELP jabos_operator_git_repository_updater_duration GitRepositoryUpdater duration
# TYPE jabos_operator_git_repository_updater_duration gauge

# HELP jabos_operator_docker_image_builder_start DockerImageBuilder start
# TYPE jabos_operator_docker_image_builder_start counter

# HELP jabos_operator_docker_image_builder_end DockerImageBuilder end
# TYPE jabos_operator_docker_image_builder_end counter

# HELP jabos_operator_docker_image_builder_duration DockerImageBuilder duration
# TYPE jabos_operator_docker_image_builder_duration gauge

# HELP jabos_operator_jsonnet_manifests_builder_start JsonnetManifestsBuilder start
# TYPE jabos_operator_jsonnet_manifests_builder_start counter

# HELP jabos_operator_jsonnet_manifests_builder_end JsonnetManifestsBuilder end
# TYPE jabos_operator_jsonnet_manifests_builder_end counter

# HELP jabos_operator_jsonnet_manifests_builder_duration JsonnetManifestsBuilder duration
# TYPE jabos_operator_jsonnet_manifests_builder_duration gauge

Use Cases

Diagrams for supported and future planned use-cases are here

Image Reuse

Build images in DEV/QA only and reuse when commit is promoted to other environments. To mark a DockerImage for reuse of an image built in another environnement add build: false to the spec.

Security

Jabos images and manifest are being scanned by CodeQL and Snyk as part of the release process using GitHub Actions.

Jabos makes no attempt at protecting applications, networks, disks from malicious access. It is the responsibility of the user to put in place such measures.

Jabos should always be contained inside a dedicated namespace to reduce risk to other systems.

Special attention must be given to the Jabos docker image builder pods which use Kaniko. At this time it is required for Kaniko to be executed with root user and with a writable file system. This known limitation is a low risk as these pods have a very short life span... however it does pose a risk especially when the code pulled from a Git repository may contain vulnerabilities.

It is advisable to always scan all code which is pulled by Jabos from Git!

It is advisable to use NetworkPolicy and other methods to ensure any egress from docker image builder pods is limited to what is required by your images to build!

Security Overview

The scan results can be found here

As of version 1.x there are no known vulnerabilities.

Reporting a Vulnerability

Create an issue here. Please add a security label for quicker response.

Development

Prerequisites

1. make installed (Depending on your OS - start here)
2. docker installed (To install see here)
3. minikube installed (To install minikube see this)
4. NodeJS installed (To install NodeJS see this)
5. Typescript development tools installed npm install -g ts-node typescript '@types/node'
6. GNU Parallel installed for your OS. For Debian based you can use sudo apt-get install parallel.
7. K9s installed (To install see here). For automated port forwarding set K9s configuration with scanForAutoPf: true. Make sure K9s version supports the feature (derailed/k9s#1498).

Environment Setup

1. Clone repo: git clone git@github.com:srfrnk/jabos.git (or using HTTPS/GitHub CLI - see instructions here)
2. CD into folder
3. Start a minikube cluster minikube start
4. Run make setup once
5. Run make build after each code change
6. To deploy examples
   1. Locally clone jabos-examples repo
   2. Follow instructions from the README file in the cloned folder
   3. The examples would be deployed into namespace example-env

Credits

• Jabos uses the kaniko project to build docker images inside the kubernetes cluster.
• Jabos uses yq to parse and update yaml and json data.
• Jabos uses jsonnet to process jsonnet templates and create K8s manifests.
• Jabos uses minikube for local development
• Jabos uses metacontroller to control K8s operators.
• Jabos uses efk-stack-helm for local centralized logging.
• Jabos uses kube-prometheus-stack for local monitoring and alerting.
• Jabos uses GNU Parallel for local port-forwarding to multiple services
• Jabos uses expressjs as the web server to run the operator
• Removed due to security audit fails - Jabos uses express-prometheus-middleware to export basic metrics to prometheus