armada-operator

• armada-operator
  • How it works
  • Installing Armada Operator
    • Prerequisites
    • Quickstart
    • Manual setup
    • Installing Armada
    • Applying PriorityClass
    • Installing armadactl
  • Using armadactl
  • Migrating
    • Migrating to v0.11 and beyond
  • Further documentation
  • Supported Armada component configurations
  • Developing locally
    • Starting a development cluster
    • Building and installing a development cluster
    • Running the Armada Operator locally
    • Stopping the development cluster
  • Contributing
  • Testing
  • FAQ
    • kube-prometheus-stack is not installing
  • License

Armada Operator is a Kubernetes-native Operator for simpler installation of Armada. This project introduces Custom Resource Definitions (CRDs) for Armada services and provides a controller to manage the lifecycle of these services.

How it works

This project aims to follow the Kubernetes Operator pattern.

It uses Controllers, which provide a reconcile function responsible for synchronising resources until the desired state is reached on the cluster.

Installing Armada Operator

Prerequisites

Before you start, make sure you have the following installed:

• kubectl
• helm
• Go v1.21+ - required for the Quickstart make kind-all target

Quickstart

To start immediately with Armada and the Operator, run the following make target:

make kind-all

This command runs the following make targets:

• kind-create-cluster - creates a kind cluster using the configuration from hack/kind-config.yaml
• install-and-wait-cert-manager - installs cert-manager and waits for it to be ready
• helm-repos - adds necessary Helm repositories for external dependencies
• helm-install - installs the latest released Armada Operator using Helm from the gresearch Helm registry
• install-armada-deps - installs required Armada dependencies such as Prometheus, PostgreSQL, Pulsar and Redis using Helm
• wait-for-armada-deps - waits for all Armada dependencies to be ready
• create-armada-namespace - creates the Armada namespace armada
• apply-armada-crs - applies the CRDs of all Armada components using kubectl
• create-armadactl-config - creates the armadactl configuration (~/.armadactl.yaml file, pointing to localhost:30002 as per the quickstart guide
• apply-default-priority-class - applies the default priority class required by Armada for all jobs
• get-armadactl - downloads the armadactl binary for interacting with the Armada API

Manual setup

You’ll need a Kubernetes cluster to run the Armada Operator. You can use Kubernetes in Docker (KIND) to run a local cluster for testing, or you can run against a remote cluster.

To install the latest release of Armada Operator in your cluster, run the following commands:

# Add the G-Research Helm repository
helm repo add gresearch https://g-research.github.io/charts
# Install the Armada Operator
helm install armada-operator gresearch/armada-operator --namespace armada-system --create-namespace

This installs the Armada Operator in the armada-system namespace.

Installing Armada

When installing Armada, you first need to install the Armada external dependencies:

• kube-prometheus-stack
• Pulsar
• Redis
• Postgres

You can install the required Armada dependencies either manually or by running the following make target:

make install-armada-deps

Note: You need to wait for all dependencies to be running before proceeding to the next step.

Then run the following command to install Armada components using the CRs provided in dev/quickstart/armada-crs.yaml:

# Create armada namespace
kubectl create namespace armada
# Install Armada components
kubectl apply -n armada -f dev/quickstart/armada-crs.yaml

dev/quickstart/armada-crs.yaml uses NodePort Services for exposing Armada components For example, the Lookout UI at 30000, the Armada gRPC API at 30001 and the Armada REST API at 30002.

Once every Armada service is deployed, you should have a fully functional installation of Armada.

Note: This example, created for use with the kind config defined at hack/kind-config.yaml, is for demonstration purposes only. Do not use it in production.

Applying PriorityClass

Armada also requires a default PriorityClass to be set for all jobs. You can apply the default PriorityClass by running the following command:

kubectl apply -f dev/quickstart/priority-class.yaml

Installing armadactl

To install armadactl, run the following make target:

make get-armadactl

Alternatively, download it from the GitHub release page.

Create a configuration file for armadactl by running the following command:

make create-armadactl-config

Alternatively, create a configuration file manually in ~/.armadactl.yaml with the following content:

currentContext: main
contexts:
  main:
    # URL of the Armada gRPC API
    armadaUrl: localhost:30002 # This uses the NodePort configured in the Quickstart guide

Using armadactl

Create a queue called example by running:

armadactl create queue example

Submit a job to the example queue by running:

armadactl submit dev/quickstart/example-job.yaml

Check the status of your job in the Lookout UI by visiting http://localhost:30000 in your browser. (This assumes that you installed Armada using the Quickstart guide and exposed it via a NodePort service.)

Migrating

Migrating to v0.11 and beyond

Since v0.11, Armada Scheduler requires you to configure permissionGroupsMapping.

Make sure the applicationConfig field in the Armada Scheduler CRD includes the permissionGroupsMapping field.

Quickstart example that allows anonymous auth:

auth:
  anonymousAuth: true
  permissionGroupMapping:
    execute_jobs: ["everyone"]

Further documentation

To install Armada using the Armada Operator and interact with the Armada API, see the Quickstart guide and follow the runbooks in the dev/runbooks/ directory.

For more info on Armada, see the Armada website and the GitHub repository armadaproject/armada

To understand the minimal configuration required to deploy Armada services, see the armada-crs.yaml file.

For advanced usage, see the Armada CRD reference docs in the dev/crd/ directory.

Supported Armada component configurations

Each Armada Operator CRD supports a .spec.applicationConfig field, which injects configuration into the Armada component, for example:

• Armada Server
• Armada Executor
• Armada Scheduler
• Armada Scheduler Ingester
• Armada Lookout
• Armada Lookout Ingester
• Armada Event Ingester
• Armada Binoculars

Developing locally

This section assumes you have a kind cluster named armada running on your machine (it will appear as kind-armada in your kubeconfig).

See the Makefile for more commands to help you with your development, or run make help for a list of available commands.

Your controller will automatically use the current context in your kubeconfig file (whatever cluster kubectl cluster-info shows).

Starting a development cluster

This section assumes you have KIND installed.

If you do not have a Kubernetes cluster to test against, you can start one using the following command:

make kind-create-cluster

Building and installing a development cluster

Run the following make command:

make kind-deploy

This command does the following:

• builds the armada-operator binary for linux/amd64
• builds the armada-operator Docker image
• loads the Docker image into the kind cluster
• installs each CRD supported by the armada-operator on the cluster
• installs the armada-operator on the cluster using kustomize

Running the Armada Operator locally

To run the operator locally, use one of the following commands:

# Run the operator locally with webhooks enabled
make run
# Run the operator locally with webhooks disabled
make run-no-webhook

Stopping the development cluster

To stop the development cluster:

make kind-delete-cluster

This will totally destroy your development Kind cluster.

Contributing

You can contribute bug-reports or ideas for enhancements via GitHub's issue system.

Code contributions are also welcome. When submitting a pull request, make sure:

• it references a relevant issue
• all CI checks pass

Testing

You should test contributions thoroughly before you ask for a review. At a minimum, the following should all run without error:

# Lint code using golangci-lint.
make lint
# Run unit tests.
make test-unit
# Run integration tests.
make test-integration

Add and change appropriate unit and integration tests to ensure your changes are covered by automated tests and appear to be correct.

FAQ

kube-prometheus-stack is not installing

If you get the following error, upgrade your Helm version to v3.16.2 or later:

Error: template: kube-prometheus-stack/templates/prometheus/prometheus.yaml:262:11: executing "kube-prometheus-stack/templates/prometheus/prometheus.yaml" at <ne .Values.prometheus.prometheusSpec.scrapeConfigNamespaceSelector nil>: error calling ne: uncomparable type map[string]interface {}: map[]

License

Copyright 2024.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.