pulumilab

A Pulumi/Python orchestrator that deploys and manages Nomad jobs across a homelab cluster. It handles templating, Consul KV sync, Nomad variable secrets, and storage volume registration — all driven by a simple directory-per-service layout.

How it works

pulumi up
  └─ main.py            scans nomad_jobs/, groups by tier, deploys in order
       └─ engine.py     per-job: render HCL → register volumes → sync KV → push secrets → submit job
            └─ storage.py  registers host or CSI volumes on Nomad nodes

Deployment is tier-ordered. Each tier completes before the next starts:

core → primary → secondary → tertiary

Jobs declare their tier in manifest.yaml. The default is primary.

Prerequisites

• Python 3.x with a virtualenv (venv/)
• Pulumi CLI
• A running Nomad + Consul cluster (addresses set in globals.yaml)

python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
pulumi stack init <stack-name>

Configuration

globals.yaml is the single source of truth for global settings:

Section	Purpose
nomad_addr / consul_addr	Cluster endpoints
mount_point	Base path for host volumes on nodes
tiers	Ordered deployment tiers
vars	Template variables available in every .hcl and kv/ file
nomad_vars	Secrets/env vars pushed to the Nomad variable store

Secrets in nomad_vars reference Pulumi config values:

nomad_vars:
  postgres_password:
    pulumi_secret_ref: postgres_password   # key in pulumi config

Set them with:

pulumi config set --secret postgres_password "your-value"

Deploying

pulumi up          # preview + deploy
pulumi up --yes    # deploy without confirmation
pulumi destroy     # tear everything down

---

Creating a new job

1. Create the job directory

nomad_jobs/
└── my_service/
    ├── manifest.yaml       # required
    ├── my_service.hcl      # required — the Nomad jobspec
    └── kv/                 # optional — files synced to Consul KV
        └── config.toml

The directory name is arbitrary. Each .hcl file in it becomes a separate Nomad job.

2. Write manifest.yaml

Minimal:

tier: primary

Full options:

# Deployment tier (core | primary | secondary | tertiary)
tier: primary

# Per-job variable overrides — merged on top of globals.yaml vars
vars:
  storage_type: csi

# Nomad variables (secrets/env) pushed to nomad/jobs/<job_name>
# List form: reference keys defined in globals.yaml nomad_vars
nomad_vars:
  - postgres_password
  - mqtt_password

# Dict form: define inline or override globals
# nomad_vars:
#   MY_KEY: plaintext_value
#   MY_SECRET:
#     pulumi_secret_ref: pulumi_config_key

3. Write the HCL jobspec

The filename (without .hcl) becomes the Nomad job name. Use [[ var_name ]] for template variables — this avoids conflicts with Nomad's own {{ }} Go-template syntax.

job "my_service" {
  datacenters = ["[[ datacenter ]]"]
  type        = "service"

  group "my_service" {
    network {
      port "http" { to = "8080" }
    }

    task "my_service" {
      driver = "docker"

      config {
        image = "myimage:latest"
        ports = ["http"]
      }

      # Access Nomad variables (set via manifest.yaml nomad_vars)
      template {
        data        = <<EOF
{{ with nomadVar "nomad/jobs/my_service" }}
MY_SECRET={{ .my_secret }}
{{ end }}
EOF
        destination = "local/.env"
        env         = true
      }
    }
  }
}

Available template variables come from globals.yaml vars, overridden by manifest.yaml vars:

Variable	Example value
datacenter	dc1
storage_type	host
tld	bakos.me
consul_addr	http://192.168.2.82:8500
nomad_addr	http://192.168.2.82:4646
uid / gid	1000
timezone	America/Denver
traefik_vip	192.168.2.3

4. Storage volumes (optional)

Declare a volume block directly in the HCL. The orchestrator reads it automatically and registers the volume before submitting the job — no extra config needed.

Host volume (default):

volume "my_service" {
  type            = "[[ storage_type ]]"
  source          = "my_service"          # subdirectory under /mnt on each node
  attachment_mode = "file-system"
  access_mode     = "single-node-writer"
}

CSI volume:

volume "my_service" {
  type            = "csi"
  source          = "my_service"
  attachment_mode = "file-system"
  access_mode     = "single-node-writer"
}

Mount it inside the task:

volume_mount {
  volume      = "my_service"
  destination = "/data"
  read_only   = false
}

5. Consul KV config files (optional)

Put files in kv/ and they are rendered with the same [[ ]] template syntax and pushed to Consul at <lab_name>/<job_name>/<filename>.

For multi-job groups, use kv/<job_name>/ to keep per-job files separate from shared files.

nomad_jobs/my_service/
└── kv/
    └── config.toml      → pushed to pulumilab/my_service/config.toml

Read from the job using the Nomad template block:

template {
  data        = "{{ key \"pulumilab/my_service/config.toml\" }}"
  destination = "local/config.toml"
}

6. Multiple jobs in one directory

A group directory can contain multiple .hcl files sharing a single manifest.yaml. Each file becomes its own Nomad job.

nomad_jobs/batch_jobs/
├── manifest.yaml
├── docker_cleanup.hcl
├── journalctl_cleanup.hcl
└── nomad_cleanup.hcl

For per-job KV files in a group, place them under kv/<job_name>/:

nomad_jobs/batch_jobs/
├── manifest.yaml
└── kv/
    ├── docker_cleanup/
    │   └── config.toml
    └── journalctl_cleanup/
        └── config.toml

---

Project structure

pulumilab/
├── Pulumi.yaml          # project metadata
├── globals.yaml         # global config, vars, secrets registry
├── globals.py           # loads globals.yaml into Python constants
├── main.py              # entrypoint — scans and orchestrates tiers
├── engine.py            # per-job deploy logic
├── storage.py           # volume registration (host + CSI)
├── requirements.txt
└── nomad_jobs/
    └── <service>/
        ├── manifest.yaml
        ├── <job>.hcl
        └── kv/
            └── <config-file>