chipfarm

A fully containerized mini HPC cluster that simulates a silicon-design compute farm — SLURM scheduling, LDAP identity, internal DNS, NFS homes over autofs, Prometheus/Grafana observability, a fake EDA workload generator, and an AI-assisted triage CLI. Runs entirely on Docker Desktop on Apple Silicon. Zero cloud, zero paid services.

Architecture

flowchart TB
    subgraph identity["Identity & Infra"]
        LDAP[openldap<br/>alice / bob / eve]
        DNS[bind9<br/>chipfarm.local]
        NFS[nfs<br/>/home over NFSv4]
    end

    subgraph slurm["SLURM"]
        CTLD[slurmctld]
        DBD[slurmdbd]
        DB[(MariaDB)]
        C1[compute01]
        C2[compute02]
        C3[compute03]
    end

    subgraph obs["Observability"]
        EXP[slurm-exporter<br/>:9341]
        NE[node_exporter ×3<br/>:9100]
        PROM[Prometheus<br/>:9090]
        GRAF[Grafana<br/>:3000]
    end

    subgraph tools["Tooling (host)"]
        SIM[edasim<br/>workload generator]
        DOC[farmdoctor<br/>AI triage CLI]
        NIM[NVIDIA NIM API]
    end

    CTLD --> DBD --> DB
    CTLD --- C1 & C2 & C3
    C1 & C2 & C3 -- SSSD --> LDAP
    C1 & C2 & C3 -- autofs --> NFS
    C1 & C2 & C3 -- resolve --> DNS
    EXP -- squeue/sinfo --> CTLD
    PROM --> EXP & NE
    GRAF --> PROM
    SIM -- sbatch as LDAP users --> CTLD
    DOC -- collect state --> CTLD
    DOC -- hypothesis --> NIM

Loading

Quickstart

Requirements: Docker Desktop (Apple Silicon or x86), Python 3.9+, ~4 GB free RAM.

cp .env.example .env      # set the three passwords; add NIM_API_KEY for farmdoctor
make up                   # build + start the full stack (first build ~3 min)
make status               # everything Up, 3 nodes idle

Submit something:

docker compose exec slurmctld bash -c \
  'runuser -u alice -- sbatch --chdir=/home/alice --wrap="hostname"'
docker compose exec slurmctld squeue

Grafana: http://localhost:3000 (anonymous viewer) — dashboard chipfarm — Cluster Health & Queue. Prometheus: http://localhost:9090.

Phase walkthrough

1 — Cluster core

slurmctld + slurmdbd/MariaDB accounting + three slurmd nodes, one image for all roles (Rocky 9 + EPEL aarch64 RPMs). Two partitions — batch (default, 24 h) and priority (4 h, PriorityTier 10) — multifactor fairshare priority, normal/high QOS.

docker compose exec slurmctld sinfo
docker compose exec slurmctld sacct

2 — Enterprise services

LDAP users (alice, bob, eve in group chipdesign) resolved on every node through SSSD; bind9 is authoritative for chipfarm.local; homes live on an NFSv4 server and mount on demand through autofs.

docker compose exec compute01 id alice                       # uid 2001 via LDAP
docker compose exec compute01 dig +short compute01.chipfarm.local
docker compose exec compute01 ls /home/alice                 # autofs mounts NFS

3 — Observability

Custom ~130-line Python SLURM exporter (jobs by state, queue depth, node states, pending-wait p50/p95), node_exporter per compute node, provisioned Grafana dashboard, alert rules: node drained >5 min, pending-wait p95 >10 min, node down >2 min.

curl -s localhost:9090/api/v1/rules | python3 -m json.tool | grep name

4 — Workload simulator

make demo-congestion      # 2-CPU jobs oversubscribe the 6-CPU cluster
make demo-stuck           # held jobs + impossible dependencies + hangers
docker compose exec slurmctld python3 /opt/edasim --scenario node-failure

5 — AI triage

make doctor               # detect → NIM hypothesis → suggested fix (dry-run)
python3 -m farmdoctor scan --execute   # y/n confirmation before any command
tail farmdoctor/audit.jsonl            # append-only log of every decision

Detection is pure rules; the LLM only explains findings and proposes one safe command. Responses are cached (farmdoctor/.cache/) keyed on a state hash.

Screenshots

• docs/img/workflow.gif — end-to-end demo (above)

Placeholders — add after running the demos:

• docs/img/grafana-dashboard.png — cluster health during demo-congestion
• docs/img/squeue-fairshare.png — eve's jobs queued behind alice's

Design decisions

EPEL packages over source builds. Rocky 9 + EPEL ships SLURM aarch64 RPMs. Compiling adds minutes to every build and teaches nothing this project cares about.

One SLURM image, role chosen at runtime. slurmctld/slurmd/slurmdbd differ only in the daemon started; compose command: picks the role. One build to maintain.

autofs over fstab. A single wildcard map (* -> nfs:/&) mounts any user's home on first access — no per-user static entries, no mount-at-boot ordering problems, mounts expire when idle. Exactly how real design farms handle thousands of project/tool mounts.

NFSv4 only. One port (2049), no rpcbind/portmapper, simpler client options. v3 buys nothing here.

Fairshare weighted 10:5:1 over QOS and age. Fairshare dominates so the alice (50) / bob (30) / eve (10) share split visibly reorders the queue under contention; QOS adds a lever for urgent work; age breaks ties so nothing starves forever.

cgroup v1 plugin with automount off. slurmd force-initializes its cgroup plugin; the v2 plugin demands dbus/systemd, absent in containers. The 22.05 escape hatch is the v1 plugin with automount disabled — a no-op unless a cgroup-using plugin is configured (CgroupPlugin=disabled only exists from 23.02).

Rules detect, LLM explains. farmdoctor's detectors are deterministic code. The LLM never decides whether something is wrong — only hypothesizes why and proposes a command, which a human must approve. Wrong-but-confident LLM output can waste an operator's time; letting it gate detection or run commands unsupervised could break the cluster.

Secrets never in the repo. Munge key is generated into a named volume on first up; DB/LDAP passwords come from .env; the NIM key is env-only. .gitignore covers all of it.

Repo layout

docker/          slurm, ldap, nfs images
config/          slurm.conf, cgroup.conf, bind zone, sssd, autofs maps
monitoring/      prometheus config + rules, grafana provisioning, exporter
edasim/          workload generator (runs inside slurmctld)
farmdoctor/      triage CLI (runs on the host)