Skip to content

Configuration

Jump to bottom
Biowilko edited this page Aug 5, 2026 · 4 revisions

Configuration

squarepeg reads config from a layered set of YAML files, resolved preferentially: several sources are located, loaded, and deep-merged together before any CLI flag is applied — rather than there being just one config file.

Resolution order

Lowest precedence to highest — each layer can override just the keys it cares about, without having to restate everything from the layers below it:

Order Source Notes
1 Built-in defaults Not a file — hardcoded fallbacks (mode: pod, timeout: 300, cleanup: true, orphan_sweep: true, orphan_sweep_min_age: 300, etc.)
2 ~/.config/squarepeg/config.yaml The single default location. Silently skipped if it doesn't exist. Skip it entirely with --no-default-config.
3 $SQUAREPEG_CONFIG A colon-separated list of paths (same convention as $PATH/$KUBECONFIG), loaded left-to-right. A listed-but-missing file is an error (unlike the default location).
4 --config PATH Repeatable CLI flag, loaded in the order given, left-to-right.
5 Individual CLI flags -e, --cpus, -n, etc. — always win over everything above.

Every layer (2–4) is deep-merged key by key, not replaced wholesale — a later file overriding just mode: job doesn't wipe out a namespace or tolerations set by an earlier one. See How merging works below for the exact algorithm.

# Site-wide defaults plus a per-user override, composed automatically:
SQUAREPEG_CONFIG=/etc/squarepeg/site.yaml squarepeg run --config ~/my-overrides.yaml alpine true

Top-level schema

Key Type Description
namespace string Target namespace.
mode pod | job Default execution mode.
cleanup bool false behaves as if --keep were always passed.
timeout int Seconds to wait for the container to start (see Execution Lifecycle).
quiet bool Suppress squarepeg's own status messages.
allow_host_path_mounts bool Opt in to -v /host:/container bind mounts (only sensible against a local cluster).
split_streams bool Reserved for splitting stdout/stderr into separate streams (depends on an apiserver feature gate; not fully wired up yet).
orphan_sweep bool Delete your own leftover squarepeg pods/jobs in the namespace as part of each run's cleanup — both finished-but-undeleted and never-started-and-stuck. false disables it. Requires list on pods/jobs (broader than the create/delete/get squarepeg otherwise needs) — see Execution Lifecycle → Clean up.
orphan_sweep_min_age int (seconds) Default 300. How long a resource must have been finished — or, for a never-started pod, how long it must have existed — before it's eligible for the sweep.
defaults mapping See defaults below.
volumes mapping Named volume sources; can also declare auto-mounts via mount_path — see Volumes below.
kubernetes mapping Arbitrary Kubernetes passthrough — see below.
job mapping Job-level passthrough (ignored in --mode pod) — see below.
profiles mapping Named config overlays, selected with --profile — see Profiles.

Unknown top-level keys are a hard error, naming the offending file — this is deliberate typo protection (a silently-ignored namesapce: would be a bad day). The kubernetes and job sections are the exception: they're deliberately free-form and not validated by squarepeg at all — the Kubernetes apiserver validates them, and --dry-run lets you inspect the result before it gets there.

The defaults section

Applied whenever the equivalent CLI flag isn't given:

Key Maps to Units
cpus --cpus docker-style (e.g. "1", "0.5")
memory --memory docker-style (e.g. 2g)
image_pull_policy --pull Kubernetes-native (Always/IfNotPresent/Never, already-cased)
workdir -w/--workdir —
env -e/--env mapping of KEY: value; CLI -e entries win over these on a key collision, but config-only keys still get included

Volumes

Each entry under volumes.NAME holds a Kubernetes volume source (persistentVolumeClaim, emptyDir, etc.) plus two optional squarepeg-owned fields:

Key Type Description
mount_path string If present, this volume is mounted at this path on every run, with no -v flag needed at all. If absent, the entry behaves as it always has — available for a user to reference with -v NAME:/path, not mounted otherwise.
read_only bool Whether the auto-mount is read-only. Default false. Only meaningful alongside mount_path. 
volumes:
  refdata:                                  # only mounted if the user does -v refdata:/x
    persistentVolumeClaim: {claimName: refdata-pvc, readOnly: true}
  shared-tools:                             # auto-mounted at /opt/tools on EVERY run, no -v needed
    persistentVolumeClaim: {claimName: shared-tools-pvc, readOnly: true}
    mount_path: /opt/tools
    read_only: true

A few things worth knowing:

  • A volume can be both auto-mounted and separately -v-referenced. If a user does -v shared-tools:/somewhere-else for a volume already auto-mounted above, both mounts exist side by side in the container — Kubernetes allows mounting the same volume at multiple paths, so this needs no special handling.
  • mount_path/read_only never leak into the generated manifest — squarepeg strips them before emitting the Kubernetes volume source, whether the volume was auto-mounted or resolved via -v.
  • No CLI opt-out. There's no --no-auto-mount flag; auto-mounted volumes are controlled purely by whoever owns the config layer that set mount_path. A user who wants different behaviour edits their own --config layer (which can override or remove mount_path for that name, since volumes.NAME is a normal dict that deep-merges across layers same as everything else) or selects a different --profile.
  • Auto-mounts are config-driven, not CLI-driven, so they aren't "claimed" — kubernetes.spec passthrough in the same resolved config can still override an auto-mount by name, consistent with how other config defaults (e.g. defaults.cpus) can be overridden by passthrough in the same config.
  • This is friendlier sugar over something you could already do by hand-writing matching kubernetes.spec.volumes + containers[].volumeMounts entries — see below — the mount_path field just saves you from writing both halves of that yourself.

The kubernetes and job passthrough sections

These merge directly into the generated manifest, letting you set any Kubernetes field squarepeg has no dedicated flag for — node selectors, tolerations, service accounts, image pull secrets, security contexts, GPU resources, and so on.

kubernetes.spec (and kubernetes.metadata) is always a Pod spec/metadata, even when --mode job is selected. squarepeg re-homes it under the Job's pod template automatically, so the same config file works unchanged in either mode — without this rule, switching --mode would silently drop every toleration and node selector you'd configured.

kubernetes:
  metadata:
    labels: {team: bioinf}
  spec:
    serviceAccountName: squarepeg
    nodeSelector: {"kubernetes.io/arch": amd64}
    tolerations:
      - {key: dedicated, operator: Equal, value: batch, effect: NoSchedule}
    imagePullSecrets:
      - {name: regcred}
    containers:
      - name: main                                # matched by name; merges into squarepeg's own container
        securityContext: {allowPrivilegeEscalation: false}

job:                                                # Job-level only; ignored in --mode pod
  spec:
    backoffLimit: 0
    ttlSecondsAfterFinished: 3600

CLI flags always win over passthrough on the same field. If you pass -e FOO=bar on the command line, a kubernetes.spec.containers[].env entry for FOO in your config is silently overridden for that field only — every other field you set in passthrough still applies. This is what "layered, not replaced" means in practice; see How merging works.

Profiles

profiles:
  gpu:
    defaults: {cpus: "8", memory: 32g}
    kubernetes:
      spec:
        nodeSelector: {"nvidia.com/gpu.present": "true"}
        containers:
          - name: main
            resources: {limits: {"nvidia.com/gpu": 1}}
squarepeg run --profile gpu my-training-image train.py

A profile is itself a config document (using the same schema, minus a nested profiles key — that's rejected) that gets deep-merged over the rest of the resolved config when selected with --profile NAME.

Two things worth knowing:

  • Same-named profiles across multiple config layers deep-merge, last-loaded wins per key — not an error. This lets a site-wide file own a profile's scheduling details (node selector, toleration) while a per-user file adds just a resource size to the same profile name, without either file needing to know about the other's contents.
  • Profile bodies are validated and interpolated eagerly, regardless of whether you select them. A typo or an unset ${VAR} reference inside profiles.gpu is a hard error on every invocation of squarepeg against that config file, not just ones using --profile gpu. This makes "is my config file valid?" a property of the file, not of the flags you happen to pass.

How merging works

Every layering step — config file over config file, profile over base config, and Kubernetes passthrough over the generated manifest — uses the same deep-merge routine:

  1. Dicts merge key by key, recursively.
  2. Lists where every element has a name key merge by name (this is what makes containers, env, volumes, volumeMounts behave intuitively — a later layer adding one env var doesn't wipe out the others).
  3. Any other list is replaced wholesale by the later layer — this is deliberate: tolerations is a list without a stable key to merge by, and replacement is the only way a later layer can remove an inherited toleration (an append-only merge would give no way to say "actually, none of these").
  4. CLI-set fields are "claimed" and cannot be overridden by config passthrough, even if the passthrough is loaded after the CLI flags are parsed — e.g. -e FOO=bar always beats a kubernetes...env entry for FOO, but a passthrough BAR env var still lands untouched.

Example

See examples/config.yaml in the repo for a complete worked example covering every section above, including ${VAR} references (see Environment Variable Interpolation).

squarepeg

Clone this wiki locally