-
Notifications
You must be signed in to change notification settings - Fork 0
Troubleshooting
squarepeg raises a specific error type for every failure condition it detects, each carrying a fixed process exit code — see CLI Reference → Exit codes for the summary. This page covers what triggers each one and what to do about it.
| Situation | Message tells you |
|---|---|
Unsupported docker flag (--network, --gpus, -p, ...) |
The flag name, why it can't work on a remote cluster, and the config-passthrough equivalent if one exists — see the full table. |
Malformed -e/-v/quantity value |
The offending token and the accepted forms. |
-e KEY (bare, no =) but KEY isn't set in your local shell |
Which variable was missing. |
-v /host/path:/container/path (host bind mount) without opting in |
Points at a config-defined PVC-backed volume, or allow_host_path_mounts: true. |
--rm combined with --keep
|
These are mutually exclusive. |
--name isn't a valid RFC1123 label |
What the valid format is (lowercase alphanumeric + -, ≤63 chars). |
| A leading-dash token where the image name should be | Names the token, explains it looks like a flag that landed before the image position. |
| Situation | Message tells you |
|---|---|
--config PATH / a $SQUAREPEG_CONFIG entry doesn't exist |
The exact path, and which source named it (--config foo.yaml vs $SQUAREPEG_CONFIG). The default location (~/.config/squarepeg/config.yaml) is the one exception — silently skipped if absent, since it's never explicitly requested. |
| Malformed YAML | The file path and line number. |
Unknown top-level config key (e.g. namesapce:) |
The key name and the exact file it came from. |
Unknown --profile NAME
|
Every profile name found across all merged config layers. |
${VAR} reference to an unset environment variable |
The variable name, the exact key path (e.g. kubernetes.spec.imagePullSecrets[0].name), the source file, and a reminder that ${VAR:-default} exists — see Environment Variable Interpolation. |
Malformed ${...} syntax (${}, ${1BAD}, unterminated ${VAR) |
The malformed token itself. |
A config value squarepeg needs as an int/bool doesn't coerce (e.g. timeout: "soon") |
The key name and the value it couldn't parse. |
volumes.NAME.mount_path isn't an absolute path |
The volume name, the source file, and the value it got instead. |
volumes.NAME.read_only doesn't coerce to a bool |
Same as the general int/bool coercion case above, scoped to that volume's key. |
These mean squarepeg itself couldn't get the container to a known-good completed state — as distinct from the container running and exiting with its own non-zero code, which is reported verbatim instead.
| Situation | What happens |
|---|---|
| No kubeconfig found, and not running in-cluster either | Both failures are named in the message — the kubeconfig error and the in-cluster attempt's error — since squarepeg tries in-cluster config as a fallback when no --context was given (see Execution Lifecycle → Connect to the cluster). |
Unknown --context NAME
|
Names the context that wasn't found in your kubeconfig. This does not fall back to in-cluster config — an explicit --context failure is always surfaced directly. |
Image pull failure (ImagePullBackOff, ErrImagePull, InvalidImageName, ...) |
Fails fast — doesn't wait out the full --timeout. Names the specific reason and message from the container's waiting state. |
Container never starts within --timeout
|
Reports the last known phase, plus any relevant Kubernetes events (scheduling failures, unbound volumes, etc.), then cleans up (unless --keep). |
| Container terminates but its exit code can't be read (evicted, deleted mid-run) | Falls back to 125, matching docker run's own convention for "the runner failed" rather than fabricating a container exit code. |
These same statuses, hints, and exit code also arise from --dry-run-server — it makes a real authenticated apiserver call (just one that persists nothing), so a rejection there is identical to a rejection on a real run, minus anything to clean up afterwards.
squarepeg always tries to surface the apiserver's own Status.message (from the response body) rather than just the bare HTTP reason phrase — so instead of a bare Bad Request, you should see the specific field the apiserver rejected, e.g. Pod "x" is invalid: spec.nodeSelector: Invalid value: .... This is the single most useful thing in the error, since it usually names the exact malformed field.
| HTTP status | Hint squarepeg adds |
|---|---|
| 400 | "the apiserver rejected the manifest as malformed; run with --dry-run to inspect it" — commonly a field with the wrong shape in kubernetes/job passthrough config (e.g. nodeSelector written as a bare key=value string instead of a mapping, or annotations written as a list instead of a mapping) |
| 401 | "authentication failed; try kubectl get pods to check your credentials" |
| 403 (RBAC) | "permission denied; check with kubectl auth can-i create pods" |
| 403 (ResourceQuota) | A different hint replaces the RBAC one above when the message contains failed quota — see below. |
| 404 | "not found; check the namespace exists" |
| 409 | "already exists; omit --name, or clean up a --keep'd previous run" |
| 422 | "the apiserver rejected the manifest; run with --dry-run to inspect it" — the most common symptom of a typo in kubernetes/job passthrough config, since that section is otherwise unvalidated by squarepeg |
Some clusters/namespaces enforce a ResourceQuota that requires every container to declare CPU/memory requests (and sometimes limits). Because squarepeg doesn't set any resources block unless you ask it to, a plain squarepeg run alpine ... on such a cluster fails with something like:
squarepeg: failed to create pod 'squarepeg-alpine-163c34': pods "squarepeg-alpine-163c34"
is forbidden: failed quota: quota: must specify requests.cpu for: main; requests.memory for: main
(the target namespace enforces a ResourceQuota requiring cpu/memory requests (and possibly
limits) on every container; set --cpus/--memory (or --request-cpu/--request-memory/--limit-cpu/
--limit-memory), or 'defaults.cpus'/'defaults.memory' in config so you don't have to pass them
on every run)
This is an HTTP 403 from the apiserver, but it has nothing to do with RBAC permissions — squarepeg detects the failed quota wording in the rejection message and swaps in this quota-specific hint instead of the generic "check with kubectl auth can-i" one, since that would send you down the wrong troubleshooting path entirely.
Fix by either passing --cpus/--memory (or the more explicit --request-cpu/--request-memory/--limit-cpu/--limit-memory) on each run, or — the more usual fix for a cluster you use repeatedly — setting defaults.cpus/defaults.memory in your config so every run gets sane defaults without needing the flags each time. See Configuration → The defaults section.
Ctrl+C (or SIGTERM) during a run. See Execution Lifecycle → Interrupt handling for the two-stage behaviour — the first press cleans up and exits 130; a second press abandons the resource running instead.
Every run's cleanup step also sweeps your own other orphaned squarepeg resources in the namespace (see Execution Lifecycle → The orphan sweep). This can surprise you in a few specific ways:
"An old pod I expected to still be there is gone." Two recognisably different scenarios:
-
A pod that had finished — it was
Succeeded/Failedfor longer thanorphan_sweep_min_age(default 300s). -
A pod that was stuck and never started — typically one showing
ImagePullBackOff/ErrImagePull/InvalidImageNameinkubectl get pods, left behind by a squarepeg process that was killed before it could clean up after its own typo'd image reference. This is the scenario most likely to catch you off guard, since such a pod can sit inPendingfor days and start to feel permanent — and you may well have been about tokubectl describeit to find out what was wrong with the image reference.
If you want to keep a pod like this around for inspection, pass --keep on the run (or set orphan_sweep: false) — kubectl describe pod, or the events squarepeg already printed to stderr on fail-fast, give the same diagnosis without needing the pod to survive. To see what's currently sweepable in a namespace: kubectl get pods,jobs -l app.kubernetes.io/managed-by=squarepeg,squarepeg.io/created-by=<you>.
"My --keep'd pod disappeared." A pod kept by a version of squarepeg older than the orphan sweep feature carries no squarepeg.io/keep label, so it's indistinguishable from a genuine orphan — the first post-upgrade run may sweep it. There's no way to recover that distinction after the fact. If you have something in this state right now: label it manually (kubectl label pod NAME squarepeg.io/keep=true), or set orphan_sweep: false before your first run on the upgraded version. Note the sweep only ever touches resources matching your own created-by, so a teammate's --keep'd pod was never at risk.
orphan sweep skipped: ... 403 ... printed on every run. The namespace's Role likely lacks list on pods/jobs (a broader grant than the create/delete/get squarepeg otherwise needs). Harmless — this run's own outcome is never affected — but if it bothers you, either set orphan_sweep: false in config, or ask whoever manages the namespace's RBAC for the list verb.
| Symptom | Try |
|---|---|
| "Why is my namespace/resource-size/whatever what it is?" |
squarepeg config show — prints every config file that was loaded, in merge order, plus the fully resolved (post-interpolation) result. |
| "Is my config producing the manifest I expect?" |
squarepeg run --dry-run ... — renders the exact manifest without touching the cluster at all. |
| "Will the cluster actually accept it?" |
squarepeg run --dry-run-server ... — the same manifest, but validated by the real apiserver (RBAC, ResourceQuota, admission webhooks) and discarded without being persisted. The natural next step once --dry-run shows the manifest looks right on the client side — see Execution Lifecycle → Dry run. |
| A run seems to hang with no output | Check stderr — squarepeg's own progress/event messages go there, separately from container stdout. If genuinely stuck past --timeout, it will time out and report the last known phase. |
A container that reads stdin hangs after -t
|
This is the documented tty implying stdin hazard — see Docker Compatibility → Interactive mode. Real stdin forwarding isn't implemented; avoid -t for containers that block on input. |
"Why is this volume mounted — I never passed -v?" |
A config layer declared it as an auto-mount (volumes.NAME.mount_path) — see Configuration → Volumes. squarepeg config show will show you exactly which volumes.NAME entries have a mount_path set, and which config file set it. |