-
Notifications
You must be signed in to change notification settings - Fork 0
Execution Lifecycle
What actually happens, in order, between running squarepeg run and getting your prompt back.
- Docker-style flags are parsed. Everything after
IMAGE— including lookalike flags — is passed through untouched as the container command (ignore_unknown_options/allow_interspersed_argsin click terms). - Every config layer is loaded,
${VAR}interpolated, validated, and deep-merged (see Configuration). - CLI flags are applied on top; each explicitly-set flag is recorded as a claim on a specific manifest field (e.g.
/spec/containers/[name=main]/env/[name=FOO]) so that config passthrough can never silently override something you set on the command line, while everything you didn't set is still open to passthrough.
The Pod or Job is built as a plain Python dict (not a typed Kubernetes model) — this is what makes --dry-run output exactly what would be sent to the apiserver, with no serialization round-trip, and makes arbitrary passthrough config trivial to merge in.
--name given |
Validated as a lowercase RFC1123 label (alphanumeric + -, ≤63 chars) and used verbatim. |
--name omitted |
Generated as squarepeg-<image-slug>-<6 random hex chars>, e.g. squarepeg-alpine-a1b2c3. The image reference has its registry host, tag, and digest stripped first, so registry.example.com:5000/foo/bar:tag becomes bar. |
Every Pod/Job squarepeg creates carries:
| Label | Value |
|---|---|
app.kubernetes.io/managed-by |
squarepeg |
squarepeg.io/run-id |
A random 8-hex-character token, unique per invocation. Printed in every status line's prefix as [squarepeg:<run-id>] , so you can go straight from a line in your terminal to kubectl get pods -l squarepeg.io/run-id=<value> without reading the manifest — see CLI Reference → The run-id tag. |
squarepeg.io/created-by |
Your local username (sanitized to a valid label value) |
squarepeg.io/keep |
Present ("true") only when the resource was created with --keep or cleanup: false. Absent otherwise. |
These do more than make orphaned resources identifiable later (e.g. via kubectl get pods -l app.kubernetes.io/managed-by=squarepeg) — squarepeg acts on them: every run's cleanup step also sweeps other orphaned resources matching your own created-by, and squarepeg.io/keep is what exempts a deliberately-kept resource from that sweep. See Clean up below. The run id is the only one of these labels squarepeg surfaces in its own output; the others are for querying after the fact.
Pod (--mode pod, default) |
Job (--mode job) |
|
|---|---|---|
restartPolicy |
Never |
Never (on the pod template) |
| Retries | None |
backoffLimit: 0 (also none, by default) |
| Extra fields | — |
completions: 1, parallelism: 1, ttlSecondsAfterFinished (a backstop cleanup in case squarepeg itself is killed before it can delete the resource) |
| Naming the actual pod | The name you gave/generated | Kubernetes generates the pod name; squarepeg discovers it via the job-name label |
kubernetes.spec/kubernetes.metadata passthrough in config is always expressed as a Pod spec/metadata, even in Job mode — squarepeg re-homes it under spec.template.{spec,metadata} automatically so the same config file works in either mode without rewriting.
squarepeg resolves credentials the same way kubectl does, then falls back to the pod's own service account if it's running inside a cluster itself:
| Order | Source | When it applies |
|---|---|---|
| 1 | Kubeconfig ($KUBECONFIG, then ~/.kube/config, honouring --context) |
The default — squarepeg on your laptop/CI runner talking to a remote cluster. |
| 2 | In-cluster config (the pod's own service account token, mounted at /var/run/secrets/kubernetes.io/serviceaccount/) |
Only tried if no --context was given and the kubeconfig load failed outright — i.e. squarepeg is itself running as a container (inside Argo/Tekton/a custom operator/etc.) and needs to spawn sibling pods/jobs using its own pod's permissions. |
Two things worth knowing:
-
An explicit
--contextthat fails never falls back to in-cluster. If you asked for a specific context and it's not there, that failure is surfaced directly — it's not silently masked by an in-cluster attempt you didn't ask for. -
Namespace resolution differs slightly in-cluster. There's no "context" concept once running in-cluster, so the equivalent fallback is the namespace projected into the pod at
/var/run/secrets/kubernetes.io/serviceaccount/namespace, read in the same precedence slot a kubeconfig context's namespace would otherwise occupy: explicit-n/--namespacebeats it, and it beats squarepeg's owndefaultfallback.
squarepeg prints which one it resolved to stderr on startup (context name — or in-cluster — plus namespace), unless --quiet.
squarepeg creates the resource, then watches it with a bounded timeout (--timeout, default 300s) that covers only startup — image pull, scheduling, container creation.
Fails fast (doesn't wait out the full timeout) if the container's waiting-state reason is one of:
ImagePullBackOff · ErrImagePull · InvalidImageName · CreateContainerConfigError
CreateContainerError · RunContainerError · CrashLoopBackOff
This list matters beyond the current invocation, too: a pod left behind by a squarepeg process that died before it could react to one of these reasons (machine killed, kill -9, a network partition) is exactly what the orphan sweep's "stuck" category cleans up later — see Clean up.
If the pod is still Pending and something's blocking it (insufficient resources, an unschedulable node selector, an unbound PVC), squarepeg surfaces the relevant Kubernetes events to stderr — the difference between "it just hung" and "0/12 nodes available: insufficient memory".
Once the container reaches Running (or has already terminated by the time squarepeg checks), startup is considered complete and the timeout no longer applies.
Log streaming runs in a background thread, concurrently with waiting for the container to actually finish:
- Output is written as raw bytes straight to your terminal's stdout — not decoded/re-encoded line by line — so partial lines, ANSI colour codes, and
\rprogress bars all come through correctly. - Reconnect on drop: if the log stream disconnects mid-run (apiserver restart, proxy timeout), squarepeg reconnects automatically, with exponential backoff, up to 5 attempts. Because the Kubernetes log API only supports "give me the last N seconds" (not "give me everything after this exact timestamp"), a reconnect necessarily re-fetches a small overlapping window — squarepeg deduplicates by timestamp so you don't see repeated lines.
-
squarepeg's own messages never mix with container output: everything the container writes goes to your stdout; every status/warning/diagnostic message from squarepeg itself goes to stderr. This means
squarepeg run alpine cat bigfile.txt > out.txtcaptures exactly the container's output, nothing else.
Once the container reaches Succeeded/Failed, squarepeg reads its final container status and extracts terminated.exitCode (matched by container name, not by list position — relevant if passthrough config adds sidecar containers).
| Situation | Exit code squarepeg reports |
|---|---|
| Container terminated normally | Its own exit code, verbatim |
| Container was OOM-killed |
137, with the OOMKilled reason additionally printed to stderr
|
| Container never actually terminated (evicted, pod deleted out from under squarepeg, etc.) |
125 — "the runner itself failed", matching docker run's own convention (as opposed to a code the container chose) |
| Situation | Behaviour |
|---|---|
| Default | The Pod/Job is deleted once the exit code has been captured. Job deletion uses propagationPolicy: Background explicitly, so the child pod isn't orphaned. Deletion tolerates a 404 (already gone) without erroring. |
--keep |
Nothing is deleted, and it is labelled (squarepeg.io/keep) so future runs' orphan sweeps skip it too. squarepeg prints the exact kubectl describe/kubectl logs commands to inspect it afterwards. |
Job's ttlSecondsAfterFinished
|
A backstop for completed Jobs, independent of squarepeg's own cleanup — if squarepeg itself is kill -9'd before it can delete anything, the cluster cleans up the Job on its own after this TTL elapses. Bare Pods have no such mechanism at all, which is exactly why the orphan sweep below exists. |
| Orphan sweep | Runs on every invocation, alongside (never instead of) the delete above — see below. |
Beyond deleting this invocation's own resource, every run also opportunistically sweeps the namespace for other squarepeg-managed Pods/Jobs left behind by a previous invocation that crashed or was killed before it could clean up after itself (machine died, kill -9, a network partition). Two independent categories, because a resource that never managed to start looks nothing like one that finished and wasn't deleted:
-
Finished but not deleted — phase
Succeeded/Failed, older thanorphan_sweep_min_age(default 300s) since it finished. NeverRunning/Pending/Unknownat any age. -
Never managed to start — phase
Pendingwith a container stuck on one of the fail-fast reasons above (a typo'd image reference being the common cause), older thanorphan_sweep_min_agesince it was created. This category exists because such a pod never reaches a terminal phase — the kubelet retries the pull forever, and a bare Pod has nobackoffLimit/TTL to stop it — so without the sweep it would sit in the namespace indefinitely. A pod that's merely slow to start (a large image pulling normally, an unbound PVC, pending scheduling) has no fail-fast reason and is never swept, however long it sits there. - Jobs are judged by their own
Complete/Failedcondition (or, for a Job whose child pod is stuck per the category above with no condition yet — e.g.backoffLimit: 0never records a failure — by that child's state) and deleted with Background propagation so child pods cascade. Job-owned pods are never deleted directly.
Key properties:
-
Scoped to your own resources only — the sweep's label selector matches
squarepeg.io/created-byagainst your own username, enforced server-side, so a teammate's resource in a shared namespace is never even returned by the apiserver, let alone touched. -
--keep'd resources are exempt, via thesquarepeg.io/keeplabel. -
Best-effort and cannot affect this run's own exit code. A sweep failure (a transient API error, insufficient RBAC, anything) is logged to
stderrand swallowed — it never changes the outcome of the run it rode along with. -
Orthogonal to
--keep/cleanup: false— a kept run still sweeps other orphans; these are independent decisions about independent resources. - Configurable via
orphan_sweep(bool, defaulttrue) andorphan_sweep_min_age(seconds, default300) — see Configuration. No CLI flag exists for either; the sweep needslistpermission on pods and Jobs in the namespace.
squarepeg installs handlers for SIGINT/SIGTERM for the duration of a run, implementing a two-stage interrupt:
| Press | Effect |
|---|---|
| First Ctrl+C | Stops log streaming, waits for the exit-code extraction to finish, deletes the resource (unless --keep) and runs the orphan sweep, and exits 130. Prints a message telling you a second Ctrl+C will leave it running. |
| Second Ctrl+C | Abandons cleanup immediately — the Pod/Job is left running in the cluster, and the orphan sweep is skipped too. Prints its name so you can find and manage it manually (kubectl describe/kubectl delete). |
Exit code on interrupt is always 130 (128 + SIGINT), the standard shell convention.
Two flavours, both stopping short of creating anything real, but differing in what they check and what they print.
squarepeg run --dry-run --cpus 0.5 -m 512m alpine echo hiRenders the fully-merged manifest (CLI flags + all config layers + passthrough, all applied) as YAML to stdout, and exits 0 — without creating anything and without requiring a reachable cluster at all. The only thing that needs to succeed is local kubeconfig parsing for namespace resolution, and even that degrades gracefully if it fails.
This is the fastest way to check "did my config actually produce what I expected" before committing to a real run against the cluster.
squarepeg run --dry-run-server --cpus 0.5 -m 512m alpine echo hiBuilds the exact same manifest --dry-run would print, but instead of just rendering it locally, submits it to the apiserver as a server-side dry run (dryRun=All) — the same connect-to-the-cluster step a real run goes through, including the kubeconfig/in-cluster fallback, honouring --context/-n, and printing the "using context X, namespace Y" banner to stderr (suppressed by --quiet).
The apiserver runs its full validation, defaulting, and admission/mutating webhook pipeline against the manifest, then discards it without persisting anything. squarepeg prints the apiserver's response object (not the client-rendered manifest) as YAML to stdout — this is the whole point of the flag: you see fields the apiserver itself added or changed (a resolved metadata.namespace, spec.dnsPolicy, a webhook-injected sidecar or label, etc.) that plain --dry-run can never show you, since it never leaves your machine.
Because nothing is actually created, none of the steps below this one run: no watch for startup, no log stream, no wait for completion, no exit-code extraction, no cleanup, and no orphan sweep. There's also nothing to interrupt cleanly, so no two-stage Ctrl-C handler is installed — a Ctrl-C here behaves like an ordinary Python KeyboardInterrupt.
--dry-run |
--dry-run-server |
|
|---|---|---|
| Requires a cluster connection | No | Yes |
| What's printed | The client-rendered manifest | The apiserver's response object |
| Validation performed | None — squarepeg's own manifest construction only | Full apiserver validation, RBAC, ResourceQuota, admission/mutating webhooks |
| Exit code on rejection | N/A (never contacts a cluster) |
125, same hints as a real create failure |
The two flags are mutually exclusive — pass one or the other, not both.
--dry-run-server is the way to catch a ResourceQuota rejection, an RBAC denial, or an admission-webhook rejection before committing to a real run, once --dry-run has already confirmed the manifest looks right on the client side.