Execution Lifecycle

Execution Lifecycle

What actually happens, in order, between running squarepeg run and getting your prompt back.

1. Parse and resolve

1. Docker-style flags are parsed. Everything after IMAGE — including lookalike flags — is passed through untouched as the container command (ignore_unknown_options/allow_interspersed_args in click terms).
2. Every config layer is loaded, ${VAR} interpolated, validated, and deep-merged (see Configuration).
3. CLI flags are applied on top; each explicitly-set flag is recorded as a claim on a specific manifest field (e.g. /spec/containers/[name=main]/env/[name=FOO]) so that config passthrough can never silently override something you set on the command line, while everything you didn't set is still open to passthrough.

2. Build the manifest

The Pod or Job is built as a plain Python dict (not a typed Kubernetes model) — this is what makes --dry-run output exactly what would be sent to the apiserver, with no serialization round-trip, and makes arbitrary passthrough config trivial to merge in.

Naming

--name given	Validated as a lowercase RFC1123 label (alphanumeric + -, ≤63 chars) and used verbatim.
--name omitted	Generated as squarepeg-<image-slug>-<6 random hex chars>, e.g. squarepeg-alpine-a1b2c3. The image reference has its registry host, tag, and digest stripped first, so registry.example.com:5000/foo/bar:tag becomes bar.

Labels

Every Pod/Job squarepeg creates carries:

Label	Value
app.kubernetes.io/managed-by	squarepeg
squarepeg.io/run-id	A random UUID, unique per invocation
squarepeg.io/created-by	Your local username (sanitized to a valid label value)
squarepeg.io/keep	Present ("true") only when the resource was created with --keep or cleanup: false. Absent otherwise.

These do more than make orphaned resources identifiable later (e.g. via kubectl get pods -l app.kubernetes.io/managed-by=squarepeg) — squarepeg acts on them: every run's cleanup step also sweeps other orphaned resources matching your own created-by, and squarepeg.io/keep is what exempts a deliberately-kept resource from that sweep. See Clean up below.

Pod vs Job

	Pod (--mode pod, default)	Job (--mode job)
restartPolicy	Never	Never (on the pod template)
Retries	None	backoffLimit: 0 (also none, by default)
Extra fields	—	completions: 1, parallelism: 1, ttlSecondsAfterFinished (a backstop cleanup in case squarepeg itself is killed before it can delete the resource)
Naming the actual pod	The name you gave/generated	Kubernetes generates the pod name; squarepeg discovers it via the job-name label

kubernetes.spec/kubernetes.metadata passthrough in config is always expressed as a Pod spec/metadata, even in Job mode — squarepeg re-homes it under spec.template.{spec,metadata} automatically so the same config file works in either mode without rewriting.

3. Connect to the cluster

squarepeg resolves credentials the same way kubectl does, then falls back to the pod's own service account if it's running inside a cluster itself:

Order	Source	When it applies
1	Kubeconfig ($KUBECONFIG, then ~/.kube/config, honouring --context)	The default — squarepeg on your laptop/CI runner talking to a remote cluster.
2	In-cluster config (the pod's own service account token, mounted at /var/run/secrets/kubernetes.io/serviceaccount/)	Only tried if no --context was given and the kubeconfig load failed outright — i.e. squarepeg is itself running as a container (inside Argo/Tekton/a custom operator/etc.) and needs to spawn sibling pods/jobs using its own pod's permissions.

Two things worth knowing:

• An explicit --context that fails never falls back to in-cluster. If you asked for a specific context and it's not there, that failure is surfaced directly — it's not silently masked by an in-cluster attempt you didn't ask for.
• Namespace resolution differs slightly in-cluster. There's no "context" concept once running in-cluster, so the equivalent fallback is the namespace projected into the pod at /var/run/secrets/kubernetes.io/serviceaccount/namespace, read in the same precedence slot a kubeconfig context's namespace would otherwise occupy: explicit -n/--namespace beats it, and it beats squarepeg's own default fallback.

squarepeg prints which one it resolved to stderr on startup (context name — or in-cluster — plus namespace), unless --quiet.

4. Create and wait for start

squarepeg creates the resource, then watches it with a bounded timeout (--timeout, default 300s) that covers only startup — image pull, scheduling, container creation.

Fails fast (doesn't wait out the full timeout) if the container's waiting-state reason is one of:

ImagePullBackOff · ErrImagePull · InvalidImageName · CreateContainerConfigError
CreateContainerError · RunContainerError · CrashLoopBackOff

This list matters beyond the current invocation, too: a pod left behind by a squarepeg process that died before it could react to one of these reasons (machine killed, kill -9, a network partition) is exactly what the orphan sweep's "stuck" category cleans up later — see Clean up.

If the pod is still Pending and something's blocking it (insufficient resources, an unschedulable node selector, an unbound PVC), squarepeg surfaces the relevant Kubernetes events to stderr — the difference between "it just hung" and "0/12 nodes available: insufficient memory".

Once the container reaches Running (or has already terminated by the time squarepeg checks), startup is considered complete and the timeout no longer applies.

5. Stream logs

Log streaming runs in a background thread, concurrently with waiting for the container to actually finish:

• Output is written as raw bytes straight to your terminal's stdout — not decoded/re-encoded line by line — so partial lines, ANSI colour codes, and \r progress bars all come through correctly.
• Reconnect on drop: if the log stream disconnects mid-run (apiserver restart, proxy timeout), squarepeg reconnects automatically, with exponential backoff, up to 5 attempts. Because the Kubernetes log API only supports "give me the last N seconds" (not "give me everything after this exact timestamp"), a reconnect necessarily re-fetches a small overlapping window — squarepeg deduplicates by timestamp so you don't see repeated lines.
• squarepeg's own messages never mix with container output: everything the container writes goes to your stdout; every status/warning/diagnostic message from squarepeg itself goes to stderr. This means squarepeg run alpine cat bigfile.txt > out.txt captures exactly the container's output, nothing else.

6. Wait for completion, extract the exit code

Once the container reaches Succeeded/Failed, squarepeg reads its final container status and extracts terminated.exitCode (matched by container name, not by list position — relevant if passthrough config adds sidecar containers).

Situation	Exit code squarepeg reports
Container terminated normally	Its own exit code, verbatim
Container was OOM-killed	137, with the OOMKilled reason additionally printed to stderr
Container never actually terminated (evicted, pod deleted out from under squarepeg, etc.)	125 — "the runner itself failed", matching docker run's own convention (as opposed to a code the container chose)

7. Clean up

Situation	Behaviour
Default	The Pod/Job is deleted once the exit code has been captured. Job deletion uses propagationPolicy: Background explicitly, so the child pod isn't orphaned. Deletion tolerates a 404 (already gone) without erroring.
--keep	Nothing is deleted, and it is labelled (squarepeg.io/keep) so future runs' orphan sweeps skip it too. squarepeg prints the exact kubectl describe/kubectl logs commands to inspect it afterwards.
Job's ttlSecondsAfterFinished	A backstop for completed Jobs, independent of squarepeg's own cleanup — if squarepeg itself is kill -9'd before it can delete anything, the cluster cleans up the Job on its own after this TTL elapses. Bare Pods have no such mechanism at all, which is exactly why the orphan sweep below exists.
Orphan sweep	Runs on every invocation, alongside (never instead of) the delete above — see below.

The orphan sweep

Beyond deleting this invocation's own resource, every run also opportunistically sweeps the namespace for other squarepeg-managed Pods/Jobs left behind by a previous invocation that crashed or was killed before it could clean up after itself (machine died, kill -9, a network partition). Two independent categories, because a resource that never managed to start looks nothing like one that finished and wasn't deleted:

• Finished but not deleted — phase Succeeded/Failed, older than orphan_sweep_min_age (default 300s) since it finished. Never Running/Pending/Unknown at any age.
• Never managed to start — phase Pending with a container stuck on one of the fail-fast reasons above (a typo'd image reference being the common cause), older than orphan_sweep_min_age since it was created. This category exists because such a pod never reaches a terminal phase — the kubelet retries the pull forever, and a bare Pod has no backoffLimit/TTL to stop it — so without the sweep it would sit in the namespace indefinitely. A pod that's merely slow to start (a large image pulling normally, an unbound PVC, pending scheduling) has no fail-fast reason and is never swept, however long it sits there.
• Jobs are judged by their own Complete/Failed condition (or, for a Job whose child pod is stuck per the category above with no condition yet — e.g. backoffLimit: 0 never records a failure — by that child's state) and deleted with Background propagation so child pods cascade. Job-owned pods are never deleted directly.

Key properties:

• Scoped to your own resources only — the sweep's label selector matches squarepeg.io/created-by against your own username, enforced server-side, so a teammate's resource in a shared namespace is never even returned by the apiserver, let alone touched.
• --keep'd resources are exempt, via the squarepeg.io/keep label.
• Best-effort and cannot affect this run's own exit code. A sweep failure (a transient API error, insufficient RBAC, anything) is logged to stderr and swallowed — it never changes the outcome of the run it rode along with.
• Orthogonal to --keep/cleanup: false — a kept run still sweeps other orphans; these are independent decisions about independent resources.
• Configurable via orphan_sweep (bool, default true) and orphan_sweep_min_age (seconds, default 300) — see Configuration. No CLI flag exists for either; the sweep needs list permission on pods and Jobs in the namespace.

Interrupt handling (Ctrl-C)

squarepeg installs handlers for SIGINT/SIGTERM for the duration of a run, implementing a two-stage interrupt:

Press	Effect
First Ctrl+C	Stops log streaming, waits for the exit-code extraction to finish, deletes the resource (unless --keep) and runs the orphan sweep, and exits 130. Prints a message telling you a second Ctrl+C will leave it running.
Second Ctrl+C	Abandons cleanup immediately — the Pod/Job is left running in the cluster, and the orphan sweep is skipped too. Prints its name so you can find and manage it manually (kubectl describe/kubectl delete).

Exit code on interrupt is always 130 (128 + SIGINT), the standard shell convention.

Dry run

Two flavours, both stopping short of creating anything real, but differing in what they check and what they print.

Client-side (--dry-run)

squarepeg run --dry-run --cpus 0.5 -m 512m alpine echo hi

Renders the fully-merged manifest (CLI flags + all config layers + passthrough, all applied) as YAML to stdout, and exits 0 — without creating anything and without requiring a reachable cluster at all. The only thing that needs to succeed is local kubeconfig parsing for namespace resolution, and even that degrades gracefully if it fails.

This is the fastest way to check "did my config actually produce what I expected" before committing to a real run against the cluster.

Server-side (--dry-run-server)

squarepeg run --dry-run-server --cpus 0.5 -m 512m alpine echo hi

Builds the exact same manifest --dry-run would print, but instead of just rendering it locally, submits it to the apiserver as a server-side dry run (dryRun=All) — the same connect-to-the-cluster step a real run goes through, including the kubeconfig/in-cluster fallback, honouring --context/-n, and printing the "using context X, namespace Y" banner to stderr (suppressed by --quiet).

The apiserver runs its full validation, defaulting, and admission/mutating webhook pipeline against the manifest, then discards it without persisting anything. squarepeg prints the apiserver's response object (not the client-rendered manifest) as YAML to stdout — this is the whole point of the flag: you see fields the apiserver itself added or changed (a resolved metadata.namespace, spec.dnsPolicy, a webhook-injected sidecar or label, etc.) that plain --dry-run can never show you, since it never leaves your machine.

Because nothing is actually created, none of the steps below this one run: no watch for startup, no log stream, no wait for completion, no exit-code extraction, no cleanup, and no orphan sweep. There's also nothing to interrupt cleanly, so no two-stage Ctrl-C handler is installed — a Ctrl-C here behaves like an ordinary Python KeyboardInterrupt.

	--dry-run	--dry-run-server
Requires a cluster connection	No	Yes
What's printed	The client-rendered manifest	The apiserver's response object
Validation performed	None — squarepeg's own manifest construction only	Full apiserver validation, RBAC, ResourceQuota, admission/mutating webhooks
Exit code on rejection	N/A (never contacts a cluster)	125, same hints as a real create failure

The two flags are mutually exclusive — pass one or the other, not both.

--dry-run-server is the way to catch a ResourceQuota rejection, an RBAC denial, or an admission-webhook rejection before committing to a real run, once --dry-run has already confirmed the manifest looks right on the client side.