Execution Lifecycle

Execution Lifecycle

What actually happens, in order, between running squarepeg run and getting your prompt back.

1. Parse and resolve

1. Docker-style flags are parsed. Everything after IMAGE — including lookalike flags — is passed through untouched as the container command (ignore_unknown_options/allow_interspersed_args in click terms).
2. Every config layer is loaded, ${VAR} interpolated, validated, and deep-merged (see Configuration).
3. CLI flags are applied on top; each explicitly-set flag is recorded as a claim on a specific manifest field (e.g. /spec/containers/[name=main]/env/[name=FOO]) so that config passthrough can never silently override something you set on the command line, while everything you didn't set is still open to passthrough.

2. Build the manifest

The Pod or Job is built as a plain Python dict (not a typed Kubernetes model) — this is what makes --dry-run output exactly what would be sent to the apiserver, with no serialization round-trip, and makes arbitrary passthrough config trivial to merge in.

Naming

--name given	Validated as a lowercase RFC1123 label (alphanumeric + -, ≤63 chars) and used verbatim.
--name omitted	Generated as squarepeg-<image-slug>-<6 random hex chars>, e.g. squarepeg-alpine-a1b2c3. The image reference has its registry host, tag, and digest stripped first, so registry.example.com:5000/foo/bar:tag becomes bar.

Labels

Every Pod/Job squarepeg creates carries:

Label	Value
app.kubernetes.io/managed-by	squarepeg
squarepeg.io/run-id	A random UUID, unique per invocation
squarepeg.io/created-by	Your local username (sanitized to a valid label value)

These make orphaned resources identifiable later (e.g. via kubectl get pods -l app.kubernetes.io/managed-by=squarepeg) if a run is ever interrupted before cleanup.

Pod vs Job

	Pod (--mode pod, default)	Job (--mode job)
restartPolicy	Never	Never (on the pod template)
Retries	None	backoffLimit: 0 (also none, by default)
Extra fields	—	completions: 1, parallelism: 1, ttlSecondsAfterFinished (a backstop cleanup in case squarepeg itself is killed before it can delete the resource)
Naming the actual pod	The name you gave/generated	Kubernetes generates the pod name; squarepeg discovers it via the job-name label

kubernetes.spec/kubernetes.metadata passthrough in config is always expressed as a Pod spec/metadata, even in Job mode — squarepeg re-homes it under spec.template.{spec,metadata} automatically so the same config file works in either mode without rewriting.

3. Connect to the cluster

squarepeg resolves credentials the same way kubectl does, then falls back to the pod's own service account if it's running inside a cluster itself:

Order	Source	When it applies
1	Kubeconfig ($KUBECONFIG, then ~/.kube/config, honouring --context)	The default — squarepeg on your laptop/CI runner talking to a remote cluster.
2	In-cluster config (the pod's own service account token, mounted at /var/run/secrets/kubernetes.io/serviceaccount/)	Only tried if no --context was given and the kubeconfig load failed outright — i.e. squarepeg is itself running as a container (inside Argo/Tekton/a custom operator/etc.) and needs to spawn sibling pods/jobs using its own pod's permissions.

Two things worth knowing:

• An explicit --context that fails never falls back to in-cluster. If you asked for a specific context and it's not there, that failure is surfaced directly — it's not silently masked by an in-cluster attempt you didn't ask for.
• Namespace resolution differs slightly in-cluster. There's no "context" concept once running in-cluster, so the equivalent fallback is the namespace projected into the pod at /var/run/secrets/kubernetes.io/serviceaccount/namespace, read in the same precedence slot a kubeconfig context's namespace would otherwise occupy: explicit -n/--namespace beats it, and it beats squarepeg's own default fallback.

squarepeg prints which one it resolved to stderr on startup (context name — or in-cluster — plus namespace), unless --quiet.

4. Create and wait for start

squarepeg creates the resource, then watches it with a bounded timeout (--timeout, default 300s) that covers only startup — image pull, scheduling, container creation.

Fails fast (doesn't wait out the full timeout) if the container's waiting-state reason is one of:

ImagePullBackOff · ErrImagePull · InvalidImageName · CreateContainerConfigError
CreateContainerError · RunContainerError · CrashLoopBackOff

If the pod is still Pending and something's blocking it (insufficient resources, an unschedulable node selector, an unbound PVC), squarepeg surfaces the relevant Kubernetes events to stderr — the difference between "it just hung" and "0/12 nodes available: insufficient memory".

Once the container reaches Running (or has already terminated by the time squarepeg checks), startup is considered complete and the timeout no longer applies.

5. Stream logs

Log streaming runs in a background thread, concurrently with waiting for the container to actually finish:

• Output is written as raw bytes straight to your terminal's stdout — not decoded/re-encoded line by line — so partial lines, ANSI colour codes, and \r progress bars all come through correctly.
• Reconnect on drop: if the log stream disconnects mid-run (apiserver restart, proxy timeout), squarepeg reconnects automatically, with exponential backoff, up to 5 attempts. Because the Kubernetes log API only supports "give me the last N seconds" (not "give me everything after this exact timestamp"), a reconnect necessarily re-fetches a small overlapping window — squarepeg deduplicates by timestamp so you don't see repeated lines.
• squarepeg's own messages never mix with container output: everything the container writes goes to your stdout; every status/warning/diagnostic message from squarepeg itself goes to stderr. This means squarepeg run alpine cat bigfile.txt > out.txt captures exactly the container's output, nothing else.

6. Wait for completion, extract the exit code

Once the container reaches Succeeded/Failed, squarepeg reads its final container status and extracts terminated.exitCode (matched by container name, not by list position — relevant if passthrough config adds sidecar containers).

Situation	Exit code squarepeg reports
Container terminated normally	Its own exit code, verbatim
Container was OOM-killed	137, with the OOMKilled reason additionally printed to stderr
Container never actually terminated (evicted, pod deleted out from under squarepeg, etc.)	125 — "the runner itself failed", matching docker run's own convention (as opposed to a code the container chose)

7. Clean up

Situation	Behaviour
Default	The Pod/Job is deleted once the exit code has been captured. Job deletion uses propagationPolicy: Background explicitly, so the child pod isn't orphaned. Deletion tolerates a 404 (already gone) without erroring.
--keep	Nothing is deleted. squarepeg prints the exact kubectl describe/kubectl logs commands to inspect it afterwards.
Job's ttlSecondsAfterFinished	A backstop, independent of squarepeg's own cleanup — if squarepeg itself is kill -9'd before it can delete anything, the cluster cleans up the Job on its own after this TTL elapses.

Interrupt handling (Ctrl-C)

squarepeg installs handlers for SIGINT/SIGTERM for the duration of a run, implementing a two-stage interrupt:

Press	Effect
First Ctrl+C	Stops log streaming, waits for the exit-code extraction to finish, deletes the resource (unless --keep), and exits 130. Prints a message telling you a second Ctrl+C will leave it running.
Second Ctrl+C	Abandons cleanup immediately — the Pod/Job is left running in the cluster. Prints its name so you can find and manage it manually (kubectl describe/kubectl delete).

Exit code on interrupt is always 130 (128 + SIGINT), the standard shell convention.

Dry run

squarepeg run --dry-run --cpus 0.5 -m 512m alpine echo hi

Renders the fully-merged manifest (CLI flags + all config layers + passthrough, all applied) as YAML to stdout, and exits 0 — without creating anything and without requiring a reachable cluster at all. The only thing that needs to succeed is local kubeconfig parsing for namespace resolution, and even that degrades gracefully if it fails.

This is the fastest way to check "did my config actually produce what I expected" before committing to a real run against the cluster.