Skip to content

Releases

labctl 1.11.0

Choose a tag to compare

@github-actions github-actions released this 12 Nov 09:55

SnowOps Labs 1.11.0

This release adds team game days: one lab machine, a facilitator who breaks it on purpose, and a team that finds and fixes the fault together. It also moves the project to a new GitHub organization, which changes the Go module path (see Upgrade notes).

Run a game day for a team

A facilitator injects a fault silently, the team diagnoses it with kubectl, Grafana and the pager, and the fault's own detection check decides whether it is really fixed. Start with the new guide to running a game day.

  • Shared team server. labctl ui --tls-self-signed serves the web UI over HTTPS with a certificate made for the machine and kept in the lab state, and prints its fingerprint so people can verify it before accepting the browser warning. The steps for logins, kubectl access and dashboards are in the team server reference.
  • Roles that mean something. With LABCTL_AUTH=true, only an operator can inject a fault or end a round with the escape hatch. A participant who tries gets 403 forbidden_role, and the web UI shows each person only the actions their role allows, with the signed-in name and role in the header.
  • Logins survive a restart. Sessions are stored in the lab database as hashes, so restarting labctl ui no longer signs everyone out.
  • Results you can use. labctl results export writes recorded runs as JSON or CSV, filtered by kind and time, with a documented set of fields. labctl results card prints a Markdown card for a round (or an SVG with -o svg) that says it is self-reported. The leaderboard can be limited to today or the last four hours, and it no longer counts a round ended with the escape hatch as a fix.
  • Silent rounds stay silent. A silent fault no longer names itself on Prometheus' Alerts page, and a refused second inject does not name the active fault.

Also new

  • labctl scenario list -o json and labctl incident list -o json print an array you can pipe to jq.
  • Traffic start, stop and status now run as durable runs, so a traffic operation that is interrupted can be recovered instead of left half done.
  • Recovering an interrupted run is safer: a live run is never cancelled by another command starting, runs whose owner has gone are recoverable, and unused lock files are cleaned up.
  • The README now leads with the lab itself, and there are catalogs of every fault and scenario. Shell tab-completion is documented.

Fixes

  • k3d labs. The load balancer no longer restart-loops after a Docker restart when a node was replaced, and each node reserves an even share of Docker's memory so a three-node lab on an 8 GB VM no longer offers the scheduler about 25 GB. After a Docker VM restart the inotify limit is raised back automatically, and labctl status and labctl doctor report a low limit and nodes that are NotReady while their container runs.
  • First labctl init. The harmless "couldn't get resource list" discovery messages from helm and kubectl are hidden, so a first install no longer scrolls dozens of lines that read as failure.
  • cost-right-sizing sizes its inflated requests to what the lab has free, so it no longer leaves the pod Pending on a 2 CPU lab.
  • Incidents view. If an inject or resolve request outlasts the browser's 30 second timeout, the page keeps following it and reports the result instead of showing "Inject failed" for a fault that went in.
  • Build. labctl is built with Go 1.26.11, which fixes the reachable net/http vulnerability GO-2026-6859, on top of the earlier x509 fix.

Upgrade notes

  • The repository moved to snowops-oss/snowopslabs. GitHub redirects the old address for browsing, git clone and downloads, so existing clones keep working. Point origin at the new address: git remote set-url origin https://github.com/snowops-oss/snowopslabs.git.
  • The Go module path changed to github.com/snowops-oss/snowopslabs. If you import pkg/checks, pkg/scenario or pkg/extension, change your imports and your go.mod. Go does not follow GitHub redirects, so the old path will not resolve to this release.
  • JSON timestamps with no value are left out of the incident, lab and platform status responses (for example firstCheckedAt and since), instead of appearing as 0001-01-01T00:00:00Z. A client that reads those fields should treat a missing field as unset.
  • The lab database gains a sessions table through an automatic migration the first time you start this version. Migrations only move forward, so keep a copy of ~/.snowops/snowops.db if you may want to go back to an older labctl.

Known limitations

  • A lab brought up with labctl init shows Loki and Tempo datasources in Grafana that fail their health check until the logging and tracing providers are installed (#212).
  • The k3d fixes above are covered by tests and shell checks, but this release was walked through on a kind lab, not on a k3d lab.
  • stray-pod-behind-service is still marked unverified in the incident catalog.

Contributors

Thank you to everyone who contributed to this release:

  • @kimjooyoon: the durable run engine, owner recovery and traffic operations as runs (#144, #153)
  • @lui01212: the k3d load balancer, node memory and inotify fixes, the install output cleanup, the cost-right-sizing fix and the Incidents view timeout fix (#166, #172, #189, #190, #192, #208)
  • @jay23606: role-aware incident actions in the UI and the zero-time JSON fix (#206, #207)
  • @amandeavor: shell tab-completion documentation (#149)

Full change list: v1.10.0...v1.11.0

labctl 1.10.0

Choose a tag to compare

@github-actions github-actions released this 29 Oct 13:07

Changelog

labctl 1.9.0

Choose a tag to compare

@github-actions github-actions released this 28 Oct 07:28

Highlights

Content for Course 1, "Kubernetes in Production":

  • A new incident, stray-pod-behind-service: a hand-run pod carrying the Deployment's labels joins the Service and fails a steady share of requests, while every pod is Running and Ready and the endpoints list is full. The app's own error panel stays flat (the stray pod is never scraped), so its alert reads the client-side failure ratio from k6. make-the-slo-green now uses it as its setup.
  • Every incident is verified by the nightly cluster suite: cpu-throttling, dns-egress-blocked, liveness-probe-kill, missing-secret-key and pending-unschedulable join the rest.
  • cluster-upgrade-drill rolls workers onto v1.33.13-k3s1, a newer patch than the v1.33.6 k3d v5.8.3 creates the lab on, so the drill upgrades for real. roll-node.sh now refuses a patch downgrade as well as a minor one.

Fixes and improvements:

  • labctl platform status lists what the cluster holds.
  • backup-restore-drill leaves kube-root-ca.crt out of its backup.
  • install.sh prints a PATH command that is safe to paste into the login shell.
  • A codespace on main builds labctl instead of failing its setup.
  • The CLI and dashboard point to the newsletter signup.

Upgrading

Merge origin/stable as usual. Nothing needs rebuilding.

Changelog

labctl 1.8.0

Choose a tag to compare

@github-actions github-actions released this 18 Oct 18:20

Highlights

Content for Course 1, "Kubernetes in Production":

  • Five new incidents, each diagnosable with get, describe, logs, events, top and rollout: pending-unschedulable (requests decide scheduling, not usage), liveness-probe-kill (the kubelet kills a healthy app), missing-secret-key (a rotated key no new pod can find), cpu-throttling (a CPU limit that adds latency, not errors) and dns-egress-blocked (an egress policy that forgot port 53).
  • on-call-shift, the course's final challenge: two faults from different categories land together, unnamed. Challenges can now set up several faults (setup.refs) and keep them unnamed (silent: true).
  • go-api gains UPSTREAM_URL, STARTUP_DELAY_SECONDS and CPU_WORK_MS, all off by default, and reads API_TOKEN from its own Secret.
  • Pod Resources gains Pods waiting for a node and CPU throttled panels.

Fixes:

  • cluster-upgrade-drill's replacement node gets every build of the lab's apps, so a later scenario (backup-restore-drill) can start on it.
  • Challenge grading passes each fault's target to its detection check; a fault aimed at anything but the default app is now graded correctly.
  • A cancelled run ends when its script does and leaves no process behind; following a run no longer loses output written just before it ended.

Upgrading

Merge origin/stable as usual, then rebuild go-api once, since the new image and chart are what missing-secret-key and dns-egress-blocked run on:

labctl app build go-api && labctl app deploy go-api

Both faults refuse with that command until you do.

Changelog

  • 19ce03f: Merge pull request #102 from sagar2395/ccr-8e694dec-x5pbks (@sagar2395)
  • 2cc34cf: Merge pull request #104: five Course 1 incidents, and go-api switches for them (@sagar2395)
  • 509038b: Merge pull request #105: on-call-shift, two faults at once, and the engine change behind it (@sagar2395)
  • 99e253f: Merge pull request #106: the six Course 1 scenarios on the default lab (@sagar2395)
  • 981c00a: feat(challenge): set up several faults together, and keep them unnamed (@sagar2395)
  • 541a5fc: feat(challenges): on-call-shift, two faults at once with no names (@sagar2395)
  • 5b55483: feat(go-api): an upstream, a slow start and per-request CPU work, all off (@sagar2395)
  • 4436697: feat(incidents): cpu-throttling, a CPU limit set to save cost (@sagar2395)
  • 8bbbc46: feat(incidents): dns-egress-blocked, a security policy that forgot port 53 (@sagar2395)
  • 2418bde: feat(incidents): liveness-probe-kill, a probe path the app does not serve (@sagar2395)
  • b7f8a45: feat(incidents): missing-secret-key, a rotated key no new pod can find (@sagar2395)
  • add1585: feat(incidents): pending-unschedulable, replicas no node can hold (@sagar2395)
  • 9d21136: fix(challenge): a silent challenge's failing submit names no fault (@sagar2395)
  • bb577db: fix(cli): a followed run no longer loses output it wrote just before it ended (@sagar2395)
  • 4d169ef: fix(cluster-upgrade-drill): the replacement node gets every build of the lab's apps (@sagar2395)
  • 0a7b3aa: fix(dns-egress-blocked): grade only an app that really calls its upstream (@sagar2395)
  • 0c0582f: fix(toolchain): a cancelled run ends when its script does, and leaves nothing behind (@sagar2395)
  • ecd15c3: release: v1.8.0 (@sagar2395)

labctl 1.7.0

Choose a tag to compare

@github-actions github-actions released this 17 Oct 06:40

Changelog

What's Changed

  • release: v1.6.0 by @sagar2395 in #97
  • docs(cli): document config precedence and lab discovery (fixes #67) by @lui01212 in #99
  • docs: human punctuation, a writing guide, and hosted labs as a long-term goal by @sagar2395 in #100
  • feat: run the lab in GitHub Codespaces; fix blank .env lines hiding the profile's value by @sagar2395 in #101

Full Changelog: v1.6.0...v1.7.0

labctl 1.6.0

Choose a tag to compare

@github-actions github-actions released this 15 Oct 19:50

Changelog

What's Changed

  • ci(release): make release opens the release PR; merging it tags and builds by @sagar2395 in #64
  • docs(cli): add WSL2 guidance to doctor and ui reference (fixes #70) by @lui01212 in #75
  • docs(runbooks): add task-oriented situation index to README (fixes #69) by @lui01212 in #79
  • docs(wsl): consolidate WSL2 troubleshooting guide by symptom (fixes #74) by @lui01212 in #80
  • Replace viper in LoadAppConfig with the existing env parser by @Ashutosh-Panda2004 in #76
  • Add schema-versus-doc check for scenario-schema.md by @Ashutosh-Panda2004 in #77
  • docs(backlog): four problems found recording the macOS setup video by @sagar2395 in #88
  • Return a copy from Engine.Get so it never writes to the cached scenario by @Ashutosh-Panda2004 in #86
  • build: pin toolchain go1.26.9; CI installs the newest 1.26 patch by @sagar2395 in #90
  • fix(platform): clear a helm release stuck in a pending state by @sagar2395 in #87
  • fix(platform): traefik binds the ingress node's host ports on kind; the nightly e2e passes again by @sagar2395 in #91
  • fix(incidents): consolidated incident, dashboard and traffic fixes by @sagar2395 in #92
  • Claude/snowopslabs readme video gsfc0t by @sagar2395 in #93
  • Test strategy (RFC 0001): gates, content tests, cluster suite, and the bugs they found by @sagar2395 in #95
  • fix: the bugs the first nightly cluster-suite and test-health runs found by @sagar2395 in #96

New Contributors

Full Changelog: v1.5.0...v1.6.0

labctl 1.5.0

Choose a tag to compare

@github-actions github-actions released this 29 Sep 13:49
7f43209

Changelog

  • e74f3d4: Merge pull request #61 from sagar2395/refactor/go-code-quality (@sagar2395)
  • 618e5c6: Merge pull request #62 from sagar2395/feat/onboarding-honest-failures (@sagar2395)
  • 7f43209: Merge pull request #63 from sagar2395/ccr-48094db0-chvhux (@sagar2395)
  • b5db72e: Refractor go code (@sagar2395)
  • 378f9e9: ci(onboarding): install and init the lab on a fresh Ubuntu runner (@sagar2395)
  • 1ba60ab: docs(onboarding): a user-only README, per-OS guides, resources and troubleshooting (@sagar2395)
  • 8558c65: feat(capacity): admit scenarios and faults only when the lab has room (@sagar2395)
  • a7a293c: feat(capacity): measured footprints, advice that frees what it names, agents given back (@sagar2395)
  • 3a8f51a: feat(ingress): labctl init moves an existing lab to *.localhost URLs (@sagar2395)
  • 5ea9a3d: feat(ingress): per-lab *.localhost URLs that reach this lab from this machine (@sagar2395)
  • a82671a: feat(install): learners clone the stable branch and install the matching labctl (@sagar2395)
  • c57fc52: feat(onboarding): fail loudly on an undersized or unreachable lab (@sagar2395)
  • f9fb666: feat(onboarding): install tools per OS without sudo (@sagar2395)
  • 65688e7: feat(onboarding): ports follow the cluster; init keeps an existing lab (@sagar2395)
  • 9c91d5f: feat(release): release acceptance — walk the live lab as a learner before a release (@sagar2395)
  • 914d396: feat(release): ship the lab content with labctl and install both in one step (@sagar2395)
  • 36d6eac: feat(state): keep lab state per cluster outside the clone; find the lab from anywhere (@sagar2395)
  • 5438cbc: feat: Refractor comments (@sagar2395)
  • f391ebf: fix(api,build): unmatched /api/v2 requests answer problem+json; cli-install records the lab (@sagar2395)
  • 9aaaa6e: fix(content): hints and scripts work from any directory (@sagar2395)
  • 97c7cdd: fix(content): no hosts-file step in scenarios; TLS check works on *.localhost (@sagar2395)
  • 44484b0: fix(init): repair Grafana that cannot query Prometheus; keep move-domain apply-safe (@sagar2395)
  • 9909899: fix(install): name the real problem when a release has no bundled content (@sagar2395)
  • 4bea7fb: fix(k3d): bring back a lab whose server came back with another IP (@sagar2395)
  • ff44188: fix(k3d): find added agents by the name k3d gives them (@sagar2395)
  • ad62be1: fix(lint): use strings.Cut for the helm repo name (@sagar2395)
  • f954491: fix(onboarding): ignore evicted pods on init; let WSL users choose native Docker (@sagar2395)
  • 21a55b2: fix(platform): say how to pick a provider when a category has several (@sagar2395)
  • 862f07b: fix(prereqs): stop on missing platform components, install them with --deploy-prereqs (@sagar2395)
  • fbcc9b8: fix(wsl): end-to-end fixes from a full run on Windows (WSL2) (@sagar2395)
  • 5747a48: refactor(go): apply cc-skills-golang and the comment rule to the branch (@sagar2395)
  • 2d89f34: release: v1.5.0 (@claude)
  • 653b6ee: test(shell): keep the runner's own kubectl out of the setup-tools install tests (@sagar2395)

labctl 1.4.0

Choose a tag to compare

@sagar2395 sagar2395 released this 21 Sep 17:12
0ae4911

What's Changed

  • fix(ci): make the kind e2e wait for rollouts and walk the promotion d… by @sagar2395 in #57
  • feat: testing and improving scenarios by @sagar2395 in #58
  • Feature/app scenario decouple by @sagar2395 in #59

Full Changelog: v1.3.0...v1.4.0

labctl 1.3.0

Choose a tag to compare

@sagar2395 sagar2395 released this 04 Sep 19:39
6a9ca7f

What's Changed

  • fix(observability): make the whole metrics/logs/traces pipeline actually deliver data by @sagar2395 in #45
  • feat: documentation as source of truth; remove the unversioned /api by @sagar2395 in #50
  • Feature/comment cleanup by @sagar2395 in #52
  • docs: track the authoring skills by @sagar2395 in #53
  • feat: end-to-end review harnesses for scenarios, incidents and learning by @sagar2395 in #55

Full Changelog: v1.2.0...v1.3.0

labctl 1.2.0

Choose a tag to compare

@github-actions github-actions released this 02 Sep 06:47
23e75e5

Changelog

  • 38e1840: Fixing backup restore and cost right sizing ( <>)
  • 23e75e5: Merge pull request #42 from sagar2395/claude/pr-main-issue-linking-9homur (@sagar2395)
  • 987516c: fix(ci): use portable date -r instead of stat -c for backup age (@claude)