diff --git a/README.md b/README.md index 4cf03df..22579fa 100644 --- a/README.md +++ b/README.md @@ -6,9 +6,9 @@ The customer-facing CLI for the tracebloc declarative ingestion path. Wraps the ## Status -**v0.3.0 is released** — the latest stable [release](https://github.com/tracebloc/cli/releases/latest), cut from `develop`. It builds on v0.2.0's guided `dataset push` and `dataset rm` with a new `dataset list` command plus home-screen / output polish (clearer copy, guided-first framing). The binary implements `version`, `completion`, `ingest validate`, `cluster info`, and the full `dataset push` / `dataset list` / `dataset rm` flow — local schema validation, cluster discovery, data staging, submission, and Job watching, end to end. +**v0.3.0 is released** — the latest stable [release](https://github.com/tracebloc/cli/releases/latest), cut from `develop`. It builds on v0.2.0's guided `data ingest` and `dataset rm` with a new `dataset list` command plus home-screen / output polish (clearer copy, guided-first framing). The binary implements `version`, `completion`, `data validate`, `cluster info`, and the full `data ingest` / `dataset list` / `dataset rm` flow — local schema validation, cluster discovery, data staging, submission, and Job watching, end to end. -`dataset push` covers **9 of 10 task categories**: `image_classification`, `object_detection`, `keypoint_detection`, `text_classification`, `masked_language_modeling`, `tabular_classification`, `tabular_regression`, `time_series_forecasting`, and `time_to_event_prediction`. `semantic_segmentation` is pending mask-sidecar support upstream ([data-ingestors#136](https://github.com/tracebloc/data-ingestors/issues/136)); `instance_segmentation` is not yet implemented. +`data ingest` covers **9 of 10 task categories**: `image_classification`, `object_detection`, `keypoint_detection`, `text_classification`, `masked_language_modeling`, `tabular_classification`, `tabular_regression`, `time_series_forecasting`, and `time_to_event_prediction`. `semantic_segmentation` is pending mask-sidecar support upstream ([data-ingestors#136](https://github.com/tracebloc/data-ingestors/issues/136)); `instance_segmentation` is not yet implemented. The release pipeline ships [`v0.3.0`](https://github.com/tracebloc/cli/releases/latest) as **cosign-signed, multi-arch binaries** — Linux (`amd64`, `arm64`, `386`, `arm`), macOS (`amd64`, `arm64`), and Windows (`amd64`, `arm64`) — each with `SHA256SUMS` and the install scripts. Install via [Customer experience](#customer-experience) or [build from source](#building-from-source). (A Homebrew tap and the `install.tracebloc.io` vanity URL are later follow-ups; the GitHub release URL serves installs today.) @@ -24,7 +24,7 @@ The CLI is a sibling interface to the chart, not a replacement. Both translate t Customer interfaces (pick one or many): ┌─────────────────────────────────────────────────┐ │ Web UI Studio for clicking through │ ← future -│ CLI `tracebloc dataset push ./data` │ ← this repo +│ CLI `tracebloc data ingest ./data` │ ← this repo │ Python SDK `IngestConfig(...).submit()` │ ← future │ K8s CRD `kubectl apply` Ingestion CR │ ← future │ Helm chart tracebloc/ingestor │ ← today @@ -50,9 +50,10 @@ The protocol — the v1 schema + the POST endpoint — is the stable point. Ever curl -fsSL https://github.com/tracebloc/cli/releases/latest/download/install.sh | sh # Install — Windows (PowerShell) irm https://github.com/tracebloc/cli/releases/latest/download/install.ps1 | iex +# The installer also creates a short alias: tb works everywhere tracebloc does. # Per dataset -tracebloc dataset push ./my-data \ +tracebloc data ingest ./my-data \ --table cats_dogs_train \ --category image_classification \ --intent train \ @@ -108,15 +109,15 @@ All v0.1 phases are merged: | Phase | Ticket | What | Status | |---|---|---|---| | 0 | [#148](https://github.com/tracebloc/client/issues/148) | Repo bootstrap + Go module + CI + `tracebloc version` | ✅ | -| 1 | [#149](https://github.com/tracebloc/client/issues/149) | Embed `ingest.v1.json` + `tracebloc ingest validate ` (local-only) | ✅ | +| 1 | [#149](https://github.com/tracebloc/client/issues/149) | Embed `ingest.v1.json` + `tracebloc data validate ` (local-only) | ✅ | | 2 | [#150](https://github.com/tracebloc/client/issues/150) | Cluster discovery + ingestor SA token via TokenRequest | ✅ | | 3 | [#151](https://github.com/tracebloc/client/issues/151) | Stage data into the shared PVC via ephemeral Pod | ✅ | | 4 | [#152](https://github.com/tracebloc/client/issues/152) | Submit to jobs-manager + watch ingestor Job + summary | ✅ | | 5 | [#153](https://github.com/tracebloc/client/issues/153) | GitHub Releases + install.sh distribution (Homebrew tap deferred) | ✅ — [`v0.1.0`](https://github.com/tracebloc/cli/releases/tag/v0.1.0) released (stable, 8-platform) | -Beyond the original phases, `dataset push` was widened from image-classification-only to 9 of 10 modalities, and the test suite gained unit-coverage wins plus a kind-based integration harness for the real-I/O seams. +Beyond the original phases, `data ingest` was widened from image-classification-only to 9 of 10 modalities, and the test suite gained unit-coverage wins plus a kind-based integration harness for the real-I/O seams. -**v0.2.0** added a friendlier guided `dataset push` and `dataset rm` on the home screen (#44, #47). **v0.3.0** added the `dataset list` command (#53) plus home-screen / output-spacing polish and feedback-copy refinements (#52, #56). **Next:** cloud-source ingestion (S3/GCS/HTTPS) for datasets above the 1 GiB local cap; `semantic_segmentation` ([data-ingestors#136](https://github.com/tracebloc/data-ingestors/issues/136)) and `instance_segmentation`. Smaller follow-ups are tracked as [open issues](https://github.com/tracebloc/cli/issues). +**v0.2.0** added a friendlier guided `data ingest` and `dataset rm` on the home screen (#44, #47). **v0.3.0** added the `dataset list` command (#53) plus home-screen / output-spacing polish and feedback-copy refinements (#52, #56). **Next:** cloud-source ingestion (S3/GCS/HTTPS) for datasets above the 1 GiB local cap; `semantic_segmentation` ([data-ingestors#136](https://github.com/tracebloc/data-ingestors/issues/136)) and `instance_segmentation`. Smaller follow-ups are tracked as [open issues](https://github.com/tracebloc/cli/issues). Epic: [tracebloc/client#147](https://github.com/tracebloc/client/issues/147). diff --git a/docs/rfcs/0001-cli-auth-and-client-provisioning.md b/docs/rfcs/0001-cli-auth-and-client-provisioning.md new file mode 100644 index 0000000..745f646 --- /dev/null +++ b/docs/rfcs/0001-cli-auth-and-client-provisioning.md @@ -0,0 +1,1306 @@ +# RFC 0001 — Browser-based auth & one-command client provisioning + +> **Status: ACCEPTED — implemented.** The design in this RFC shipped in +> **CLI v0.4.0** ([cli#107](https://github.com/tracebloc/cli/pull/107)); the +> tracking epic ([cli#54](https://github.com/tracebloc/cli/issues/54)) is closed. +> This document is retained as the design-of-record. The Phase 2 items in §11 +> (server-side token revoke [backend#887], short-lived auto-refreshing tokens, +> `client rotate`, atomic fleet enrollment) remain tracked but out of the v0.4.0 +> scope. Owner: @saadqbal. Last updated: 2026-07-03. +> +> The revision history below is kept as the record of how the design converged; +> Rev 6–7 are accuracy fixes reconciling the doc with what actually shipped. +> +> **Rev 2 (2026-06-23)** folds in the code-grounded review on the tracking epic +> ([backend#830](https://github.com/tracebloc/backend/issues/830)) and a +> user-perspective teardown of the end-to-end CLI flow. Net change: the +> auth handshake turned out to be the *easy* half; the design now leads with the +> **client lifecycle on a machine** (§7), which is where the real bugs hide. Three +> product decisions are settled (§0). +> +> **Rev 3 (2026-06-23)** makes the **cluster the idempotency anchor** (1:1 +> client↔cluster, keyed on the `kube-system` UID) and closes the review's blocking +> gaps: cross-account adoption (§6.3/§7.2/R6), the existing-fleet `cluster_id` +> backfill via the heartbeat (§4.5/§10/R7), server-side `logout` revoke (§7.5/§9), +> and the orphan password-reset gate (§7.9). cluster-id + Q1 + Q5 are confirmed (§12). +> +> **Rev 4 (2026-06-23)** adds a hardening pass from an independent security + product +> review: bootstrap supply-chain (R8), audit trail (R9), multi-env config clobber +> (R10, a confirmed bug), version negotiation (R11), clean uninstall (R12), plus +> machine-credential revoke + authenticated `cluster_id` (§9), the silent-failure +> story (§8.5), adopt-keeps-namespace (§7.2), and delete-mid-experiment (§7.4). +> +> **Rev 5 (2026-06-23)** adds **Appendix C — the implementation-grade API & data +> contracts** (grounded in the shipped CLI client) to pin before parallel work, so +> the doc can be built from directly without a separate SDD. +> +> **Rev 6 (2026-06-23)** closes the two anchor residuals flagged on #96: `cluster_id` +> is set/backfilled by the **CLI/installer** (the kubeconfig-holder that can read the +> `kube-system` UID) — **not** the heartbeat, whose sender can't (§4.5/§10/C.4/R7); +> and §7.2's account-scoped check now gates **adoption itself**, so the live-release +> path can't bypass the cross-account `409` (R6). +> +> **Rev 7 (2026-06-25)** FR correction (connect-flow FR on dev, #877): `logout` +> server-side revoke reframed as **pending** — today logout is **local-only**; the +> server-side revoke needs the `POST /auth/revoke` endpoint (backend#887, not built) +> + a CLI `logout`→call (backend#845 shipped only the `revoke()` primitive) — +> §6.3/§7.5/§9/§13/C.6. The earlier "revokes server-side via #845" claim overstated it. +> +> **Rev 8 (2026-07-07)** product deviation on the two zero-prompt inputs (cli#137, +> installer UX v2 — decided by Lukas 2026-07-06): +> - **Name is `-NN`, not the hostname.** The auto-generated name is now +> `-NN` from the signed-in identity (fallback: email local-part), +> numbered per account — the hostname is neither stable nor account-scoped (§6.6/§7.7). +> - **Location is optional, not auto-detected.** With no `--location` the CLI omits +> it and the backend records the client with **no location** (an explicit "not set" +> state — backend#993), rather than auto-detecting a zone at provision time. The +> cloud-metadata auto-detect (`internal/geo`, cli#93) is **removed** — the silent +> path no longer detects, and the backend is the source of truth for valid zones +> (a bad `--location` surfaces as its real create error). §6.7/§7.7. +> +> The §6.6/§6.7/§7.7 bodies below are retained as the original design-of-record; the +> inline **Rev 8** callouts mark where the shipped behavior now differs. + +## 0. Decisions settled in this revision + +These were open forks between the first draft, the in-flight implementation, and +the review. They are now decided; the rest of the doc assumes them. + +| # | Decision | Choice | +|---|---|---| +| D1 | **Setup is silent / auto, not interactive.** | Common path asks **zero questions**: name = sanitized hostname, location = auto-detect, both *surfaced* in progress and correctable with flags — never prompted. (§6.7, §7.7, §8) *(Amended Rev 8 / cli#137: name = `-NN`; location optional & omitted when unset — see §6.6/§6.7 callouts.)* | +| D2 | **The machine credential is never shown.** | `client create` prints only name + status. The credential is written straight into the cluster secret (mode `0600`) + stored hashed in the backend, and never touches stdout, scrollback, the clipboard, or `~/.tracebloc`. Rotation = delete + recreate. (§7.1, §7.8, §9) | +| D3 | **Clients are referred to by a human handle, never a secret or backend id.** | The handle is the per-account-unique namespace **slug** (e.g. `munich-hospital-radiology`); bare `use` / `delete` open an arrow-key picker. The UUID / username / password are never displayed. (§7.1) | + +The cluster identifier (`kube-system` UID) and the two epic scope calls — **air-gap +out of scope** (Q1) and **one client per cluster / re-parenting deferred** (Q5) — +are now confirmed (§12). + +## 1. Summary + +Replace the current "go to the web UI, hand-create a client, copy a Client ID + +password, paste them into the installer" onboarding with a single flow: + +``` +sign in (browser) → (this machine names + locates itself automatically) → done +``` + +The human authenticates once in a browser (works even on a headless box over +SSH), the CLI provisions the client automatically and silently, the machine +credential is written straight into the cluster, and the installer proceeds. No +copied secrets, no separate visit to `/clients`, and — in the common case — no +prompts at all. + +## 2. Motivation + +Today (`client/scripts/install.sh`) the operator must: + +1. Sign up at `https://ai.tracebloc.io`. +2. Navigate to `https://ai.tracebloc.io/clients` and hand-create a client. +3. Copy a **Client ID + password** out of the web UI. +4. Paste both into the installer prompt — often over SSH, into a `curl | bash` + process. + +Problems: + +- **Context-switch + copy-paste of two long secrets**, frequently over SSH — + error-prone and intimidating for a junior operator. +- **The installer process sees the user's password.** For a piped-from-the-internet + installer this is a trust and phishing concern, and it blocks SSO/MFA/SAML — + table stakes for selling an on-prem, private-data product into regulated orgs. +- **The credential is a long-lived static secret** that doubles as both the + human's proof and the machine's permanent credential (see §3.1). + +Goal: make first-time setup *stupid simple* for a beginner, while staying secure +and working on the headless/remote/proxied boxes where tracebloc actually runs. + +## 3. The core reframe — two identities, two contexts + +### 3.1 Two identities, not one + +There are **two** things being authenticated, with opposite lifetimes: + +| | **Human** (account) | **Client** (machine/daemon) | +|---|---|---| +| Who | A person | An EC2 box / on-prem server | +| Lifetime | A login session | Runs 24/7 for months | +| Auth | Browser SSO/MFA | A long-lived machine credential | +| Created at | `ai.tracebloc.io` | Provisioned by backend | + +**A client is attached 1:1 to a cluster** — one cluster holds exactly one client, +and a client connected to a live cluster is *in use*. Client identity is therefore +*per-cluster*: that is the anchor the whole lifecycle keys on (§7.2). + +Today both collapse into one **Client ID + password**. The fix is **not** "browser +auth instead of credentials" — it's: *authenticate the human in the browser, and +let that authorization mint the machine credential automatically.* (This is the +`tailscale up` / `cloudflared tunnel login` / `aws sso login` model: a browser +authorizes a long-running daemon, the control plane issues the node its own key.) + +### 3.2 Two operational contexts (this is where the work actually is) + +Authenticating is the easy half. The review made clear that the hard half is the +**lifecycle of a client on a machine** — and the CLI lives in two contexts, with +a hand-off between them: + +- **Account context** — *"you are a signed-in user."* You hold a user token and + manage the *clients* (machines) in your account: create, list, select, delete. +- **Client context** — *"a client is active **and connected**."* Commands now act + on the active client's **cluster** (on-prem data never leaves it): ingest a + dataset, list datasets, delete a dataset. + +The bridge between them — *create or select ⇒ a client is active on this machine* +— is the single most bug-prone seam in the product, because "active" is a local +pointer while the data commands need a *reachable cluster*. §7 is devoted to it. + +**Command map (grounded to the repo, 2026-06-23):** + +| Context | Command | Status | This RFC | +|---|---|---|---| +| (signed out) | any client/data command | — | must refuse with *"run `tracebloc login`"* (§7.6) | +| login | `tracebloc login` | ✅ merged (cli#83) | keep | +| account | `client create` | ⚠️ in flight (cli#84 / PR #92) | **revise** → silent + idempotent + auto name/location, never print the credential | +| account | `client use` / select | ⚠️ in flight | **revise** → by slug / arrow-key picker, not numeric id | +| account | `client list` | ⚠️ in flight | **revise** → show *selected* vs *connected* | +| account | `client delete` | 🆕 | **new** — destructive guards (§7.4) | +| account | `logout` · `auth status` | ✅ merged (cli#83) | revise → scope active client to account; show token expiry | +| client | `data ingest` | ✅ built (was `dataset push`) | bind target to the active client's cluster (§7.3) | +| client | `data list` | ✅ built (was `dataset list`) | same | +| client | `data delete` | ✅ built (was `dataset rm`) | same | + +## 4. What already exists (grounded findings, refreshed 2026-06-23) + +A survey of `backend`, `client-runtime`, and `cli` shows **most of the data model +and much of the CLI already exist** — the net-new surface is the browser handshake +plus the lifecycle wiring in §7. + +### 4.1 Client = `EdgeDevice` (a Django `User` subclass) + +`backend/metaApi/models/User.py:314`. A client *is* an account (`type=EDGE`) with +`username` (UUID) + hashed `password` — i.e. the "Client ID + password". + +| Need | Already exists | Field | +|---|---|---| +| Human-readable display name | ✅ | `first_name` | +| Stable machine ID | ✅ | `username` (UUID) | +| DNS-safe slug | ✅ (separate) | `namespace` | +| Physical location (structured) | ✅ | `location` → `ZONE_CHOICES` (350+ Electricity Maps grid zones) | +| Carbon intensity | ✅ | `carbon_intensity` (gCO₂/kWh) | +| Account → many clients | ✅ | `account` FK | + +The two requirements we set out to add — **a human-readable name** and **physical +location for gCO₂** — are *already* `first_name` and `location`. `location` is a +controlled vocabulary (`backend/metaApi/models/zone_choices.py`) wired to a real +carbon pipeline (daily Electricity Maps fetch → `CarbonIntensity` model → +`update_estimated_gco2()` → `ExperimentSustainabilityMatrix`). **No new model +fields required.** + +### 4.2 Provisioning API already exists + +`POST /edge-device/` (`EdgeDeviceViewSet`, permission `CanManageClient`). Writable +fields are exactly `('first_name', 'account', 'location', 'password')` +(`edge_device_serializer.py:94`). `username`/email are auto-generated server-side +(`create()`); `namespace` is **not** set here today — the client heartbeat +(`EdgeDeviceHeartbeatView`) stores whatever the client reports, **verbatim: no slug +derivation, no format validation, no uniqueness** (`common/utils/edge_device_utils.py` +keeps `client_info.namespace` as-is). A CLI holding a user token can call this +endpoint to auto-provision a client. (Namespace sequencing is the catch — see §6.6; +the slug rule + uniqueness are net-new — R4.) + +### 4.3 Auth — the device grant is half-built + +- DRF Token auth: `POST /api-token-auth/` (login → token), `POST /register/` + (signup → token). **Google + GitHub OAuth already wired for web** — these mint + the *same* token, so the activation page needs **no new IdP wiring** (Q6). +- **CLI side is shipped:** `tracebloc login` / `logout` / `auth status` + + `internal/api` (env→base-URL, proxy/CA-aware HTTP) + `internal/config` + (`~/.tracebloc/config.json`, `0600`) landed in **cli#83** (commit `e322613`). + `login` already implements the RFC 8628 poll loop (`authorization_pending` / + `slow_down` / `expired_token`). +- **Backend side is the gap:** the `/device/*` endpoints + ([backend#835](https://github.com/tracebloc/backend/issues/835)) and the + provisioning hardening ([backend#836](https://github.com/tracebloc/backend/issues/836)) + don't exist yet — `login` returns a clear *"this backend doesn't support browser + login yet"* until they land. ← *This is the only net-new backend surface.* + +### 4.4 There is already a Go CLI, and it's the right home + +`tracebloc/cli` — Go + Cobra, v0.2.0+, cosign-signed, multi-arch, actively +maintained, and **already installed by the client bash installer** +(`client/scripts/lib/install-cli.sh`). It does dataset/ingest work and now carries +the auth scaffold (§4.3). This is the correct home for the lifecycle commands. + +### 4.5 Energy/carbon telemetry + +`client-runtime/Node-deploy/resource_monitor.py` sends CPU/GPU TDP + utilization +to `/edge-device-heartbeat/`. Carbon is computed backend-side from +`EdgeDevice.carbon_intensity` (location-driven). The heartbeat does **not** +auto-detect or report location — confirming location must be captured at +provisioning time, which is exactly what this RFC does (silently — §6.7). The +heartbeat *does* re-report `namespace` on every ping, which constrains §6.6. (It +does **not** carry `cluster_id`: the heartbeat sender can't read the `kube-system` +UID — the CLI/installer sets `cluster_id` instead — §7.2, R7.) + +### 4.6 How data commands target a cluster today (sets up §7.3) + +`data ingest` / `data delete` (today `dataset push` / `dataset rm`) resolve their cluster from +`--kubeconfig` / `--context` / `-n ` flags (default `$KUBECONFIG` → +`~/.kube/config`, current-context), then discover the parent release + shared PVC +by reading the chart's Deployment labels (`cluster.DiscoverParentRelease`, +`cluster.DiscoverSharedPVC`). **They do not read `config.ActiveClientID` at all.** +So today "the active client" and "the cluster the data commands act on" are two +unrelated mechanisms. Closing that gap is loophole §7.3. + +## 5. Goals / Non-goals + +**Goals** +- One-command setup for a beginner; **zero prompts in the common case** (D1). +- Browser-based human auth that works on headless/SSH/proxied boxes. +- Auto-provision the client (no manual `/clients` visit, no copied secrets, no + printed secrets — D2). +- Capture a human-readable name + structured location at provisioning — silently. +- Keep a non-interactive path for automation (`--token`, env vars). +- Don't break existing Client ID + password installs (dual-mode). +- **Make the whole client lifecycle safe and idempotent** — re-runs, deletes, + account switches, and interrupted installs all behave (§7). + +**Non-goals (this RFC)** +- Building account signup *in the terminal* (browser activation page handles + login **and** signup — keep ToS/GDPR/MFA/CAPTCHA where they already live). +- Short-lived/rotating client credentials + revocation (desirable; deferred to a + later phase — see §11). Rotation in phase 1 = delete + recreate. +- Fleet/enrollment-key management UI (phase 2). +- **True air-gapped (no-egress) installs** — out of scope per Q1; see §6.5. + +## 6. Proposed design + +### 6.1 Auth mechanism — OAuth 2.0 Device Authorization Grant (RFC 8628) + +Chosen because installs are **headless** (remote servers over SSH). The +browser-and-CLI need not share a machine, network, or continent. The CLI half is +already built (§4.3); the backend endpoints are §6.3. + +Rejected alternatives: +- **Type credentials into the installer** (today): blocks SSO/MFA; exposes + password to the script; phishing-prone. +- **Localhost-callback / PKCE** (`gcloud`, `vercel`): requires a browser on the + *same* machine — breaks over SSH. +- **Paste a token**: kept as the *fallback* (§6.5), not the default. + +### 6.2 CLI commands (in `tracebloc/cli`) + +``` +tracebloc login # device flow → store user token (~/.tracebloc, 0600) [✅] +tracebloc logout # clear token AND the active-client pointer (§7.5) [revise] +tracebloc auth status # account + env + token expiry + active/connected client [revise] + +tracebloc client create # silent, idempotent provision for THIS machine (§7.2) [revise] +tracebloc client list # show each client's slug + selected/connected state [revise] +tracebloc client use [] # select by slug; bare → arrow-key picker (§7.1, §7.3) [revise] +tracebloc client delete []# guarded teardown; bare → picker (§7.4) [new] + +tracebloc data ingest|list|delete # act on the ACTIVE client's cluster (§7.3) [revise] +``` + +**Data verbs — `ingest`, never `push` (locked, 2026-06-23).** The data commands are +`data ingest / list / delete`. *ingest*, not *push*: the data is loaded **into** the +client's own on-prem cluster and never leaves it — *push* wrongly implies sending it to +a remote (git/cloud) and undermines the product's core trust message. *delete*, not *rm*: +spelled-out and consistent with `client delete`, for beginner clarity. `dataset` + +`push`/`rm` stay as hidden aliases for one deprecation cycle. (Convention applies to all +copy, not just the command name.) + +`login` stores a short-lived **user** token. `client create` mints the **machine** +credential and routes it straight into the cluster (never to stdout — D2/§9). +`client create` **operates against an already-reachable cluster** — it reads that +cluster's identity as the anchor (§7.2), so the k8s API must be up first (the +installer bootstraps the base cluster before calling `create`; a clear error if +none is reachable). `create` never *creates* a cluster. + +Global: `--verbose` / `TRACEBLOC_LOG_LEVEL` for diagnosability (§8.5), and a +`--uninstall` teardown on `client delete` / the installer for clean offboarding +(R12). `tracebloc cluster doctor` extends to also check auth/config/token state, so +it can diagnose a failed *provision*, not just cluster health. + +### 6.3 Backend additions (in `tracebloc/backend`) + +- `POST /device/code` → `{ device_code, user_code, verification_uri, + verification_uri_complete, expires_in, interval }`. ([backend#835]) +- `POST /device/token` → polled by the CLI; returns a user token once approved + (`authorization_pending` / `slow_down` / `expired_token` per RFC 8628). ([backend#835]) +- A web **activation page** `https://ai.tracebloc.io/activate` — a *token-authed* + endpoint that reuses the existing web login/signup (incl. Google/GitHub) and + binds the approval to `request.user`. No new IdP wiring (Q6). It shows **what is + being authorized** ("Connect machine *X* to account *Y*?") as the phishing + mitigation. +- **Split the client permission read/write** (Q4): listing clients must not + require `CanManageClient`. Today one permission gates both, so a user who may + *select* an existing client but not *create* one gets a bare `403`. Split into a + read scope (list/use) and a write scope (create/delete); a write `403` routes to + "ask an admin" (§7.4), already stubbed as `askAnAdmin` in PR #92. ([backend#836]) +- **Enforce `namespace` uniqueness** per-account at the DB layer + ([backend#863]) — the CLI's collision suffix is advisory + racy; only a + `UniqueConstraint(account, namespace)` actually guarantees it (§6.6). +- **Record the attached cluster + enforce one client per cluster** — add a + `cluster_id` to `EdgeDevice` (no such field today — only `namespace`) carrying + the target cluster's stable fingerprint (the `kube-system` namespace UID) with + `unique=True`. This is the durable attachment record that makes `client create` + idempotent across CLI-config loss and the pre-install orphan window (§7.2), and a + stronger backstop than the namespace constraint: `create` becomes get-or-create + keyed on `cluster_id`. Two musts, because the `kube-system` UID is **not a + secret** (anyone with cluster access reads it): + - **Account-scoped get-or-create.** Return the existing client **only if it's in + the requester's account**; a `cluster_id` already bound to a *different* account + is an explicit **409 conflict**, never a silent adoption — otherwise re-pointing + a cluster from another account would hijack the first account's client (R6). + - **Backfill existing clients.** `cluster_id` is net-new, so every current client + has it null. The **CLI/installer** PATCHes it on the next run — it holds the + operator kubeconfig and can read the `kube-system` UID, which the heartbeat + sender can't (§10, R7). Until then, the §7.2 step-2a live-release guard stops a + re-mint. (The backend accepts a CLI-supplied `cluster_id`; see C.3.) + - (Anchor field + get-or-create + 409 + adopt-backfill: [backend#883], + split out of #836 — which ships namespace validation + RBAC only (see PR + [backend#862]). Server-side token revoke for `logout` lands as the `POST /auth/revoke` + endpoint ([backend#887] + a CLI call); backend#845 shipped only the `revoke()` + primitive, so logout is local-only until #887.) + +### 6.4 Installer reorder (in `tracebloc/client`) + +Move CLI install + `tracebloc login` + `tracebloc client create` to run **before** +the Helm install, because the minted credential feeds the chart. (Today the CLI +installs *after* the cluster.) The credential is written to the chart's +values/secret (mode `0600`) **before** `helm install` runs, so an interrupted +install can be resumed without re-minting (§7.9). Keep CLI-install failure +non-fatal only for the *data* convenience path, not for the auth path. The +installer's existing one-per-cluster guard and the CLI's idempotent +`create` must key on the **same** cluster identity (§7.2). + +### 6.5 Fallbacks — automation (air-gap is out of scope) + +- `TRACEBLOC_ENROLL_TOKEN` / `--token`: a pre-issued credential for + Ansible/Terraform/CI/golden-images and for **egress-restricted-but-online** + on-prem boxes (the #172 corporate-proxy segment). The device-flow HTTP client + honors `HTTPS_PROXY`/`NO_PROXY` + custom CAs (reuse the #172 hardening). +- **True air-gap (no egress at all) is out of scope (Q1).** Preflight hard-fails + on no egress; #172 is *corporate-proxy* support (TLS-inspecting proxy), not + air-gap. If a real no-egress segment appears later, the enrollment-key fallback + becomes first-class — but we will not design for a customer we don't have. +- Existing **Client ID + password** path stays working (dual-mode) for one full + deprecation cycle. + +### 6.6 Name → namespace: derive once, set both, then freeze + +Today there are effectively two names: `first_name` (display) and `namespace` +(k8s). Asking for both is redundant; in the silent flow we ask for **neither** +(§6.7) — we derive both from the hostname. + +> **Amended — Rev 8 / cli#137:** the silent flow no longer derives the name from the +> hostname. It auto-generates `-NN` from the signed-in identity +> (the account — and thus the user's first name — is known at create time; the +> hostname is neither stable nor account-scoped). `NN` is the next free two-digit +> number across the account's existing client names/namespaces, so a second machine +> is `lukas-02` rather than a slug `-2` bump. The derived name is already slug-clean, +> so the derive-once / set-both / freeze namespace rule below is unchanged (name = +> namespace by construction). `--name` / `TRACEBLOC_CLIENT_NAME` still override. + +> **All of this is net-new.** Today the backend does *no* namespace processing — it +> stores the client-reported `namespace` verbatim (§4.2), with no slug derivation, +> no format validation, and no uniqueness. The slug rule below, setting `namespace` +> at create, and the §6.3 constraint are all new work; the slug rule currently lives +> only in the CLI (`cli/internal/slug`) + Appendix A, not in the backend. + +**Proposal: derive the `namespace` slug from the name once, at creation, set it on +*both* `EdgeDevice.namespace` and the install-time `TB_NAMESPACE`, and freeze it.** + +Why derive-and-freeze, and why set *both*: + +- **Kubernetes namespaces are immutable.** You cannot rename one, and the name is + baked into resource names (`-jobs-manager`, `-requests-proxy`), DNS, and + PVCs. Deriving once and freezing keeps `first_name` (mutable display) decoupled + from `namespace` (frozen), matching the model. +- **The heartbeat re-reports `namespace` on every ping** (§4.5). So if the + provisioned slug and the install-time `TB_NAMESPACE` disagree, the heartbeat + will overwrite the backend's namespace and the two **drift**. Resolution (Q2): + `name → slug → set EdgeDevice.namespace at create AND pass the same slug as + TB_NAMESPACE to the chart`. They are equal by construction, so the heartbeat is + a no-op re-report. + +Derivation rules (reference algorithm + validation in Appendix A): + +- **Slugify:** lowercase, transliterate unicode, spaces/punctuation → `-`, collapse + repeats, strip to DNS-1123 (`[a-z0-9-]`, ≤63 chars, no leading/trailing `-`). +- **Collision-suffix:** append `-2`, `-3`, … against the account's existing + namespaces. This is the friendly UX layer; the **DB constraint** (backend#863) + is what actually guarantees uniqueness against races / direct API calls. It + fires only for *different* clusters that derive the same base name — never for + the same cluster re-running, which the cluster anchor catches first (§7.2). +- **Empty-slug guard:** a name that slugifies to empty (e.g. all-CJK) falls back to + `client-`. +- **Surface, don't ask:** show the derived slug in progress + (`slug: munich-hospital-radiology`); `--namespace` overrides for multi-client + hosts / naming conventions. +- **Backfill:** existing clients keep their current `namespace`; only set + `first_name` as a display backfill — never re-derive an existing slug. + +### 6.7 Location: silent auto-detect, never block + +`location` is **optional at the model layer** (`CharField(..., blank=True)`), so a +client can be created with no location and `carbon_intensity` defaults to `0` — +i.e. it silently reads as "carbon-free", quietly corrupting the exact metric +tracebloc sells. + +> **Amended — Rev 8 / cli#137:** location is no longer auto-detected on the silent +> path. With no `--location` the CLI omits it (`CreateClientRequest.location` is +> `omitempty`) and the backend records the client with **no location** — the explicit +> "not set" state (backend#993), not a silent zero and not a provision-time GeoIP +> guess. The cloud-metadata auto-detect (`internal/geo`, cli#93) is **removed**, not +> parked: the silent path never detects, and the backend is the source of truth for +> valid zones (a bad `--location` surfaces as its real create error). The +> "never block / never fake a zero" principle below stands; "auto-detect silently" +> is the part that's gone. + +**Proposal: auto-detect the zone and use it silently; never prompt, never block, +never fake a zero.** + +- **Detection (`internal/geo`, cli#93):** cloud instance metadata first (AWS + IMDSv2/v1, GCP, Azure — probed concurrently under one short deadline, first to + answer wins), GeoIP fallback (Cloudflare `cdn-cgi/trace`, flagged low + confidence). Output is always an ISO alpha-2 country code, always a valid + top-level `ZONE_CHOICES` value. +- **Silent (D1):** the detected zone is used directly and *surfaced* in progress + (`Setting up gpu-box-01 in 🇩🇪 DE`), not prompted. `--location DE` overrides + and skips detection entirely. +- **Never block (the §7.7 fallback):** bare-metal / offline / egress-restricted → + no detection. Rather than prompt (D1) or fake a zero, fall back to an + **account-default zone** if one is set, else mark the client **location-unset** + (an explicit, visible state — *not* `0`) and nudge in the dashboard. Setup still + completes. +- **Keep the DB `blank=True`** for backward compatibility; enforce "must be an + explicit value or an explicit unset" at the provisioning layer, not with a DB + constraint. +- **Mutable post-install** (`--location` / dashboard). Changing it affects + **future** readings only — gCO₂ is a frozen per-experiment snapshot and is never + re-derived from current location (Q3, confirmed in the carbon pipeline). + +## 7. Client lifecycle — the loopholes and how the design closes them + +This is the heart of rev 2. Each item is a user-perspective failure mode from the +review, with the resolution and its status. Ordered by how much rides on it; +**[decision]** items shape the command surface and were settled in §0. + +### 7.1 Picking a client without ever showing an id or secret — **[D3]** + +**Risk.** "No secret in the terminal" (D2) collides with `use` / `delete` / `list`, +which all need to *name* a client. Hide everything and the user can't pick one. + +**Resolution.** The user-facing handle is the **namespace slug** (unique per +account, DNS-safe, human-meaningful, e.g. `gpu-box-01`) plus status — never the +backend UUID / username / password. `use ` / `delete ` take the slug; +run bare, they drop into an arrow-key **picker** over the account's clients. Only +the *credential* is truly hidden; the *name* is the interface. + +**Rename is cosmetic.** The slug is frozen at creation (it *is* the k8s namespace — +§6.6); renaming a client changes only `first_name` (the display name), **not** the +handle — the slug you type in `use`/`delete` stays the original. The picker is right +for a handful of clients; an account with dozens needs filter/search (phase 2, +alongside the enrollment-key path). + +### 7.2 Re-running setup must not mint a duplicate client — **[decision]** + +**Invariant.** A client is attached to exactly one cluster, and a cluster holds +exactly one client (1:1, §3.1). Identity is *per-cluster*, so "does this host +already have a client?" is really *"is this cluster already attached to a +client?"* — and a cluster can always answer that. + +**Risk.** Run the installer (or `client create`) twice against one cluster → two +backend clients, doubled "capacity", a confusing dashboard. Today `create` always +mints — and worse, the collision suffix (§6.6) turns the re-run into a *new* slug +(`gpu-server` → `gpu-server-2`), actively manufacturing the duplicate, because the +slug is derived against the account's namespaces *including this cluster's own*. + +**Resolution.** `create` is **get-or-create keyed on the cluster**, not a mint. +The anchor is the **cluster identity** (proposed: the `kube-system` namespace UID — +the conventional stable fingerprint), readable *before* anything is installed, so +it exists from t=0 — which the CLI config, a mere cache, cannot provide once it is +lost: + +1. Read the target cluster's identity (the `kube-system` UID) — and, if a tracebloc + release already sits in the namespace, the in-cluster `TB_CLIENT_ID`. +2. **Ask the backend, account-scoped: is this cluster already mine?** This runs + *before* any adoption, so the live-release path can't bypass it (R6): + - **Owned by this account** — matched by `cluster_id`, or by the live + `TB_CLIENT_ID` (whose null `cluster_id` the CLI backfills now) → adopt, + reconcile/upgrade. **No mint.** (normal re-run / orphan resume, §7.9) + - **Bound to a *different* account** → `409`, refuse — **never silent-adopt**, + even when a live release occupies the namespace. +3. **Not attached anywhere** → mint, stamp `cluster_id`, write the credential + (`0600`) before Helm, install. + +One-client-per-cluster is enforced server-side by the `unique` `cluster_id` (§6.3): +a second attach returns the existing client, never a duplicate, even under a race. +Because step 2's account-scoped check gates **adoption itself** (not just the mint), +reading a live `TB_CLIENT_ID` off the cluster can't sidestep the cross-account +`409`. The installer's one-per-cluster guard and the CLI agree because they key on +the **same** cluster identity. The `-2`/`-3` suffix only disambiguates cosmetic name +clashes *across different clusters* — it can't produce a same-cluster duplicate, +because the cluster-id is checked first. + +**Existing fleet — `cluster_id` is null until backfilled (R7).** Current clients +predate the anchor, so a bare `cluster_id` lookup can't match them yet. The guard: +the installer **never mints when a live tracebloc release already occupies the +target namespace** — it reads the in-cluster `TB_CLIENT_ID`, confirms (account-scoped, +step 2) that the client is this account's, **PATCHes the freshly-read `cluster_id` +onto it** (the CLI/installer holds the kubeconfig and can read the `kube-system` UID; +the heartbeat sender can't), and adopts. After that the cluster_id lookup takes over. +Without this guard, the first re-run on every *existing* box would mint a duplicate +and orphan the live client. + +**Adopt keeps the existing namespace.** On the adopt branches (2a/2b), identity and +`namespace` come from the existing client/cluster — the silent flow's +hostname-derived slug and collision suffix apply **only** on the mint branch (step +3). A re-run from a different jump host (or after a hostname change) must never +re-derive a slug for an adopted client: that would drift the backend or mismatch the +live, immutable `TB_NAMESPACE`. + +### 7.3 "Selected" is not "connected" — **[decision]** + +**Risk.** `client use` sets a *local pointer*. But `data ingest` talks to the +client's **cluster** (§4.6). If the active client lives on another machine, ingest +can't reach it from here — and today the data commands don't even consult the +pointer, so they'd silently act on whatever `~/.kube/config` points at. + +**Resolution.** Bind the active client to a **reachable cluster context**: + +- The active client carries its `namespace`; data commands default `-n` to it and + resolve a kube-context that hosts `-jobs-manager` (reuse + `DiscoverParentRelease`). `--context` / `-n` still override. +- If no reachable context hosts the active client (it runs elsewhere), **fail + clearly**: *"client `X` runs on another machine — run `data` commands there, or + `tracebloc client use` a local one."* No silent wrong-target. +- `client list` distinguishes **selected** (the local pointer) from **connected** + (cluster reachable + recent heartbeat = 🟢). + +### 7.4 Delete is silently destructive — **[decision]** + +**Risk.** Deleting a client orphans its cluster install + on-prem data; if it's +bound to a remote running machine, that machine's pods crash-loop (their backend +identity vanished). + +**Resolution.** `client delete`: + +- **Confirms** (the *one* justified interactive prompt — destructive; D1's "no + prompts" is about *setup*, not destruction). +- **Refuses/warns** if the client is online (recent heartbeat), holds datasets, or + has a **running experiment / training job** — the heartbeat guard is advisory, so + gate on live job state too; a delete (or a `cluster_id`-changing rebuild) mid-run + silently loses work. `--force` to override. +- **Offers to tear down** the local Helm release for the active client. +- **Checks RBAC** (write `403` → "ask an admin", §6.3). +- **Clears the stale active pointer** afterward (§7.5). + +### 7.5 Stale active client — cross-account *and* cross-env (confirmed bug today) + +**Risk.** The active client is cached locally. **`logout` today clears only the +token + email and leaves `ActiveClientID` set** ([auth.go](internal/cli/auth.go)); +log into a *different* account and the cached pointer references a client you no +longer own → data commands hit something foreign (`401`/`403`, or worse, a +wrong-but-valid target). + +**Resolution.** Scope the active client to the account: + +- `logout` clears the active-client pointer (and `login` to a different account + drops it if the client isn't in the new account). +- **`logout` must also revoke the token server-side**, not just locally — a DRF + token is static, so a copied/leaked token survives a local-only clear for its full + life (R2). **Not wired yet (FR-confirmed 2026-06-25):** today `logout` clears only + the local token; server-side revoke needs the `POST /auth/revoke` endpoint + ([backend#887], not built) plus a CLI `logout`→revoke call. backend#845 shipped + only the underlying `revoke()` primitive. +- **Scope the active client to the *environment* too, not just the account (R10).** + `~/.tracebloc` holds one `Env`+`Token`+`ActiveClientID`; `login --env` overwrites + env+token but today leaves the *old* env's `ActiveClientID` stranded → prod + commands silently hit a dev client. Either key config by env (a profile map) or + clear / reselect the active client whenever `login` changes `Env`. +- Cheap re-validation on each client/data command: if the active client isn't in + the signed-in account, drop it → *"your selected client is gone — pick one."* + +### 7.6 Auth lifecycle & expiry + +**Risk.** The user token expires mid-use → raw `401`s. Wrong env (dev vs prod). A +headless box with no browser. + +**Resolution.** Map `401` → *"session expired — run `tracebloc login`."* `login` +sets the env; `auth status` shows account + env + **token expiry** (today it shows +account + env but not expiry — add it). Signed-out client/data commands refuse with +*"run `tracebloc login`."* Headless already works via the device flow (URL + code +to open on another device) — keep that copy clear. + +### 7.7 Auto name & location with no prompt — **[D1]** + +**Risk.** Silent setup means name = hostname and location = auto-detect — but two +machines can share a hostname, and a bare-metal host may have no detectable +location, and we've committed to *no prompts*. + +**Resolution.** Name = sanitized hostname; hostname collisions get the `-2` +namespace suffix (§6.6) so the *slug* stays unique even when display names match. +Location = the cli#93 auto-detect; if undetectable, fall back to the account +default / `unset` rather than block (§6.7). Surface the chosen name + zone in +friendly progress (*"Setting up gpu-box-01 in DE"*) — visible but not +interactive; correct later with `--name` / `--location` or in the dashboard. + +> **Amended — Rev 8 / cli#137:** name = `-NN` from the signed-in identity +> (not the hostname), and location is **optional** — omitted when no `--location` is +> given, leaving the client in the explicit "not set" state (backend#993) rather than +> auto-detected at provision time. See the §6.6 and §6.7 Rev 8 callouts. Both inputs +> stay zero-prompt on the common path; `--name` / `--location` still override. + +### 7.8 If nothing is shown, how does the user manage it later? — **[D2]** + +**Risk.** "Never show the credential" (D2) is right — but the user still needs to +find, re-point, or rotate a client later. + +**Resolution.** Management is always **by slug**, via the CLI (`list` / `use` / +`delete`) and the web app — never by handling secrets. The credential lives only +in the cluster secret (mode `0600`) + the backend (hashed); the CLI never persists +or prints it. **Rotation = delete + recreate** in phase 1 (a dedicated `rotate` +verb is a phase-2 nicety). Confirm the credential is genuinely +non-user-retrievable end to end. + +### 7.9 Interrupted setup leaves an orphan + +**Risk.** `login` ✓ → `create` ✓ → Helm **fails**. A client now exists in the +backend but no cluster runs it. A naive re-run mints a *second* orphan. Worse, if +the CLI minted the password and didn't route it anywhere before Helm failed, the +plaintext is lost (backend stores only the hash) and the orphan is unusable. + +**Resolution.** The orphan is found by its `cluster_id` (§7.2 step 2b): the re-run +reads the same cluster identity, the backend returns the recorded client, and the +CLI **resumes into it** instead of minting a second. + +- The CLI writes the machine credential into the chart's values/secret (`0600`) + **before** invoking Helm. An interrupted install leaves that file in place, so + the resume reuses the credential and just re-runs Helm. +- If the credential was lost (no values file — e.g. a fresh box), resume **resets** + the orphan's password (a `PATCH` on the existing `EdgeDevice`) and rewrites it — + but **only after confirming the client is offline** (no recent heartbeat). A + client that connected once and *then* lost its local values file still has a + running pod using the old credential; a blind reset would break it. Gate the reset + on heartbeat recency, not just "no values file." +- Concurrent attaches to one cluster are serialized by the `unique` `cluster_id` + constraint (§6.3); the loser fetches and adopts the winner. + +## 8. UX — drafted flows + +> **Tracked as four acceptance families** on the epic (#830) — **#877** connect/install · +> **#878** manage clients · **#879** data · **#880** uninstall — that FR runs end-to-end, +> while the build work stays in the per-component tickets (§13). All four follow one +> **2-phase shape**: a single human gate (browser sign-in), then unattended, idempotent +> convergence. The design is graded against seven principles — front-loaded human gate · +> idempotency-as-backbone · never-silent / never-lie · quiet-by-default · failure-UX +> first-class · secure-by-invisibility · installer = thin CLI orchestrator. The +> connect/install flow's full ordered step-spec lives with #877. + +### 8.1 First-time, headless box (zero prompts) + +``` +$ bash <(curl -fsSL https://tracebloc.io/i.sh) +✔ Checking this machine… ready (8 CPU · 30 GiB RAM · 46 GiB free · network OK) + + To connect this machine, sign in to tracebloc: + → https://ai.tracebloc.io/activate + code: WDJB-MJHT + Waiting for you to finish in your browser… (Ctrl-C to cancel) + +# (user opens URL on laptop → logs in / signs up → approves "WDJB-MJHT") + +✔ Signed in as asad@acme.com + Setting up this machine as “gpu-box-01” in 🇩🇪 DE (rename later — display only; handle stays gpu-box-01) +✔ Provisioned — credential written to the cluster (not shown; managed by name) +→ Installing… (streaming rollout) jobs-manager ✔ requests-proxy ✔ resource-monitor … ~3 min +✔ Connected — this machine is 🟢 Online https://ai.tracebloc.io/clients +``` + +No name prompt, no location prompt, **no credential on screen**. The two values +are *surfaced* and correctable, not asked (D1, D2). + +### 8.2 Returning / re-run (already enrolled) + +Read the target cluster's identity → it's already attached to a client (§7.2) → +**skip auth and setup entirely** → resume into that client and reconcile / upgrade. +Idempotent re-runs are non-negotiable; a re-run after a failed install resumes the +same client rather than minting a second (§7.9). + +### 8.3 Automation (non-interactive, online) + +``` +TRACEBLOC_ENROLL_TOKEN=… TRACEBLOC_CLIENT_NAME="Lab A" TRACEBLOC_LOCATION=DE \ + bash <(curl -fsSL https://tracebloc.io/i.sh) # zero prompts, zero browser +``` + +(True air-gap is out of scope — §6.5. This path still needs egress to the backend.) + +### 8.4 Managing clients later (by name, never by secret) + +``` +$ tracebloc client list + SLUG STATE LOCATION + gpu-box-01 (active) 🟢 connected DE + munich-radiology ⚪ selected-elsewhere DE + lab-a ⚫ offline FR + +$ tracebloc client use # bare → arrow-key picker +$ tracebloc client delete lab-a # confirms; refuses if online/holds data; offers teardown +``` + +### 8.5 When setup fails (headless, no prompts) + +The flow is zero-prompt on a box with no human at the console, so the *failure* +path has to stand on its own — otherwise "easy when it works" becomes "stuck when +it doesn't": + +- Always write a persistent `~/.tracebloc/install-.log`; add `--verbose` / + `TRACEBLOC_LOG_LEVEL` that streams the device-flow → provision → Helm steps (today + the CLI has no verbosity flag, only `--plain`). +- On any failure, print **the exact resume command** (re-running is idempotent — + §7.2/§8.2) and point at `tracebloc cluster doctor`, **extended to also check + auth/config/token state** (today it's cluster-health only, so it can't diagnose a + failed *provision*). +- Stream per-Deployment rollout progress with an overall timeout and a "still + pulling images" heartbeat, so a slow first run is distinguishable from a wedge — + and *"🟢 Connected" must reflect a real readiness probe, not a sleep* (the core + Deployments need readiness/liveness probes for that to be honest). + +## 9. Security considerations + +- **The credential never enters the terminal, scrollback, clipboard, shell history, + or `~/.tracebloc`** (D2). The CLI generates it, `POST`s it (backend stores the + hash), and writes the plaintext only into the cluster secret / Helm values + (mode `0600`), which is also its sole durable home alongside the backend hash. +- Password leaves the installer process space entirely — the CLI only ever holds a + device code, then a scoped user token, then routes the per-client credential to + the cluster. +- Device-code phishing (RFC 8628): bind `user_code` → account, short TTL (the impl + uses 10 min), **rate-limit `/device/code` + `/device/token`** (they are + unauthenticated public surface — a DoS / guessing target), and an approval page + that names the device and warns *"only approve if you started this on that box."* +- Secret-at-rest: write cluster credentials `0600`. (Observed `drwxrwxrwx` data + dirs and a world-ish `values.yaml` on an existing box — tighten when we start + auto-writing credentials.) Note the credential also lands **base64 in the k8s / + Helm release Secret (etcd)** — acceptable for single-tenant on-prem, but state it + as a conscious call (encrypt etcd at rest where the customer requires it). +- Tokens: store the user token `0600` in `~/.tracebloc`; `logout` clears it **and** + the active-client pointer, and **must revoke it server-side** — but today it is + **local-only** (FR-confirmed): server-side revoke is pending the `POST /auth/revoke` + endpoint ([backend#887] + a CLI call; #845 is only the primitive). A local clear + alone leaves a static DRF token valid for its full life (R2). +- Least privilege: list/use need only the read scope; create/delete need the write + scope (§6.3, Q4). +- **Bootstrap supply-chain (R8) — the dominant gap for a regulated buyer.** "The CLI + binary is cosign-signed" covers only the leaf. The `curl|bash` entry pulls ~14 + sub-scripts from a *mutable branch ref* with no checksum/signature, and the CLI + installer's cosign check is **skipped when cosign is absent** (the default), + degrading to a SHA256 fetched over the *same channel* as the binary. The most + privileged code (writes the credential, runs Helm) is the least verified. *Fix:* + pin to an immutable release tag, verify each sub-script against a signed manifest, + and make signature verification mandatory on the default path. +- **Audit trail (R9) — table-stakes for SOC 2 / HIPAA.** Emit an append-only event + (actor, account, `cluster_id`, time, source IP) for every device-flow approval, + client mint/adopt/delete, credential issue/reset (§7.9), cross-account `409` (R6 — + an *attempted* tenant crossing), and logout/revoke. None exists today. +- **Revoke a *compromised machine credential* in phase 1**, not just delete+recreate: + a stolen edge-node key needs a first-class backend "revoke this client now" + (invalidate the hash, force re-enroll) independent of cluster teardown — and don't + gate the §7.9 reset on heartbeat recency alone (the heartbeat isn't authenticated). +- **`cluster_id` is set by the authenticated CLI, never self-reported.** The + CLI/installer reads the real `kube-system` UID with the operator's kubeconfig and + sets `cluster_id` on a Bearer-authed call (create / adopt-backfill — §7.2), so + there's no unauthenticated client self-report to spoof and the heartbeat does + **not** carry it. Backend: reject a `cluster_id` change on an already-set client + except via an authorized re-enroll. +- **Tenancy boundary (state it).** The 1:1 client↔cluster rule + `cluster_id` + uniqueness implies **one account per cluster** — make that the explicit isolation + invariant; if multi-namespace-per-cluster is ever allowed, document the namespace + trust boundary (it gates `DiscoverSharedPVC`, §7.3 targeting, and the shared etcd). + +## 10. Backwards compatibility & migration + +- Dual-mode: Client ID + password and `--token` paths keep working for one + deprecation cycle. +- Backfill `first_name` for existing clients; do not touch `namespace`. +- **`cluster_id` backfill (blocking — R7).** The anchor is net-new, so every + existing client has `cluster_id=null` and a `cluster_id` lookup can't match them. + The **CLI/installer** backfills it on the next run: it reads the `kube-system` UID + (operator kubeconfig) and PATCHes it onto the adopted client (§7.2, C.3). The + heartbeat sender can't read that UID, so it is **not** the carrier. Until a box is + re-run, the §7.2 step-2a live-release guard stops a re-mint — so **that guard must + ship with or before the idempotent installer**, or existing customers + double-provision on their next upgrade. +- The DB namespace uniqueness constraint (backend#863) must ship with a migration + that resolves any *existing* collisions first — run the Appendix A check (R4). +- Deprecate the manual `/clients` "create" path only after device flow is GA; + keep `/clients` as **manage/revoke**. + +## 11. Phased rollout + +**Critical path (Phase 1)** — the CLI is mostly done or quick; it is *gated on* +backend + frontend work, not the reverse (R1): device endpoints (backend#835) +**and** the frontend `/activate` page unblock `login`; the RBAC read/write split +(backend#836) unblocks `list` + the picker (D3); namespace uniqueness (backend#863) +unblocks safe idempotent `create` — *after* the R4 collision check. + +- **Phase 0** (no backend work, partly shipped): stop sending users to "create a + client first"; add `--token` / `TRACEBLOC_ENROLL_TOKEN` so the secret isn't typed + inline. The CLI auth scaffold (cli#83) is already merged. +- **Phase 1 — backend + frontend first:** device-flow endpoints (backend#835); the + `/activate` page (frontend — **assign an owner, R1**); RBAC read/write split + a + `unique` `cluster_id` field (backend#836); namespace uniqueness *after the R4 + check* (backend#863). +- **Phase 1 — CLI, once the above land:** revise `client create` to silent + + idempotent (cluster anchor) + never-show + auto name/location; add `client delete` + (+ `--uninstall`, R12); slug + picker for `use`/`delete`; selected-vs-connected in + `list`; wire the active client → cluster context (§7.3); env-scope the active + pointer (R10); send a CLI version header (R11); `--verbose` + failure flow (§8.5); + installer reorder. Dual-mode. +- **Phase 1 — security must-haves (regulated buyer):** bootstrap supply-chain + hardening (R8); the provisioning/auth **audit trail** (R9); a machine-credential + **revoke**; and **authenticated `cluster_id`** claims (§9). Not deferrable for the + on-prem/regulated sell. +- **Phase 2** (hardening): short-lived auto-refreshing tokens + **server-side token + revocation (R2)**; a `client rotate` verb; atomic fleet enrollment (R5); picker + filter/search at fleet scale. + +## 12. Open questions — resolved on backend#830 (owner to confirm the two product calls) + +The original §11 questions were worked to resolution with a code-grounded sweep on +the tracking epic ([backend#830]). Most are dictated by the code: + +| Q | Topic | Resolution | +|---|---|---| +| Q1 | Air-gap segment | **Out of scope** (confirmed). Egress-restricted-but-online (TLS-inspecting proxy, #172) is in; true no-egress is not. | +| Q2 | Namespace derivation | `name → slug → set both EdgeDevice.namespace + TB_NAMESPACE` (heartbeat re-reports namespace, so they must be equal — §6.6). | +| Q3 | Location-change semantics | **Future-only**, already clean — gCO₂ is a frozen per-experiment snapshot, never re-derived. | +| Q4 | RBAC | **Split read from write**; write `403` → "pick existing / ask an admin" (§6.3, §7.4). | +| Q5 | Multi-client / re-parenting | **One client per cluster** (confirmed; 1:1, enforced by `unique` `cluster_id` — §6.3 / §7.2); "multi-client per host" means multi-*cluster*, one client each. **Re-parenting deferred** (the viewset force-stamps `account`). | +| Q6 | Device-flow IdP | **Reuse the existing web login as-is**; `/activate` is a token-authed endpoint binding to `request.user` — no new IdP wiring. | + +**Cluster identifier — confirmed:** the `kube-system` namespace UID (`EdgeDevice` +gains a `cluster_id` for it — §6.3). The §7.3 "connected" check falls out of it: +cluster-id match + heartbeat recency. With `cluster_id` as the authoritative +one-per-cluster guard, the per-account namespace `UniqueConstraint` (backend#863) is +demoted to **cosmetic cross-cluster dedup**, not an idempotency guarantee. + +## 13. Work breakdown (cross-repo; tracked on backend#830) + +- **`backend`**: `/device/code` + `/device/token` + activation page (#835); + provisioning hardening + RBAC read/write split + a `cluster_id` field + (`unique=True`) with an **account-scoped** get-or-create that gates **adoption** + (cross-account = `409`, even on the live-release path — R6/§7.2) and accepts a + CLI-supplied `cluster_id` (set at create, PATCHed on adopt-backfill — R7) (#836); + server-side token revoke for `logout` (the `POST /auth/revoke` endpoint #887; #845 + shipped only the `revoke()` primitive); `namespace` + `UniqueConstraint(account, namespace)` + collision migration after the R4 check + (#863). Plus an append-only **audit trail** (R9), a **machine-credential revoke** + endpoint, and a **min-supported CLI version** advertised for skew handling (R11). +- **`cli`**: revise `client create` → silent + idempotent (get-or-create keyed on + the cluster identity — read the target cluster's `kube-system` UID, confirm + account-scoped ownership, then adopt + backfill `cluster_id` onto a live + in-namespace release or an orphaned client before minting) + never-show + auto + name/location (cli#84/#92); location auto-detect (cli#93); `client delete`; slug + + picker for `use`/`delete`; selected-vs-connected `list`; bind active client → + cluster context for the `data` commands; scope the active pointer to the account, + clear **and server-side revoke** on logout (CLI half of [backend#887]; #845 is only + the primitive); `auth status` token expiry. + Also: a `User-Agent: tracebloc-cli/` version header (R11); env-scoped config / + profiles (R10); `--verbose` + `~/.tracebloc/install-*.log` and `cluster doctor` + auth/config checks (§8.5); `client delete --uninstall` (R12). +- **`client` (installer)**: reorder CLI install + auth before Helm; write the + credential to values/secret (`0600`) before Helm; share the one-per-cluster + cluster-id anchor with the CLI; dual-mode env fallbacks; idempotent re-run / + orphan resume. Plus **supply-chain hardening** (immutable release tag, sub-scripts + verified against a signed manifest, mandatory signature check — R8); a first-class + **uninstall/offboard** path (tear down the release, de-register the `EdgeDevice`, + clear local state — R12); and streamed per-Deployment rollout progress (§8.5). + +## 14. Risks & dependencies + +The §7 loopholes are bugs *inside* the flow. These are the risks *around* it — +delivery sequencing, security blast radius, and migration. **R1–R12 are the ones to +act on** (R6–R9 are the blocking finds; R8–R12 came from the latest hardening pass); +the rest are watch-items. + +### R1 — Critical path crosses three repos, and one piece is unowned + +`login` is merged but inert until the device endpoints (backend#835) ship; the +slug + picker (D3) and "ask an admin" need the RBAC read/write split (backend#836) +— today one permission gates both *list* and *create*, so a normal user can't even +list to pick; and the **`/activate` page is frontend work** (the Next.js app), not +covered by the backend / cli / client breakdown (§13). *Mitigation:* make the +dependency chain explicit (§11) and **assign the activation page before Phase 1** — +it is the single point of failure for the whole device flow. + +### R2 — The blast radius is the user token, not the machine credential + +The device-issued user token is account-scoped (create/delete *any* client), +long-lived, stored `0600` on every edge box, and `logout` only clears it locally — +DRF tokens are static, so a leaked token stays valid for its full life regardless +of logout. One compromised box = fleet-wide client control until expiry. D2 hid the +*small* secret and left the *big* one on disk. *Mitigation (§9):* scope the device +token to provisioning, short TTL + refresh, **discard it after install on +unattended boxes** (unneeded once the machine credential is in the cluster), and +**revoke server-side on logout** via the `POST /auth/revoke` endpoint ([backend#887]; +#845 shipped only the `revoke()` primitive) — a Phase-1 must, **not wired yet**: +logout is local-only today (FR-confirmed 2026-06-25). + +### R3 — The cluster anchor has a precondition + +Keying idempotency on the cluster identity (§7.2) assumes the cluster's k8s API is +already up when `create` runs — true for managed/EKS, but the bare-metal "bootstrap +k3s *then* install tracebloc" path now has a hard order: **k8s up → read cluster-id +→ create → helm install** (§6.4 must mean the *tracebloc* chart, not the base +cluster). And the `kube-system` UID changes on a cluster rebuild → a rebuilt +cluster reads as new and mints a new client, orphaning the old. *Mitigation:* state +the ordering in §6.4 + the installer; treat "rebuilt cluster = new client" as +intended, and add a **reaper / teardown hook** (or a dashboard sweep) so orphaned +`EdgeDevice`s from rebuilds don't accumulate — their `cluster_id` never returns. + +### R4 — The namespace-uniqueness migration can hit an immutability wall + +**Nothing prevents namespace collisions today** — the backend stores the +client-reported `namespace` verbatim (§4.2): no validation, no dedup, no uniqueness. +The only guard is the CLI's advisory `-2/-3` suffix, which a race, an older client, +or a direct API call all bypass. So backend#863's `UniqueConstraint(account, +namespace)` must assume existing duplicates and **resolve them in the migration +first** — and because `namespace` *is* the live k8s namespace (**immutable**), a real +collision can't be renamed; resolving it is destroy + rebuild of a running client. +*Mitigation:* the **code** already tells us collisions are unprevented; only the +**data** tells us whether any *exist* (paper cut vs wall), so the read-only collision +check (Appendix A) is the implementer's pre-migration step against staging/prod — not +an RFC blocker. + +### R5 — Fleet provisioning is a thundering herd on that constraint + +Automation across N identically-named boxes (`TRACEBLOC_ENROLL_TOKEN`, +Ansible/Terraform) all derive the same base slug, TOCTOU-collide, and retry against +the unique constraint at once. The cluster-id dedup covers same-box re-runs, not +N-different-boxes-same-name. *Mitigation:* atomic server-side suffix allocation, or +a per-box name convention in the automation contract (name + location are already +per-box via env, §8.3). + +### R6 — Cross-account cluster adoption (blocking) + +The `kube-system` UID is not a secret, so a *global* `cluster_id` + blind +get-or-create would let account B — pointed at account A's cluster — silently adopt +A's client. *Mitigation:* account-scoped get-or-create; a cross-account `cluster_id` +is a `409` conflict (§6.3 / §7.2). + +### R7 — Existing fleet has no `cluster_id` → re-run double-provisions (blocking) + +`cluster_id` is net-new, so every current client is null and a `cluster_id` lookup +won't match it — a naive re-run on a live box would mint a duplicate and orphan the +running client. *Mitigation:* the **CLI/installer** backfills `cluster_id` (it reads +the `kube-system` UID with the operator kubeconfig and PATCHes it on adopt — the +heartbeat sender can't read that UID), **and** the §7.2 step-2a "never mint over a +live in-namespace release" guard covers the window until a box is re-run. **Ship the +guard with/before the idempotent installer.** + +### R8 — Bootstrap supply-chain is unverified (blocking, security) + +`curl|bash` pulls ~14 sub-scripts from a mutable branch ref (no checksum/signature), +and the CLI installer's cosign check is skipped when cosign is absent (the default), +falling back to a same-channel SHA256. The most privileged code is the least +verified — upstream of every §9 mitigation. *Mitigation:* immutable release tag + +signed sub-script manifest + mandatory signature verification (§9). *(installer / cli +repos; named here.)* + +### R9 — No audit trail (blocking, compliance) + +No who-did-what-when record of provisioning / login / delete / credential-reset / +cross-account `409`. SOC 2 / HIPAA require it; the §2 regulated-org thesis depends on +it. *Mitigation:* append-only audit events from the backend, exportable (§9). +*(backend.)* + +### R10 — Multi-env config clobber (confirmed bug today) + +`~/.tracebloc` holds one `Env`+`Token`+`ActiveClientID`; `login --env` overwrites +env+token but strands the previous env's `ActiveClientID`, so a dev/stg/prod user +silently targets the wrong client. §7.5 handled only cross-*account*. *Mitigation:* +env-scoped config (profiles) or clear/reselect the active client on `Env` change +(§7.5). *(cli.)* + +### R11 — No CLI↔backend↔chart version negotiation (future-proof) + +The api client sends no `User-Agent`/version header, there's no min-version +handshake, and the chart `appVersion` isn't reported — three repos on different +cadences + a new contract = undefined failures on skew. *Mitigation:* CLI sends a +version header; backend advertises a min-supported version with a clear "upgrade +required" message; `chart_version` on the heartbeat. *(cli + backend + chart.)* + +### R12 — No clean uninstall / offboarding (ease-of-use, future-proof) + +There's no inverse of install: nothing stops the heartbeat, de-registers the box, or +clears local state, so every retired/rebuilt box becomes an R3 orphan. *Mitigation:* +`client delete --uninstall` / installer `--uninstall` that tears down the release, +de-registers the `EdgeDevice`, and clears `~/.tracebloc` — the intended cure for R3 +rebuild-orphans. *(cli + installer.)* + +### Watch-items (not blockers) + +- **`client list` "connected" is two data sources** (§12): local kube-context + reachability ("connected *here*") vs the backend heartbeat ("online *somewhere*") + — different columns, and per-row cluster probing is slow. +- **Transient credential file:** the values/secret written pre-Helm (§7.9) is a + third home for the secret — specify its cleanup post-install. +- **Heartbeat staleness** makes the delete "is it online?" guard advisory; the + teardown step is the real safety (§7.4). +- **`/device/code` is unauthenticated public surface** — rate-limit + sufficient + `user_code` entropy, or it's a DoS / guessing target (§8). +- **Compromised-client (FL) threat model** — name what a valid machine credential + can/can't do to *other* tenants' training (poisoned gradients, dataset-size lies, + model exfiltration), and link the FL-runtime doc that owns Byzantine defenses. +- **Data-residency attestation** — the auto-detected `location` (§6.7) is + residency-grade but used only for carbon; record its provenance and expose a + per-client residency attestation (a GDPR signal) separate from the mutable zone. +- **Emoji / flag glyphs** mojibake in CI logs and Windows consoles — gate *glyphs* + (not just color) on a TTY, ASCII fallbacks (`[ok]`/`[x]`), and prefer the plain + zone code (`DE`) over a flag emoji. +- **`--token` re-apply** should run the same cluster-id get-or-create (a Terraform + re-apply of the *same* box is then a no-op) and fail clearly if the k8s API isn't + reachable yet (R5 covers *different* boxes, not same-box re-apply). + +## Appendix A — name→slug reference rule & validation + +Reference algorithm (CLI ports to Go; Python shown for prototyping): + +```python +import re, unicodedata + +def slugify_dns1123(name: str) -> str: + s = unicodedata.normalize("NFKD", name).encode("ascii", "ignore").decode() + s = re.sub(r"[^a-z0-9]+", "-", s.lower()) # non-alnum -> hyphen + s = re.sub(r"-+", "-", s).strip("-") # collapse repeats, trim + return s[:63].rstrip("-") # DNS-1123 label ≤63 + +def derive(name, existing: set) -> str: + base = slugify_dns1123(name) or f"client-{short_id()}" # empty-slug guard + slug, n = base, 2 + while slug in existing: # collision suffix + suf = f"-{n}"; slug = base[:63-len(suf)].rstrip("-") + suf; n += 1 + return slug +``` + +Prototype run (2026-06-05) — every output is DNS-1123-valid (`[a-z0-9]([a-z0-9-]*[a-z0-9])?`, ≤63): + +| Input | Slug | Note | +|---|---|---| +| `divya` | `divya` | existing ns — backfill-safe, unchanged | +| `Munich Hospital — Radiology` | `munich-hospital-radiology` | em-dash + spaces | +| `GPU Server` ×3 | `gpu-server`, `gpu-server-2`, `gpu-server-3` | collision suffix | +| ` Acme Research Lab #1 ` | `acme-research-lab-1` | trim + collapse | +| `Klinikum München Röntgen` | `klinikum-munchen-rontgen` | transliteration | +| `São Paulo Edge` | `sao-paulo-edge` | transliteration | +| `北京医院` | `client-` | all-CJK → empty-slug guard | +| `---` | `client-` | punctuation-only → guard | +| `a`×80 / very long name | (truncated to 63) | length cap | + +**Known sharp edge:** a mixed name like `東京-Lab` slugifies to just `lab` (only the +ASCII survives transliteration) — semantically lossy. Acceptable for a derived slug +(display name is preserved), and the slug is surfaced in progress for the rare +hand-run that wants to override. + +**On "authoritative":** with cluster-id confirmed as the one-per-cluster guard +(§7.2), **`cluster_id` is authoritative for idempotency**; the per-account namespace +`UniqueConstraint` (backend#863) is **cosmetic cross-cluster dedup** — it keeps two +*different* clusters' slugs distinct, but is no longer what prevents a same-cluster +duplicate. The CLI `-2/-3` suffix remains best-effort UX (TOCTOU). Run this +**read-only** check (R4) against staging/prod +*before* adding the constraint — it reports the `(account, namespace)` collisions +that would block it, plus slug drift: + +```python +# READ-ONLY. python manage.py shell < check_namespace_collisions.py +from collections import Counter +from metaApi.models import EdgeDevice +try: + from common.utils.slug import slugify_dns1123 # the authoritative rule +except Exception: # fallback = RFC Appendix A rule + import re, unicodedata + def slugify_dns1123(name): + s = unicodedata.normalize("NFKD", name or "").encode("ascii", "ignore").decode() + s = re.sub(r"[^a-z0-9]+", "-", s.lower()) + return re.sub(r"-+", "-", s).strip("-")[:63].rstrip("-") + +rows = list(EdgeDevice.objects.values("id", "account_id", "first_name", "namespace")) +print(f"EdgeDevices: {len(rows)}") + +# (1) THE BLOCKER for UniqueConstraint(account, namespace): existing duplicates. +acct_ns = Counter((r["account_id"], r["namespace"]) for r in rows if r["namespace"]) +blockers = {k: c for k, c in acct_ns.items() if c > 1} +print(f"(1) (account, namespace) collisions [BLOCK the per-account constraint]: {len(blockers)}") +for (acct, ns), c in sorted(blockers.items(), key=lambda x: -x[1]): + ids = [r["id"] for r in rows if r["account_id"] == acct and r["namespace"] == ns] + print(f" account={acct} namespace={ns!r} x{c} ids={ids}") + +# (2) global namespace collisions [only matter if you pick a global constraint]. +g = Counter(r["namespace"] for r in rows if r["namespace"]) +print(f"(2) global namespace collisions: {sum(1 for v in g.values() if v > 1)}") + +# (3) blank namespace (never heartbeated) + (4) name->slug drift (informational). +print(f"(3) blank namespace: {sum(1 for r in rows if not r['namespace'])}") +drift = [(r["id"], r["first_name"], slugify_dns1123(r["first_name"]), r["namespace"]) + for r in rows if r["namespace"] and slugify_dns1123(r["first_name"]) != r["namespace"]] +empty = [r["id"] for r in rows if not slugify_dns1123(r["first_name"])] +print(f"(4) name->slug != stored namespace: {len(drift)}; slugify-to-empty: {len(empty)}") +for rid, nm, d, ns in drift[:30]: + print(f" id={rid} name={nm!r} derived={d!r} stored={ns!r}") + +print("\nVERDICT: (1) must be 0 before adding UniqueConstraint(account, namespace).") +print("If > 0, those clients share a namespace — and k8s namespaces are immutable,") +print("so resolving a collision is destroy + rebuild, not a rename (RFC R4).") +``` + +## Appendix B — closest prior art + +Tailscale (daemon enrollment via browser → node key — nearly our exact shape), +GitHub CLI (device-flow ergonomics), AWS SSO (headless device flow), cloudflared +(browser-authorized long-running tunnel). + +## Appendix C — API & data contracts (pin before parallel work) + +Everything above is design altitude; this is the wire contract the repos must agree +on. Shapes are **grounded in the CLI client already shipped/in-flight** +(`internal/api/client.go`) — the backend must match these or reconcile deliberately. +`[NEW]` marks net-new backend work; everything else the CLI already encodes. + +### C.1 Conventions + +- **Base URLs:** dev `https://dev-api.tracebloc.io` · stg `https://stg-api.tracebloc.io` + · prod `https://api.tracebloc.io` (CLI `ResolveEnv`). +- **Auth:** `Authorization: Bearer ` (ClientAccessToken — keyword **`Bearer`**, + *not* DRF's `Token`). +- **Versioning `[NEW]` (R11):** every CLI request sends + `User-Agent: tracebloc-cli/ (/)`; backend MAY reply `426 Upgrade + Required` + `{ error: "upgrade_required", min_version }` below the floor. + +### C.2 Device Authorization Grant (RFC 8628) — backend#835 `[NEW]` + +```http +POST /device/code # unauthenticated; rate-limit (§9). CLI sends no body. + 200 { device_code, user_code, verification_uri, verification_uri_complete, + expires_in, interval } + +POST /device/token # unauthenticated; rate-limit. Polled every `interval`s. + req { device_code } + 200 { token } # approved + 400 { error: "authorization_pending" } # keep polling + 400 { error: "slow_down" } # keep polling, interval++ + 400 { error: "expired_token" } # terminal + 400 { error: "access_denied" } # terminal (denied in browser) + # CLI keys off these EXACT error strings; pending/slow_down are non-terminal. + +GET /userinfo/ # Bearer → { email, type, account } (CLI confirms token post-login) +GET /activate # frontend, token-authed (R1): "connect machine X to account Y", + # binds approval to request.user (RFC §6.3) +``` + +### C.3 Provisioning — `/edge-device/` (Bearer) — backend#836 (namespace + RBAC); the `[NEW]` `cluster_id` items: backend#883 + +```http +POST /edge-device/ # get-or-create, account-scoped on cluster_id + req { first_name, namespace, location, password, cluster_id[NEW] } + # account stamped from the token (Q5); password CLI-generated, write-only + 201 # minted a new client + 200 [NEW] # adopted the existing client for this cluster_id + # (same account) — the idempotent re-run (§7.2) + 409 { error:"cluster_conflict", cluster_id } [NEW] + # cluster_id bound to ANOTHER account — never + # silent-adopt (R6) + 403 → ask-an-admin # no CLIENT_WRITE (§7.4 / Q4) + +ProvisionedClient = { id, first_name, username, namespace, location, status, + cluster_id[NEW] } + # status = the online/offline code `client list` renders; cluster_id NEW + +GET /edge-device/ → { next, results: [ProvisionedClient] } # account's clients (paginated) +GET /edge-device/admins/ → [ { name, email } ] # Q4 ask-an-admin +DELETE /edge-device// [NEW] → 204 # `client delete` (§7.4); + # guards are client-side + server RBAC +PATCH /edge-device// [NEW] { cluster_id } → 200 # adopt-backfill (R7): set cluster_id on + # an existing (null) client the CLI found live in-cluster; + # account-scoped; 409 if already set to another cluster +``` + +### C.4 Heartbeat — unchanged for `cluster_id` (R7) + +The heartbeat sender (jobs-manager) has no `namespaces` RBAC and **cannot read the +`kube-system` UID**, so it does **not** carry `cluster_id`. The CLI/installer sets +and backfills `cluster_id` instead (C.3 create + adopt-`PATCH`; the kubeconfig-holder +reads the UID and the call is Bearer-authed — no self-reported claim to spoof, §9). +The heartbeat keeps reporting `namespace`/`status`; "connected" (§7.3) = recent +heartbeat + the CLI-set `cluster_id`. + +### C.5 Audit events — backend `[NEW]` (R9), append-only + exportable + +``` +event = { id, ts, actor{ email, account }, action, cluster_id?, client_id?, source_ip, meta } +action ∈ { device.approve, client.create, client.adopt, client.delete, + credential.issue, credential.reset, client.conflict_409, + auth.login, auth.logout, token.revoke } +``` + +### C.6 Revoke — endpoint [backend#887] `[NOT YET BUILT]` (primitive: backend#845) + +```http +POST /auth/revoke # Bearer → 204 # `logout` MUST call this (not built — #887); invalidates the token + # server-side (a local clear leaves it valid — R2) +``` + +Machine-credential revoke ("kill a compromised node *now*", §9) is a **separate** +EdgeDevice action, distinct from `DELETE`: invalidate the password hash + force +re-enroll, without tearing down the cluster install. + +### C.7 Data model — backend `[NEW]` + +```python +# EdgeDevice gains one field (the kube-system namespace UID — a UUID string): +cluster_id = models.CharField(max_length=64, null=True, blank=True, db_index=True) + +class Meta: + constraints = [ + # a physical cluster maps to exactly ONE client — GLOBAL unique: + models.UniqueConstraint(fields=["cluster_id"], + condition=Q(cluster_id__isnull=False), name="uniq_edgedevice_cluster"), + # backend#863, per-account namespace (cosmetic cross-cluster dedup): + models.UniqueConstraint(fields=["account","namespace"], + name="uniq_account_namespace"), + ] +# Account-scoping for adoption lives in the POST logic (C.3), not the constraint: +# global-unique cluster_id + a 409 when the requester's account differs (R6). +``` + +**Migration order (R4 / R7) — do not reorder:** +1. Add `cluster_id` (nullable, indexed) — no constraint yet. +2. Ship the §7.2 step-2a live-release guard + the CLI/installer adopt-backfill (the + CLI PATCHes `cluster_id` on the next run — C.3) so existing boxes populate it and + never double-mint in the meantime. +3. Add the **namespace** `UniqueConstraint` *only after* the Appendix A collision + check is clean (R4). +4. Add the **cluster_id** `UniqueConstraint` once backfill coverage is acceptable. + +### C.8 CLI config — `~/.tracebloc/config.json` `[NEW]` (R10) + +```jsonc +// v2 — env-scoped (migrate v1 by wrapping it under profiles[env]); mode 0600 +{ "version": 2, "current_env": "prod", + "profiles": { + "dev": { "email", "token", "expires_at", "active_client_id" }, + "stg": { … }, "prod": { … } } } +// One active_client_id PER env → fixes the cross-env clobber (R10). +// `login --env X` switches current_env without clearing the other profiles. +``` + +[backend#830]: https://github.com/tracebloc/backend/issues/830 +[backend#835]: https://github.com/tracebloc/backend/issues/835 +[backend#836]: https://github.com/tracebloc/backend/issues/836 +[backend#862]: https://github.com/tracebloc/backend/pull/862 +[backend#863]: https://github.com/tracebloc/backend/issues/863 +[backend#883]: https://github.com/tracebloc/backend/issues/883 diff --git a/internal/api/client.go b/internal/api/client.go index 11aba24..6b182ce 100644 --- a/internal/api/client.go +++ b/internal/api/client.go @@ -177,6 +177,19 @@ func parseUpgradeRequired(raw []byte) *UpgradeRequiredError { // post sends an optional JSON body and returns the status code + raw response. func (c *Client) post(ctx context.Context, path string, body any) (int, []byte, error) { + return c.bodyRequest(ctx, http.MethodPost, path, body) +} + +// patch sends an authenticated PATCH (used for the adopt-backfill of cluster_id +// onto an existing client — RFC-0001 §7.2 / R7, backend#883). +func (c *Client) patch(ctx context.Context, path string, body any) (int, []byte, error) { + return c.bodyRequest(ctx, http.MethodPatch, path, body) +} + +// bodyRequest sends an authenticated JSON-body request (POST/PATCH) and returns +// the status code + raw response. Shared so POST and PATCH stay identical on +// auth, content-type, and the 426 upgrade-required handling. +func (c *Client) bodyRequest(ctx context.Context, method, path string, body any) (int, []byte, error) { var rdr io.Reader if body != nil { b, err := json.Marshal(body) @@ -186,7 +199,7 @@ func (c *Client) post(ctx context.Context, path string, body any) (int, []byte, rdr = bytes.NewReader(b) } url := c.BaseURL + path - req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, rdr) + req, err := http.NewRequestWithContext(ctx, method, url, rdr) if err != nil { return 0, nil, fmt.Errorf("building request: %w", err) } @@ -194,7 +207,7 @@ func (c *Client) post(ctx context.Context, path string, body any) (int, []byte, c.setAuth(req) resp, err := c.HTTP.Do(req) if err != nil { - return 0, nil, fmt.Errorf("POST %s: %w", url, err) + return 0, nil, fmt.Errorf("%s %s: %w", method, url, err) } defer func() { _ = resp.Body.Close() }() raw, err := io.ReadAll(resp.Body) @@ -318,9 +331,10 @@ func (c *Client) PollToken(ctx context.Context, deviceCode string) (string, erro // Identity is the signed-in user, from GET /userinfo/. type Identity struct { - Email string `json:"email"` - Type string `json:"type"` - Account string `json:"account"` + Email string `json:"email"` + FirstName string `json:"first_name"` + Type string `json:"type"` + Account string `json:"account"` } // WhoAmI fetches the signed-in user from the backend, authenticating with the @@ -381,8 +395,11 @@ type ProvisionedClient struct { type CreateClientRequest struct { Name string `json:"first_name"` Namespace string `json:"namespace"` - Location string `json:"location"` - Password string `json:"password"` + // Location is optional (cli#137): omitted when the operator gives no --location, + // so the backend records the client with no location rather than a silent + // default (backend#993). EdgeDevice.location is blank=True server-side. + Location string `json:"location,omitempty"` + Password string `json:"password"` // ClusterID anchors the client to this cluster (the kube-system namespace UID) // so create is get-or-create keyed on it (RFC-0001 §7.2 / backend#883). Omitted // when the cluster identity can't be read (dual-mode / legacy → plain mint). @@ -418,6 +435,31 @@ func (c *Client) CreateClient(ctx context.Context, req CreateClientRequest) (pc return &out, status == http.StatusOK, nil } +// PatchClientClusterID backfills the cluster anchor onto an existing client +// (RFC-0001 §7.2 / R7, backend#883). The existing fleet predates cluster_id, so +// a client that's already live in-cluster has a null anchor — and a plain create +// keyed on the freshly-read kube-system UID would match nothing and mint a +// duplicate. PATCH /edge-device/{id}/ stamps the UID onto the live client so it +// (not a new mint) owns this cluster. The backend enforces write-once: a 409 +// *APIError means the anchor is already set to a different value or is bound to +// another client (R6); a 403 *APIError means the caller lacks CLIENT_WRITE. +func (c *Client) PatchClientClusterID(ctx context.Context, id int, clusterID string) (*ProvisionedClient, error) { + path := fmt.Sprintf("/edge-device/%d/", id) + url := c.BaseURL + path + status, raw, err := c.patch(ctx, path, map[string]string{"cluster_id": clusterID}) + if err != nil { + return nil, err + } + if status < 200 || status >= 300 { + return nil, &APIError{StatusCode: status, Body: string(raw), URL: url} + } + var out ProvisionedClient + if err := json.Unmarshal(raw, &out); err != nil { + return nil, fmt.Errorf("decoding patch-client response: %w", err) + } + return &out, nil +} + // maxListPages bounds how many pages ListClients will follow — a backstop // against a misbehaving `next` chain, set well above any real account. const maxListPages = 100 diff --git a/internal/api/client_test.go b/internal/api/client_test.go index 621c92f..11ba575 100644 --- a/internal/api/client_test.go +++ b/internal/api/client_test.go @@ -197,6 +197,50 @@ func TestCreateClientConflict(t *testing.T) { } } +func TestPatchClientClusterID(t *testing.T) { + var gotPath, gotMethod string + var sent map[string]string + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotPath, gotMethod = r.URL.Path, r.Method + _ = json.NewDecoder(r.Body).Decode(&sent) + w.WriteHeader(http.StatusOK) + _, _ = w.Write([]byte(`{"id":7,"first_name":"c","username":"uuid-7","namespace":"ns7","cluster_id":"uid-9"}`)) + })) + defer srv.Close() + c := New("prod") + c.BaseURL = srv.URL + pc, err := c.PatchClientClusterID(context.Background(), 7, "uid-9") + if err != nil { + t.Fatal(err) + } + if gotMethod != http.MethodPatch || gotPath != "/edge-device/7/" { + t.Errorf("sent %s %s, want PATCH /edge-device/7/", gotMethod, gotPath) + } + if sent["cluster_id"] != "uid-9" { + t.Errorf("body cluster_id = %q, want uid-9", sent["cluster_id"]) + } + if pc.ClusterID != "uid-9" || pc.Username != "uuid-7" { + t.Errorf("parsed = %+v, want cluster_id=uid-9 username=uuid-7", pc) + } +} + +func TestPatchClientClusterID_ConflictIsAPIError(t *testing.T) { + // Write-once / anchor-taken → 409, surfaced as a typed APIError the caller maps + // to the cross-account guidance (R6). + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusConflict) + _, _ = w.Write([]byte(`{"error":"cluster_conflict"}`)) + })) + defer srv.Close() + c := New("prod") + c.BaseURL = srv.URL + var ae *APIError + _, err := c.PatchClientClusterID(context.Background(), 7, "uid-9") + if !errors.As(err, &ae) || ae.StatusCode != http.StatusConflict { + t.Errorf("want APIError 409, got %v", err) + } +} + // TestListClients_FollowsPagination guards that DRF pagination is still // followed end-to-end after the nextPath refactor (page 1 → page 2 → done). func TestListClients_FollowsPagination(t *testing.T) { diff --git a/internal/cli/auth.go b/internal/cli/auth.go index b30deb7..a82ef78 100644 --- a/internal/cli/auth.go +++ b/internal/cli/auth.go @@ -75,11 +75,57 @@ func runLogin(ctx context.Context, p *ui.Printer, envFlag string) error { if uri == "" { uri = dc.VerificationURI } - p.Field("open", uri) - p.Field("code", dc.UserCode) + p.Action("Open", uri) + p.Action("Enter", dc.UserCode) p.Newline() - p.Hintf("Waiting for you to approve in the browser… (Ctrl-C to cancel)") + // Poll the device-token endpoint behind a live "Waiting…" spinner (static on + // a pipe / --plain). The spinner line is cleared on return, so the ✔ / error + // below prints in its place. + tok, err := pollForToken(ctx, p, client, dc) + if err != nil { + return err + } + + // Switch the active env and write into THAT env's profile, leaving the other + // envs' tokens + active-client pointers intact (R10). Profile() returns env's + // existing profile, so a re-login preserves its active_client_id. + cfg.CurrentEnv = env + prof := cfg.Profile(env) + prof.Token = tok + // Clear any identity carried over from a PREVIOUS sign-in on this env before the + // best-effort lookup: if WhoAmI fails, a re-login as a different user on a shared + // box would otherwise keep the prior user's email/first name — and cli#137 would + // then auto-name the new client after the wrong person. Only a successful WhoAmI + // repopulates these. (Preserved from cli#137 across the cli#138 refactor.) + prof.Email, prof.FirstName = "", "" + // Confirm the freshly-issued token authenticates and capture the account to + // show + store. Best-effort: don't fail a successful sign-in if this can't run. + client.Token = tok + p.Detailf("authorized — confirming the token with the backend …") + if id, werr := client.WhoAmI(ctx); werr == nil { + prof.Email = id.Email + prof.FirstName = id.FirstName + } + if err := cfg.Save(); err != nil { + return &exitError{code: 1, err: err} + } + if prof.Email != "" { + p.Successf("Signed in as %s.", prof.Email) + } else { + p.Successf("Signed in.") + } + // The credential detail is demoted to a dim, verbose-only line — the ✔ above is + // the headline (RFC-0001 §8.1: the happy path stays quiet). + p.Detailf("token saved to ~/.tracebloc (0600)") + return nil +} + +// pollForToken runs the RFC 8628 device-token poll loop behind a live wait +// spinner, returning the issued token or an *exitError. The spinner is cleared +// on every return path (deferred Stop), so the caller prints the ✔ / error line +// on the freed line. +func pollForToken(ctx context.Context, p *ui.Printer, client *api.Client, dc *api.DeviceCodeResponse) (string, error) { interval := dc.Interval if interval <= 0 { interval = 5 @@ -89,44 +135,23 @@ func runLogin(ctx context.Context, p *ui.Printer, envFlag string) error { deadline = time.Now().Add(time.Duration(dc.ExpiresIn) * time.Second) } + sp := p.Spinner("Waiting for your browser…", "Ctrl-C to cancel") + defer sp.Stop() + for { if !deadline.IsZero() && time.Now().After(deadline) { - return &exitError{code: 1, err: errors.New("login timed out — re-run `tracebloc login`")} + return "", &exitError{code: 1, err: errors.New("login timed out — re-run `tracebloc login`")} } select { case <-ctx.Done(): - return ctx.Err() + return "", &exitError{code: 130} // Ctrl-C: exit quietly (no "Error: context canceled") case <-pollAfter(time.Duration(interval) * time.Second): } tok, err := client.PollToken(ctx, dc.DeviceCode) switch { case err == nil: - // Switch the active env and write into THAT env's profile, leaving the - // other envs' tokens + active-client pointers intact (R10). Profile() - // returns env's existing profile, so a re-login preserves its - // active_client_id rather than clobbering it. - cfg.CurrentEnv = env - prof := cfg.Profile(env) - prof.Token = tok - // Confirm the freshly-issued token actually authenticates, and - // capture the account to show + store. Best-effort: don't fail a - // successful sign-in just because this lookup couldn't run. - client.Token = tok - p.Detailf("authorized — confirming the token with the backend …") - if id, werr := client.WhoAmI(ctx); werr == nil { - prof.Email = id.Email - } - if err := cfg.Save(); err != nil { - return &exitError{code: 1, err: err} - } - p.Newline() - if prof.Email != "" { - p.Successf("Signed in as %s. Token saved to ~/.tracebloc (0600).", prof.Email) - } else { - p.Successf("Signed in. Token saved to ~/.tracebloc (0600).") - } - return nil + return tok, nil case errors.Is(err, api.ErrAuthorizationPending): // not approved yet — keep polling case errors.Is(err, api.ErrSlowDown): @@ -134,11 +159,11 @@ func runLogin(ctx context.Context, p *ui.Printer, envFlag string) error { // interval by 5 seconds for this and all subsequent polls. interval += 5 case errors.Is(err, api.ErrExpiredToken): - return &exitError{code: 1, err: errors.New("the sign-in code expired — re-run `tracebloc login`")} + return "", &exitError{code: 1, err: errors.New("the sign-in code expired — re-run `tracebloc login`")} case errors.Is(err, api.ErrAccessDenied): - return &exitError{code: 1, err: errors.New("sign-in was denied in the browser")} + return "", &exitError{code: 1, err: errors.New("sign-in was denied in the browser")} default: - return &exitError{code: 1, err: err} + return "", &exitError{code: 1, err: err} } } } @@ -209,11 +234,16 @@ func newAuthCmd() *cobra.Command { // newAuthStatusCmd implements `tracebloc auth status`. func newAuthStatusCmd() *cobra.Command { - return &cobra.Command{ + var check bool + var envFlag string + cmd := &cobra.Command{ Use: "status", Short: "Show whether you're signed in, and to which backend", Args: cobra.NoArgs, RunE: func(cmd *cobra.Command, _ []string) error { + if check { + return runAuthCheck(cmd.Context(), printerFor(cmd), envFlag) + } cfg, err := config.Load() if err != nil { return &exitError{code: 1, err: err} @@ -239,4 +269,83 @@ func newAuthStatusCmd() *cobra.Command { return nil }, } + // --check is the installer's session probe: `auth status` alone exits 0 whether + // signed in or not (it's a human display), so scripts had to grep its prose. + // --check makes the exit CODE the contract instead. + cmd.Flags().BoolVar(&check, "check", false, + "exit 0 only if signed in with a backend-valid token, else 1; silent unless --verbose") + cmd.Flags().StringVar(&envFlag, "env", "", + "backend environment the check targets: dev|stg|prod (default: $CLIENT_ENV, then prod)") + return cmd +} + +// runAuthCheck is `auth status --check`: a machine-readable session probe for the +// installer. Exit 0 = the machine is signed in to the TARGET environment with a +// token the backend accepts (a live WhoAmI); exit 1 = signed out, signed in to a +// different env, or the token was rejected/unreachable. Silent by default; +// --verbose narrates the verdict. The exit-1 paths return a nil-inner *exitError +// (IsSilentError) so main() prints nothing. +// +// The target env is resolved exactly like `login` (--env, then $CLIENT_ENV, then +// prod), and must match the signed-in CurrentEnv — otherwise the probe would OK a +// stale session for the wrong backend and the installer would skip the very +// `login` that switches env, provisioning into the wrong account (RFC-0001 §10). +func runAuthCheck(ctx context.Context, p *ui.Printer, envFlag string) error { + cfg, err := config.Load() + if err != nil { + if p.Verbose() { + p.Hintf("Not signed in. Run `tracebloc login`.") + } + return &exitError{code: 1} + } + target := api.ResolveEnv(envFlag) + if !cfg.SignedIn() || cfg.CurrentEnv != target { + if p.Verbose() { + if cfg.SignedIn() && cfg.CurrentEnv != target { + p.Hintf("Signed in to %q, but this run targets %q — run `tracebloc login`.", cfg.CurrentEnv, target) + } else { + p.Hintf("Not signed in. Run `tracebloc login`.") + } + } + return &exitError{code: 1} + } + // Signed in AND CurrentEnv == target: probe it. authedClient() builds the client + // for sessionEnv (== CurrentEnv == target) with the stored token — reuse it and + // discard its message (the exit code is the contract here). + client, _, err := authedClient() + if err != nil { + if p.Verbose() { + p.Hintf("Not signed in. Run `tracebloc login`.") + } + return &exitError{code: 1} + } + if _, err := client.WhoAmI(ctx); err != nil { + // A 426 means the CLI is too old, not that the session is invalid — surface + // the upgrade instruction (non-silent, so it shows even without --verbose) + // instead of the "re-login" advice, which wouldn't help. + var ue *api.UpgradeRequiredError + if errors.As(err, &ue) { + return &exitError{code: 1, err: ue} + } + if p.Verbose() { + // Only a 401/403 is genuinely a rejected token (where re-login helps); a + // network/DNS/5xx failure means we couldn't verify, not that the session + // is invalid — don't send the user to re-login for an outage. + var apiErr *api.APIError + if errors.As(err, &apiErr) && (apiErr.StatusCode == http.StatusUnauthorized || apiErr.StatusCode == http.StatusForbidden) { + p.Hintf("Signed-in token was rejected by the backend — run `tracebloc login`.") + } else { + p.Hintf("Couldn't verify your session with the backend (%v).", err) + } + } + return &exitError{code: 1} + } + if p.Verbose() { + if email := cfg.Current().Email; email != "" { + p.Successf("Signed in as %s.", email) + } else { + p.Successf("Signed in.") + } + } + return nil } diff --git a/internal/cli/auth_test.go b/internal/cli/auth_test.go index 8b9b3fb..6c46a38 100644 --- a/internal/cli/auth_test.go +++ b/internal/cli/auth_test.go @@ -290,3 +290,185 @@ func TestAuthStatus_NotSignedIn(t *testing.T) { t.Errorf("got:\n%s", out) } } + +// saveSignedIn writes a signed-in profile for the "dev" env into the isolated +// config dir set up by a prior withTestBackend/t.Setenv. +func saveSignedIn(t *testing.T, token string) { + t.Helper() + if err := (&config.Config{CurrentEnv: "dev", Profiles: map[string]*config.Profile{ + "dev": {Token: token, Email: "ds@co"}, + }}).Save(); err != nil { + t.Fatal(err) + } +} + +// TestAuthCheck_SignedInValid_Exit0: `auth status --check` exits 0 and is silent +// when a token is present and the backend accepts it. +func TestAuthCheck_SignedInValid_Exit0(t *testing.T) { + withTestBackend(t, func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/userinfo/" { + _, _ = w.Write([]byte(`{"email":"ds@co","account":"Acme"}`)) + return + } + t.Errorf("unexpected request path %s", r.URL.Path) + }) + saveSignedIn(t, "tok") // CurrentEnv=dev + out, err := runCmd(t, "auth", "status", "--check", "--env", "dev") + if err != nil { + t.Fatalf("--check should exit 0 when signed in + token valid, got: %v", err) + } + if strings.TrimSpace(out) != "" { + t.Errorf("--check should be silent by default, got:\n%s", out) + } +} + +// TestAuthCheck_NotSignedIn_Exit1: silent exit 1 when there's no token. +func TestAuthCheck_NotSignedIn_Exit1(t *testing.T) { + t.Setenv("TRACEBLOC_CONFIG_DIR", t.TempDir()) + out, err := runCmd(t, "auth", "status", "--check") + if got := ExitCodeFromError(err); got != 1 { + t.Fatalf("exit code = %d, want 1 (err=%v)", got, err) + } + if !IsSilentError(err) { + t.Errorf("--check exit 1 should be silent (nil-inner exitError), err=%v", err) + } + if strings.TrimSpace(out) != "" { + t.Errorf("--check should print nothing, got:\n%s", out) + } +} + +// TestAuthCheck_TokenRejected_Exit1: a stored token the backend rejects (401) +// exits 1 — not a false "signed in". +func TestAuthCheck_TokenRejected_Exit1(t *testing.T) { + withTestBackend(t, func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/userinfo/" { + w.WriteHeader(http.StatusUnauthorized) + return + } + }) + saveSignedIn(t, "stale") // CurrentEnv=dev + _, err := runCmd(t, "auth", "status", "--check", "--env", "dev") + if got := ExitCodeFromError(err); got != 1 { + t.Fatalf("exit code = %d, want 1 on a rejected token", got) + } +} + +// TestAuthCheck_VerboseNarrates: --check --verbose prints the verdict. +func TestAuthCheck_VerboseNarrates(t *testing.T) { + withTestBackend(t, func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/userinfo/" { + _, _ = w.Write([]byte(`{"email":"ds@co","account":"Acme"}`)) + } + }) + saveSignedIn(t, "tok") // CurrentEnv=dev + out, err := runCmd(t, "auth", "status", "--check", "--verbose", "--env", "dev") + if err != nil { + t.Fatalf("--check --verbose signed-in should exit 0, got: %v", err) + } + if !strings.Contains(out, "ds@co") { + t.Errorf("--verbose should narrate the account, got:\n%s", out) + } +} + +// TestAuthCheck_UpgradeRequiredSurfaces (Bugbot #146-C): a 426 from WhoAmI must +// surface the upgrade instruction (even without --verbose), not be reported like +// a rejected token telling the user to re-login. +func TestAuthCheck_UpgradeRequiredSurfaces(t *testing.T) { + withTestBackend(t, func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/userinfo/" { + w.WriteHeader(http.StatusUpgradeRequired) // 426 + _, _ = w.Write([]byte(`{"error":"upgrade_required","min_version":"1.2.3"}`)) + } + }) + saveSignedIn(t, "tok") // CurrentEnv=dev + _, err := runCmd(t, "auth", "status", "--check", "--env", "dev") // no --verbose + if got := ExitCodeFromError(err); got != 1 { + t.Fatalf("exit code = %d, want 1", got) + } + if IsSilentError(err) || err == nil || !strings.Contains(err.Error(), "too old") { + t.Errorf("a 426 must surface the upgrade message (non-silent), got: %v", err) + } +} + +// TestAuthCheck_EnvMismatch_Exit1 (Lukas #1): a valid session for one env must NOT +// pass --check for a DIFFERENT target env — otherwise the installer skips the +// login that switches env and provisions into the wrong account. Exit 1 without +// even probing the backend (no /userinfo/ call). +func TestAuthCheck_EnvMismatch_Exit1(t *testing.T) { + probed := false + withTestBackend(t, func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/userinfo/" { + probed = true + _, _ = w.Write([]byte(`{"email":"ds@co","account":"Acme"}`)) + } + }) + saveSignedIn(t, "tok") // CurrentEnv=dev, valid dev session + // The installer targets prod this run; the dev session must not satisfy it. + _, err := runCmd(t, "auth", "status", "--check", "--env", "prod") + if got := ExitCodeFromError(err); got != 1 { + t.Fatalf("exit code = %d, want 1 on an env mismatch", got) + } + if probed { + t.Error("must not probe the backend when the signed-in env differs from the target") + } +} + +// TestAuthCheck_VerboseUnreachableNotRejected (review #2): a non-401/403 WhoAmI +// failure (e.g. backend 500 / outage) must read as "couldn't verify", not a +// rejected token telling the user to re-login (which wouldn't help). +func TestAuthCheck_VerboseUnreachableNotRejected(t *testing.T) { + withTestBackend(t, func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/userinfo/" { + w.WriteHeader(http.StatusInternalServerError) // reachable but erroring — not a rejection + } + }) + saveSignedIn(t, "tok") // CurrentEnv=dev + out, err := runCmd(t, "auth", "status", "--check", "--verbose", "--env", "dev") + if got := ExitCodeFromError(err); got != 1 { + t.Fatalf("exit code = %d, want 1", got) + } + if !strings.Contains(out, "Couldn't verify your session") { + t.Errorf("a 500 should read as 'couldn't verify', got:\n%s", out) + } + if strings.Contains(out, "was rejected") { + t.Errorf("a 500 must not be labelled a rejected token: %s", out) + } +} + +// TestLogin_ClearsStaleIdentityOnWhoAmIFailure (review #3): a re-login as a +// different user must not inherit the previous user's identity if the WhoAmI +// confirmation fails — otherwise cli#137 would auto-name the new client after the +// wrong person. +func TestLogin_ClearsStaleIdentityOnWhoAmIFailure(t *testing.T) { + withTestBackend(t, func(w http.ResponseWriter, r *http.Request) { + switch r.URL.Path { + case "/device/code": + _, _ = w.Write([]byte(`{"device_code":"dc","user_code":"WDJB-MJHT","verification_uri":"https://x/activate","expires_in":600,"interval":5}`)) + case "/device/token": + _, _ = w.Write([]byte(`{"token":"bob_tok"}`)) + case "/userinfo/": + w.WriteHeader(http.StatusInternalServerError) // confirmation fails + default: + t.Errorf("unexpected request path %s", r.URL.Path) + } + }) + // Pre-existing session for a DIFFERENT user (Alice) on this env. + if err := (&config.Config{CurrentEnv: "dev", Profiles: map[string]*config.Profile{ + "dev": {Token: "alice_tok", Email: "alice@co", FirstName: "Alice"}, + }}).Save(); err != nil { + t.Fatal(err) + } + + if _, err := runCmd(t, "login"); err != nil { + t.Fatalf("login should still succeed when WhoAmI fails: %v", err) + } + + cfg, _ := config.Load() + prof := cfg.Current() + if prof.Token != "bob_tok" { + t.Errorf("token = %q, want the new bob_tok", prof.Token) + } + if prof.FirstName != "" || prof.Email != "" { + t.Errorf("stale identity leaked: FirstName=%q Email=%q (want both cleared)", prof.FirstName, prof.Email) + } +} diff --git a/internal/cli/client.go b/internal/cli/client.go index 74ad73a..99ba8e0 100644 --- a/internal/cli/client.go +++ b/internal/cli/client.go @@ -11,6 +11,7 @@ import ( "path/filepath" "strconv" "strings" + "time" "github.com/spf13/cobra" @@ -26,17 +27,22 @@ import ( // reachable cluster. var readClusterID = cluster.ClusterID +// readInClusterClient discovers a tracebloc client already live on the target +// cluster (its CLIENT_ID + namespace) — the RFC-0001 §7.2 / R7 adopt-backfill +// anchor. A package var so tests can stub it without a reachable cluster. +var readInClusterClient = cluster.DiscoverInClusterClient + // newClientCmd wires the `tracebloc client` subtree — provisioning + selecting // the client (machine) this host enrolls as. Consumes the backend provisioning // endpoints (backend#836) with the user token from `tracebloc login`. func newClientCmd() *cobra.Command { cmd := &cobra.Command{ Use: "client", - Short: "Provision and manage tracebloc clients (machines)", + Short: "Provision and manage the clients in your account", Long: `Provision a tracebloc client for this machine and list/select clients in your account. Requires sign-in first (` + "`tracebloc login`" + `).`, } - cmd.AddCommand(newClientCreateCmd(), newClientListCmd(), newClientUseCmd()) + cmd.AddCommand(newClientCreateCmd(), newClientListCmd(), newClientUseCmd(), newClientStatusCmd()) return cmd } @@ -45,17 +51,21 @@ func newClientCreateCmd() *cobra.Command { var yes bool cmd := &cobra.Command{ Use: "create", - Short: "Provision a new client for this machine (--name, --location)", + Short: "Provision a tracebloc client for this machine (auto-named; no flags required)", Args: cobra.NoArgs, RunE: func(cmd *cobra.Command, _ []string) error { return runClientCreate(cmd.Context(), printerFor(cmd), clientPrompter(), clientCreateOpts{name: name, location: location, kubeconfigPath: kubeconfigPath, contextOverride: contextOverride, credentialFile: credentialFile, yes: yes}) }, } - cmd.Flags().StringVar(&name, "name", "", - "human-readable client name (shown on your dashboard + carbon reports)") - cmd.Flags().StringVar(&location, "location", "", - "location zone for carbon footprint (e.g. DE); prompted if omitted") + // --name / --location default from their TRACEBLOC_CLIENT_* env vars so an + // unattended run (or the installer) can set them without the flag; an explicit + // flag still wins. Empty --name → auto-generated -NN; empty + // --location → sent as unset (no silent default). + cmd.Flags().StringVar(&name, "name", os.Getenv("TRACEBLOC_CLIENT_NAME"), + "client name (default: $TRACEBLOC_CLIENT_NAME, else auto-generated -NN; shown on your dashboard + carbon reports)") + cmd.Flags().StringVar(&location, "location", os.Getenv("TRACEBLOC_CLIENT_LOCATION"), + "optional location zone for carbon reporting, e.g. DE (default: $TRACEBLOC_CLIENT_LOCATION; omitted if unset)") cmd.Flags().StringVar(&kubeconfigPath, "kubeconfig", "", "path to the kubeconfig for the target cluster (default: $KUBECONFIG, then ~/.kube/config) — read to anchor the client to this cluster") cmd.Flags().StringVar(&contextOverride, "context", "", @@ -164,31 +174,15 @@ func runClientCreate(ctx context.Context, p *ui.Printer, pr prompter, opts clien ilog.Logf("authenticated; provisioning against the signed-in account") name, location := opts.name, opts.location - - // Gather inputs first (flags win; prompt only what's missing, and only on a - // TTY), then show one review + confirm — matching the dataset-push flow. - if name == "" { - if pr == nil { - return errMissingFlag("--name") - } - if name, err = pr.Input("Client name", "shown on your dashboard + carbon reports", "", validateNonEmpty); err != nil { - return mapClientErr(err) - } - } - if location == "" { - if pr == nil { - return errMissingFlag("--location") - } - // Never silent-empty: the prompt requires a non-empty zone. (Cloud / - // GeoIP auto-detect of a suggested default is a fast-follow.) - if location, err = pr.Input("Location zone (e.g. DE)", "physical zone, for the carbon footprint", "", validateNonEmpty); err != nil { - return mapClientErr(err) - } - } - // Reflect the resolved (possibly prompted) name + location back into opts, so - // the failure-path resume command includes them — opts otherwise carries only - // the flags, omitting anything the user typed at a prompt (Bugbot). - opts.name, opts.location = name, location + // cli#137 — the installer path provisions with zero flags and zero prompts: + // • name: auto-generated below (-NN) once the account's client + // list is in hand, so --name is never required; --name / + // TRACEBLOC_CLIENT_NAME still override. + // • location: optional — never prompted, never required. With no --location we + // send nothing (CreateClientRequest.Location is omitempty) and the backend + // records the client with no location rather than a silent default + // (backend#993). The backend is the source of truth for valid zones — a bad + // --location surfaces as its real create error, not a CLI-side guess. // Read the cluster anchor (kube-system UID) so create is get-or-create keyed on // it — re-running on the same cluster adopts the existing client instead of @@ -201,79 +195,179 @@ func runClientCreate(ctx context.Context, p *ui.Printer, pr prompter, opts clien } ilog.Logf("cluster anchor: %q (read err: %v)", clusterID, cidErr) - // Derive the namespace slug from the name, avoiding collisions with existing - // clients (best-effort: if the list call fails we still derive a base slug). - var existing []string - if clients, lerr := client.ListClients(ctx); lerr == nil { - for _, c := range clients { - // Skip the client already anchored to this cluster: a re-run adopts it - // (the backend keys on cluster_id), so its namespace isn't a collision. - // Counting it would bump the derived slug and show a namespace in the - // review that doesn't match the one actually adopted. + // One account-scoped client list, reused for R7 adopt-backfill and for + // namespace-collision avoidance on the mint path. A list failure is non-fatal + // for both (best-effort — we still derive a base slug / fall through to create). + accountClients, listErr := client.ListClients(ctx) + if listErr != nil { + ilog.Logf("client list failed (non-fatal): %v", listErr) + } + + // Auto-name when neither --name nor TRACEBLOC_CLIENT_NAME was given (cli#137): + // no prompt, ever. -NN, NN = the next free two-digit number across + // the account's existing clients — a second machine on the account lands on + // lukas-02, not a slug -2 bump. The derived name is already slug-clean, so it + // passes through slug.Derive unchanged (display name = namespace = handle). + if name == "" { + // Numbering is only unique if we could actually read the account's clients. + // A list failure would otherwise number against an empty set and mint a + // DETERMINISTIC duplicate (`-01` almost always already exists), whose + // name AND namespace collide with no server-side uniqueness to catch it. So + // fail closed here, exactly like the adopt pre-flight — retry, or name it by + // hand. (A supplied --name still tolerates a list blip: it's best-effort for + // slug-collision avoidance only.) + if listErr != nil { + // A 426 means the CLI is too old — retrying won't help, so surface the + // upgrade signal verbatim instead of framing it as a transient outage. + var ue *api.UpgradeRequiredError + if errors.As(listErr, &ue) { + return &exitError{code: 1, err: ue} + } + return &exitError{code: 1, err: fmt.Errorf( + "couldn't reach the backend to choose a unique client name (%v) — retry, "+ + "or pass --name explicitly", listErr)} + } + // A re-run on a cluster already anchored to a client will ADOPT that client + // (get-or-create by cluster_id), so reuse its existing name — otherwise the + // review/confirm and POST body would describe a freshly-numbered handle + // (lukas-02) that the backend then ignores in favour of the anchored record. + if anchored := anchoredClient(accountClients, clusterID); anchored != nil { + name = anchored.Name + ilog.Logf("reusing anchored client name %q (re-run on this cluster adopts it)", name) + } else { + name = autoClientName(cfg.Current(), accountClients) + ilog.Logf("auto-named client %q (no --name/TRACEBLOC_CLIENT_NAME)", name) + } + } + // Reflect the resolved name + location back into opts so the failure-path resume + // command reproduces them — opts otherwise carries only the raw flags (Bugbot). + opts.name, opts.location = name, location + + var pc *api.ProvisionedClient + var adopted bool + // password is the freshly generated machine credential; set only on the mint + // path below and consumed only by the mint output branch (an adopt keeps the + // existing credential — §7.2). + var password string + + // R7 — existing-fleet adopt-backfill (RFC-0001 §7.2). If a client is already + // live on this cluster but its backend cluster_id is null (it predates the + // anchor), a create keyed on the freshly-read UID matches nothing and mints a + // DUPLICATE, orphaning the live client. Instead adopt the live client and + // backfill its anchor onto it. + // + // Run this even when the UID read failed (clusterID == ""): detecting the live + // client (readInClusterClient) is independent of the anchor, and only the + // backfill needs it. Gating the whole block on clusterID != "" meant a + // reachable cluster whose kube-system UID read failed (RBAC on + // namespaces/kube-system, a transient API error) skipped detection entirely and + // minted over the live client — orphaning it (#158). adopt now adopts-without- + // backfill in that case; a genuinely-unreachable cluster fails detection too and + // falls through to a non-anchored mint, unchanged. + adoptedPC, handled, aerr := adoptLiveInClusterClient(ctx, p, ilog, client, opts, accountClients, listErr, clusterID) + if aerr != nil { + return aerr + } + if handled { + pc, adopted = adoptedPC, true + } + + if pc == nil { + // No live client to adopt → mint (or adopt via the backend's cluster_id + // get-or-create). Derive the namespace slug, avoiding collisions with the + // account's OTHER clients — skip the one already anchored here (a re-run + // adopts it, so its namespace isn't a collision and must not bump the slug). + // Track whether this cluster is already anchored to a client: then + // CreateClient will adopt (HTTP 200, no credential minted or printed), so the + // consent guard below must not block that idempotent re-run. + var existing []string + willAdopt := false + for _, c := range accountClients { if clusterID != "" && c.ClusterID == clusterID { + willAdopt = true continue } if c.Namespace != "" { existing = append(existing, c.Namespace) } } - } - namespace, err := slug.Derive(name, existing, "client-"+randHex(4)) - if err != nil { - return &exitError{code: 1, err: err} - } - - if pr != nil && !opts.yes { - renderClientReview(p, name, namespace, location, clusterID) - ok, cerr := pr.Confirm("Provision this client?", true) - if cerr != nil { - return mapClientErr(cerr) + namespace, derr := slug.Derive(name, existing, "client-"+randHex(4)) + if derr != nil { + return &exitError{code: 1, err: derr} } - if !ok { - ilog.Logf("cancelled by user at the confirm prompt") - p.Hintf("Cancelled.") - return nil + + if pr != nil && !opts.yes { + renderClientReview(p, name, namespace, location, clusterID) + ok, cerr := pr.Confirm("Provision this client?", true) + if cerr != nil { + return mapClientErr(cerr) + } + if !ok { + ilog.Logf("cancelled by user at the confirm prompt") + p.Hintf("Cancelled.") + return nil + } + } else if pr == nil && !opts.yes && opts.credentialFile == "" && !willAdopt { + // Non-interactive with no way to confirm AND no --credential-file: a fresh + // MINT here would side-effect silently and print the machine credential to + // stdout (into whatever captured it). Require an explicit signal first — + // --yes to consent, or --credential-file to keep the secret off stdout. + // Skipped when this cluster is already anchored (willAdopt): that re-run + // adopts and prints no credential, so it stays zero-friction. The installer + // passes both flags anyway; this only stops an accidental bare + // `client create` in a pipe / CI from leaking a freshly minted credential. + if listErr != nil { + // The list failed, so willAdopt is unknown — we can't tell a fresh mint + // (would leak a credential) from an idempotent adopt (safe). Fail closed, + // but name the real cause: a retry once the backend is reachable will + // adopt an existing client without any flag. + return &exitError{code: 1, err: fmt.Errorf( + "couldn't read the account's client list to tell whether this cluster is new "+ + "or already registered (%v) — retry when the backend is reachable (a re-run "+ + "adopts an existing client), or pass --yes/--credential-file to provision now", listErr)} + } + return &exitError{code: 1, err: errors.New( + "refusing to provision non-interactively without confirmation — pass --yes to " + + "confirm, and --credential-file to write the credential to a file instead of stdout")} } - } - // The machine credential: the CLI generates the password, the backend stores - // it (write-only). Sent on every create but used only when minting — on an - // idempotent adopt the backend keeps the existing client's credential (§7.2), - // so the generated value is never printed in that case. - password := randHex(24) - pc, adopted, err := client.CreateClient(ctx, api.CreateClientRequest{ - Name: name, - Namespace: namespace, - Location: location, - Password: password, - ClusterID: clusterID, - }) - if err != nil { - var ae *api.APIError - if errors.As(err, &ae) { - switch ae.StatusCode { - case http.StatusForbidden: - return askAnAdmin(ctx, p, client) - case http.StatusConflict: - // Per RFC C.3 the only 409 on POST /edge-device/ is cluster_conflict - // (R6): this cluster_id is bound to another account. - return &exitError{code: 1, err: errors.New( - "this cluster is already registered to a different tracebloc account — " + - "sign in to that account, or ask your admin (cluster_conflict)")} + // The machine credential: the CLI generates the password, the backend stores + // it (write-only). Sent on every create but used only when minting — on an + // idempotent adopt the backend keeps the existing client's credential (§7.2), + // so the generated value is never printed in that case. + password = randHex(24) + var cerr error + pc, adopted, cerr = client.CreateClient(ctx, api.CreateClientRequest{ + Name: name, + Namespace: namespace, + Location: location, + Password: password, + ClusterID: clusterID, + }) + if cerr != nil { + var ae *api.APIError + if errors.As(cerr, &ae) { + switch ae.StatusCode { + case http.StatusForbidden: + return askAnAdmin(ctx, p, client) + case http.StatusConflict: + // Per RFC C.3 the only 409 on POST /edge-device/ is cluster_conflict + // (R6): this cluster_id is bound to another account. + return &exitError{code: 1, err: errors.New(crossAccountConflictMsg)} + } } + return &exitError{code: 1, err: cerr} } - return &exitError{code: 1, err: err} } - cfg.Current().ActiveClientID = strconv.Itoa(pc.ID) + setActiveClient(cfg.Current(), pc) p.Newline() if adopted { - // Idempotent re-run: the backend matched this cluster_id to an existing - // client and returned it — no new credential. (The existing-fleet R7 case, - // where the backend instead matches a live in-cluster TB_CLIENT_ID whose - // cluster_id is still null and the CLI backfills it via PATCH, is the - // installer's orchestration — #838 — not done here.) + // Idempotent re-run: either the backend matched this cluster_id to an + // existing client (HTTP 200), or the R7 path above matched a live in-cluster + // client whose cluster_id was null, backfilled the anchor onto it, and + // adopted it. Either way — no new credential; the existing one stands. ilog.Logf("adopted existing client id=%d namespace=%s", pc.ID, pc.Namespace) p.Successf("This cluster is already registered as client %q (namespace %s) — adopted it.", pc.Name, pc.Namespace) p.Hintf("No new credential issued; the existing one stands. This machine is set to enroll as client %d.", pc.ID) @@ -337,6 +431,109 @@ func runClientCreate(ctx context.Context, p *ui.Printer, pr prompter, opts clien return nil } +// crossAccountConflictMsg is the guidance shown when this cluster — or the client +// already live on it — belongs to a different tracebloc account. Shared by the +// create 409 (R6) and the R7 not-owned / anchor-taken refusals so they read alike. +const crossAccountConflictMsg = "this cluster is already registered to a different tracebloc account — " + + "sign in to that account, or ask your admin (cluster_conflict)" + +// adoptLiveInClusterClient implements the RFC-0001 §7.2 / R7 adopt-backfill. It +// discovers a tracebloc client already live on the target cluster and, when the +// signed-in account owns it, backfills the cluster anchor onto it (PATCH) and +// returns it for adoption — so a re-run on a pre-anchor box reconciles the live +// client instead of minting a duplicate that orphans it. +// +// Returns (client, true, nil) when it handled provisioning (caller adopts and +// skips the mint); (nil, false, nil) when there's nothing live to adopt (caller +// mints as normal); and a non-nil error to abort — when the live client belongs +// to a DIFFERENT account (never silent-adopt across accounts — R6), when the +// anchor is already taken, or when ownership can't be verified. +func adoptLiveInClusterClient( + ctx context.Context, + p *ui.Printer, + ilog *installLog, + apiClient *api.Client, + opts clientCreateOpts, + accountClients []api.ProvisionedClient, + listErr error, + clusterID string, +) (*api.ProvisionedClient, bool, error) { + live, err := readInClusterClient(ctx, cluster.KubeconfigOptions{Path: opts.kubeconfigPath, Context: opts.contextOverride}) + if err != nil { + // Best-effort: couldn't inspect the cluster for a live client. Fall through + // to a plain create (the backend's cluster_id get-or-create still applies). + ilog.Logf("in-cluster client discovery failed (non-fatal): %v", err) + return nil, false, nil + } + if live == nil { + return nil, false, nil // fresh cluster — nothing installed to adopt + } + ilog.Logf("live in-cluster client: id=%s namespace=%s", live.ClientID, live.Namespace) + + // A client is live here — we must NOT mint over it. If the account couldn't be + // listed we can't verify ownership, so fail closed (re-run) rather than mint a + // duplicate (orphan) or adopt across accounts. + if listErr != nil { + return nil, false, &exitError{code: 1, err: fmt.Errorf( + "a tracebloc client is already running on this cluster, but listing your account to verify ownership failed (%w) — re-run once tracebloc is reachable, or resolve manually", listErr)} + } + + // Is the live client one of THIS account's? Match on the UUID auth username + // (the value stored in-cluster as CLIENT_ID); the numeric dashboard id isn't + // readable in-cluster. + var owner *api.ProvisionedClient + for i := range accountClients { + if accountClients[i].Username == live.ClientID { + owner = &accountClients[i] + break + } + } + if owner == nil { + // Live here, but not in the signed-in account — adopting it would be a silent + // cross-account takeover. Refuse (mirrors the create 409, R6). + ilog.Logf("live client %s not in this account — refusing cross-account adopt", live.ClientID) + return nil, false, &exitError{code: 1, err: errors.New(crossAccountConflictMsg)} + } + + switch { + case clusterID == "": + // A live owned client is here, but the cluster UID read failed + // (namespaces/kube-system unreadable — RBAC, a transient API error) so + // there's nothing to backfill and no anchor to compare against. The + // invariant that matters is §7.2 — never mint over a live client — so + // adopt it as-is (no Patch, no mismatch check) rather than fall through + // and mint a duplicate. Warn that idempotency wasn't (re)stamped; a + // cluster where kube-system is readable enables the backfill. + p.Hintf("A tracebloc client is already running on this cluster — adopting it. Couldn't read the cluster identity, so its idempotency anchor was left unchanged; point --kubeconfig/--context at a cluster where kube-system is readable to stamp it.") + ilog.Logf("adopting live client id=%d without anchor backfill (cluster UID unread)", owner.ID) + case owner.ClusterID == "": + // The R7 case: backfill the freshly-read anchor onto the live client. + patched, perr := apiClient.PatchClientClusterID(ctx, owner.ID, clusterID) + if perr != nil { + var ae *api.APIError + switch { + case errors.As(perr, &ae) && ae.StatusCode == http.StatusConflict: + // Anchor already taken (write-once / bound elsewhere — R6). + return nil, false, &exitError{code: 1, err: errors.New(crossAccountConflictMsg)} + case errors.As(perr, &ae) && ae.StatusCode == http.StatusForbidden: + return nil, false, askAnAdmin(ctx, p, apiClient) + } + return nil, false, &exitError{code: 1, err: fmt.Errorf("backfilling the cluster anchor onto the existing client: %w", perr)} + } + ilog.Logf("backfilled cluster_id onto client id=%d", owner.ID) + owner = patched + case owner.ClusterID != clusterID: + // The live client is anchored to a DIFFERENT cluster than the one we're + // pointed at — the kubeconfig and the in-cluster client disagree. Don't + // re-anchor (write-once); surface it rather than guess. + return nil, false, &exitError{code: 1, err: fmt.Errorf( + "the client running in this namespace is anchored to a different cluster (%s) than --kubeconfig/--context points at (%s) — check you're targeting the right cluster", + owner.ClusterID, clusterID)} + } + + return owner, true, nil +} + // resumeCommand reconstructs the `tracebloc client create` invocation to retry a // failed provision. Re-running is idempotent (RFC-0001 §7.2): on the same cluster // it adopts the existing client rather than minting a duplicate. @@ -449,17 +646,199 @@ func runClientList(ctx context.Context, p *ui.Printer) error { return nil } p.Section("Clients in your account") + active := cfg.Current().ActiveClientID for _, c := range clients { marker := "" - if strconv.Itoa(c.ID) == cfg.Current().ActiveClientID { - marker = " (active)" + if strconv.Itoa(c.ID) == active { + marker = " (active — this machine)" } p.Field(strconv.Itoa(c.ID)+marker, - fmt.Sprintf("%s namespace=%s location=%s", c.Name, c.Namespace, c.Location)) + fmt.Sprintf("%s state=%s namespace=%s location=%s", + c.Name, clientStateLabel(c.Status), c.Namespace, c.Location)) + } + // §7.3: separate "selected" (this machine's local pointer) from "connected" + // (the backend's last-heartbeat state) so a stale pointer is visible. + p.Hintf("\"active\" is this machine's selected client; state is its last reported status to tracebloc.") + return nil +} + +func newClientStatusCmd() *cobra.Command { + var wait bool + var timeout time.Duration + cmd := &cobra.Command{ + Use: "status", + Short: "Show whether tracebloc can see this machine's client (online)", + Long: `Report tracebloc's view of this machine's active client — online, offline, +or pending. With --wait, poll until tracebloc reports it online (exit 0) or the +timeout elapses (non-zero), to confirm the client connected after setup.`, + Args: cobra.NoArgs, + RunE: func(cmd *cobra.Command, _ []string) error { + // --timeout only governs the --wait poll; accepting it alone would be a + // silent no-op, so reject it rather than mislead. + if cmd.Flags().Changed("timeout") && !wait { + return &exitError{code: 1, err: errors.New("--timeout has no effect without --wait")} + } + return runClientStatus(cmd.Context(), printerFor(cmd), wait, timeout) + }, + } + cmd.Flags().BoolVar(&wait, "wait", false, "poll until tracebloc reports this client online") + cmd.Flags().DurationVar(&timeout, "timeout", 120*time.Second, "with --wait, give up after this long") + return cmd +} + +// clientStatusPollInterval is how often --wait re-checks the backend. A const, +// not a seam: tests inject through pollAfter (which ignores the duration and +// fires instantly), so the value never needs overriding. +const clientStatusPollInterval = 3 * time.Second + +func runClientStatus(ctx context.Context, p *ui.Printer, wait bool, timeout time.Duration) error { + client, cfg, err := authedClient() + if err != nil { + return &exitError{code: 1, err: err} + } + active := cfg.Current().ActiveClientID + if active == "" { + return &exitError{code: 1, err: errors.New( + "no active client on this machine — run `tracebloc client create` (or `client use `) first")} + } + + // One-shot: report the current state and exit 0 (informational). + if !wait { + st, found, lerr := lookupClientStatus(ctx, client, active) + if lerr != nil { + return &exitError{code: 1, err: lerr} + } + if !found { + return &exitError{code: 1, err: fmt.Errorf( + "active client %s isn't in your account list — run `tracebloc client list` to see "+ + "your clients, or re-run the installer to provision this machine", active)} + } + p.Section("Client status") + p.Field("state", clientStateLabel(st)) + return nil + } + + // --wait: poll the same source `client list` renders until online or timeout. + // The backend-reported status is the honest Online signal (RFC-0001 §8.5); a + // local rollout check can't tell whether tracebloc can actually see the client. + sp := p.Spinner("Waiting for tracebloc to confirm…", "") + defer sp.Stop() // leak-proof net for every return; the online path Stops explicitly before its ✔ + deadline := time.Now().Add(timeout) + var lastErr error // most recent transient (retryable) error, for the timeout message + lastState := -1 // most recent successfully-read status; -1 = none read yet + var ue *api.UpgradeRequiredError + var apiErr *api.APIError + for { + st, found, lerr := lookupClientStatus(ctx, client, active) + switch { + case errors.As(lerr, &ue): + // 426 (CLI too old) won't recover by waiting — surface the upgrade signal. + return &exitError{code: 1, err: lerr} + case errors.As(lerr, &apiErr) && (apiErr.StatusCode == http.StatusUnauthorized || apiErr.StatusCode == http.StatusForbidden): + // A revoked/expired/forbidden token (401/403) won't recover by waiting — + // the client itself may be online. Fail fast, point at sign-in. Note 429 + // and 5xx stay transient (below): those DO recover on retry. + return &exitError{code: 1, err: errors.New( + "tracebloc rejected your credentials while waiting — run `tracebloc login`, then retry")} + case lerr != nil: + lastErr = lerr // transient (5xx / 429 / network) — keep waiting, remember why + case !found: + // The active client isn't in the account (deleted / wrong account) — no + // amount of waiting surfaces it. Fail fast, matching the one-shot path. + return &exitError{code: 1, err: fmt.Errorf( + "active client %s isn't in your account list — run `tracebloc client list` to see "+ + "your clients, or re-run the installer to provision this machine", active)} + case st == clientStatusOnline: + sp.Stop() + p.Successf("tracebloc can see this client.") + return nil + default: + // A good poll (present, not online yet) supersedes any earlier transient + // error, so a later timeout reports the real state, not a stale failure. + lastErr, lastState = nil, st + } + + // --timeout caps how long we WAIT: stop once the budget is spent, and clamp + // the sleep to what remains so total runtime doesn't overshoot by a poll + // cycle. A poll begun within budget is still honored (online → ✔ above). + remaining := time.Until(deadline) + if remaining <= 0 { + switch { + case lastErr != nil: + return &exitError{code: 1, err: fmt.Errorf( + "timed out after %s waiting for tracebloc to report this client online; "+ + "the last status check failed: %v", timeout, lastErr)} + case lastState >= 0: + return &exitError{code: 1, err: fmt.Errorf( + "timed out after %s waiting for tracebloc to report this client online (last state: %s). "+ + "Run `tracebloc cluster doctor` to diagnose, or re-run the installer.", timeout, clientStateLabel(lastState))} + default: + return &exitError{code: 1, err: fmt.Errorf( + "timed out after %s before tracebloc could confirm this client — retry, "+ + "or run `tracebloc cluster doctor`.", timeout)} + } + } + wait := clientStatusPollInterval + if wait > remaining { + wait = remaining + } + select { + case <-ctx.Done(): + return &exitError{code: 130} // Ctrl-C: exit quietly (no "Error: context canceled") + case <-pollAfter(wait): + } + } +} + +// lookupClientStatus finds the account client whose numeric id matches active and +// returns its backend status code. found=false means no such client (deleted, or +// signed into the wrong account). A list error is returned verbatim so --wait can +// treat it as transient and retry. +func lookupClientStatus(ctx context.Context, client *api.Client, active string) (status int, found bool, err error) { + clients, err := client.ListClients(ctx) + if err != nil { + return 0, false, err + } + if c := findClientByID(clients, active); c != nil { + return c.Status, true, nil + } + return 0, false, nil +} + +// findClientByID returns the account client whose numeric dashboard id equals id +// (the string form stored as the active-client pointer), or nil if none match. +func findClientByID(clients []api.ProvisionedClient, id string) *api.ProvisionedClient { + for i := range clients { + if strconv.Itoa(clients[i].ID) == id { + return &clients[i] + } } return nil } +// EdgeDevice.status codes mirrored from the backend (metaApi User.py). +const ( + clientStatusOffline = 0 + clientStatusOnline = 1 + clientStatusPending = 2 +) + +// clientStateLabel maps the backend status code to a TTY/CI-safe word. Plain +// text (not an emoji glyph) on purpose — flag/emoji glyphs mojibake in CI logs +// and Windows consoles (RFC-0001 §12 watch-item). +func clientStateLabel(status int) string { + switch status { + case clientStatusOnline: + return "online" + case clientStatusOffline: + return "offline" + case clientStatusPending: + return "pending" + default: + return "unknown" + } +} + func runClientUse(ctx context.Context, p *ui.Printer, id string) error { client, cfg, err := authedClient() if err != nil { @@ -469,44 +848,114 @@ func runClientUse(ctx context.Context, p *ui.Printer, id string) error { if err != nil { return &exitError{code: 1, err: err} } - for _, c := range clients { - if strconv.Itoa(c.ID) == id { - cfg.Current().ActiveClientID = id - if serr := cfg.Save(); serr != nil { - return &exitError{code: 1, err: serr} - } - p.Successf("This machine is now set to enroll as client %s (%s).", id, c.Name) - return nil + if c := findClientByID(clients, id); c != nil { + setActiveClient(cfg.Current(), c) + if serr := cfg.Save(); serr != nil { + return &exitError{code: 1, err: serr} } + p.Successf("This machine is now set to enroll as client %s (%s).", id, c.Name) + return nil } return &exitError{code: 1, err: fmt.Errorf( "no client %s in your account — run `tracebloc client list` to see the ids", id)} } +// setActiveClient points this env's profile at c, caching its namespace and +// display name alongside the id so the data commands can bind to the active +// client's cluster (§7.3) without a backend round-trip. Callers Save() after. +func setActiveClient(p *config.Profile, c *api.ProvisionedClient) { + p.ActiveClientID = strconv.Itoa(c.ID) + p.ActiveClientNamespace = c.Namespace + p.ActiveClientName = c.Name +} + // renderClientReview shows the assembled inputs before the confirm prompt, so // the user sees the derived namespace and location before anything is created. func renderClientReview(p *ui.Printer, name, namespace, location, clusterID string) { p.Section("Review") p.Field("name", name) p.Field("namespace", namespace) - p.Field("location", location) + // Location is optional (cli#137) — only show the field when one was given, so a + // zero-prompt create doesn't render a blank "location:" line. + if location != "" { + p.Field("location", location) + } if clusterID != "" { p.Field("cluster", clusterID+" (anchors this client — re-runs adopt it)") } } -// errMissingFlag reports a required flag absent in a non-interactive run (no TTY -// to prompt — CI, a pipe, or output redirected). -func errMissingFlag(flag string) error { - return &exitError{code: 1, err: fmt.Errorf("%s is required (non-interactive — no TTY to prompt)", flag)} +// anchoredClient returns the account client already bound to clusterID (the +// kube-system UID), or nil. A non-nil result means a re-run on this cluster will +// adopt that client, so callers should reuse its name rather than mint a new one. +func anchoredClient(clients []api.ProvisionedClient, clusterID string) *api.ProvisionedClient { + if clusterID == "" { + return nil + } + for i := range clients { + if clients[i].ClusterID == clusterID { + return &clients[i] + } + } + return nil +} + +// autoClientName derives the client name used when neither --name nor +// TRACEBLOC_CLIENT_NAME was given (cli#137): -NN, where base is the +// signed-in user's first name (slugified), falling back to the email local-part, +// then a generic "client". NN is the lowest two-digit number ≥ 1 not already used +// by an existing client's name OR namespace — so a second machine on the account +// lands on lukas-02 rather than the slug package's -2 collision bump, and the +// derived name is guaranteed collision-free through slug.Derive (name = namespace). +func autoClientName(prof *config.Profile, existing []api.ProvisionedClient) string { + base := "" + if prof != nil { + if base = slug.Slugify(prof.FirstName); base == "" { + base = slug.Slugify(emailLocalPart(prof.Email)) + } + } + if base == "" { + base = "client" + } + // Reserve each existing client's handle in BOTH raw and slugified form: a legacy + // client stored with a display name like "Lukas 01" (and a blank/legacy + // namespace) must still block the derived handle "lukas-01", which is what + // slug.Derive would produce for it. + taken := make(map[string]struct{}, 4*len(existing)) + reserve := func(s string) { + if s == "" { + return + } + taken[s] = struct{}{} + taken[slug.Slugify(s)] = struct{}{} + } + for _, c := range existing { + reserve(c.Name) + reserve(c.Namespace) + } + for n := 1; ; n++ { + suffix := fmt.Sprintf("-%02d", n) + // Keep the whole handle within the DNS-1123 label cap so it survives + // slug.Derive unchanged — otherwise a long first_name yields name != namespace + // and reintroduces the exact slug -2 bump this numbering exists to avoid. + b := base + if len(b)+len(suffix) > slug.MaxLabelLength { + b = strings.TrimRight(b[:slug.MaxLabelLength-len(suffix)], "-") + } + cand := b + suffix + if _, clash := taken[cand]; !clash { + return cand + } + } } -// validateNonEmpty rejects blank prompt input. -func validateNonEmpty(s string) error { - if strings.TrimSpace(s) == "" { - return errors.New("required") +// emailLocalPart returns the part of an email before '@' (the whole string when +// there's no '@'), the fallback source for an auto-name when first_name is empty. +func emailLocalPart(email string) string { + if i := strings.IndexByte(email, '@'); i >= 0 { + return email[:i] } - return nil + return email } // mapClientErr turns a cancelled interactive prompt into a clean exit. diff --git a/internal/cli/client_test.go b/internal/cli/client_test.go index cf07021..e3c2ab6 100644 --- a/internal/cli/client_test.go +++ b/internal/cli/client_test.go @@ -6,12 +6,14 @@ import ( "encoding/json" "errors" "fmt" + "io" "net/http" "net/http/httptest" "os" "path/filepath" "strings" "testing" + "time" "github.com/tracebloc/cli/internal/api" "github.com/tracebloc/cli/internal/cluster" @@ -44,6 +46,15 @@ func withClientBackend(t *testing.T, h http.HandlerFunc) { return "", errors.New("no cluster reachable (test default)") } t.Cleanup(func() { readClusterID = origCID }) + + // Default: no client live on the cluster, so the R7 adopt-backfill path is a + // no-op and create tests never touch a real kubeconfig. Tests that exercise R7 + // override it via stubInClusterClient. + origLive := readInClusterClient + readInClusterClient = func(context.Context, cluster.KubeconfigOptions) (*cluster.InClusterClient, error) { + return nil, nil + } + t.Cleanup(func() { readInClusterClient = origLive }) } // stubClusterID overrides the cluster-anchor read for a single test. @@ -56,6 +67,37 @@ func stubClusterID(t *testing.T, uid string, err error) { t.Cleanup(func() { readClusterID = orig }) } +// stubInClusterClient overrides the live in-cluster client discovery (R7). +func stubInClusterClient(t *testing.T, lc *cluster.InClusterClient, err error) { + t.Helper() + orig := readInClusterClient + readInClusterClient = func(context.Context, cluster.KubeconfigOptions) (*cluster.InClusterClient, error) { + return lc, err + } + t.Cleanup(func() { readInClusterClient = orig }) +} + +// signInAs sets the active profile's identity (first name + email) so the cli#137 +// auto-name (-NN) is deterministic in a test. Call after +// withClientBackend, which creates the profile. +func signInAs(t *testing.T, firstName, email string) { + t.Helper() + // Guard against writing to the developer's real ~/.tracebloc: this helper only + // makes sense once withClientBackend has redirected config to a temp dir. + if os.Getenv("TRACEBLOC_CONFIG_DIR") == "" { + t.Fatal("signInAs: TRACEBLOC_CONFIG_DIR is unset — call withClientBackend first") + } + cfg, err := config.Load() + if err != nil { + t.Fatal(err) + } + p := cfg.Current() + p.FirstName, p.Email = firstName, email + if err := cfg.Save(); err != nil { + t.Fatal(err) + } +} + func TestClientCreate_Success(t *testing.T) { var body api.CreateClientRequest withClientBackend(t, func(w http.ResponseWriter, r *http.Request) { @@ -111,6 +153,141 @@ func TestClientCreate_AskAnAdmin(t *testing.T) { } } +// TestClientCreate_R7_AdoptBackfill: a client is live on this cluster with a null +// backend cluster_id (existing fleet). Create must backfill the anchor onto it +// (PATCH) and ADOPT it — never mint a duplicate (cli#131 / RFC-0001 §7.2). +func TestClientCreate_R7_AdoptBackfill(t *testing.T) { + var patchedCluster string + postCalled := false + withClientBackend(t, func(w http.ResponseWriter, r *http.Request) { + switch { + case r.Method == http.MethodGet && r.URL.Path == "/edge-device/": + // The live client is in this account, anchor still null. + _, _ = w.Write([]byte(`[{"id":7,"first_name":"box","username":"uuid-live","namespace":"ns-live","cluster_id":""}]`)) + case r.Method == http.MethodPatch && r.URL.Path == "/edge-device/7/": + var body map[string]string + _ = json.NewDecoder(r.Body).Decode(&body) + patchedCluster = body["cluster_id"] + _, _ = w.Write([]byte(`{"id":7,"first_name":"box","username":"uuid-live","namespace":"ns-live","cluster_id":"uid-9"}`)) + case r.Method == http.MethodPost && r.URL.Path == "/edge-device/": + postCalled = true + t.Error("mint POST must NOT be called on the R7 adopt-backfill path") + default: + t.Errorf("unexpected %s %s", r.Method, r.URL.Path) + } + }) + stubClusterID(t, "uid-9", nil) + stubInClusterClient(t, &cluster.InClusterClient{ClientID: "uuid-live", Namespace: "ns-live"}, nil) + + credFile := filepath.Join(t.TempDir(), "cred.env") + var out bytes.Buffer + if err := runClientCreate(context.Background(), ui.New(&out), nil, + clientCreateOpts{name: "box", location: "DE", yes: true, credentialFile: credFile}); err != nil { + t.Fatalf("create: %v", err) + } + if postCalled { + t.Fatal("minted instead of adopting") + } + if patchedCluster != "uid-9" { + t.Errorf("backfilled cluster_id = %q, want uid-9", patchedCluster) + } + cfg, _ := config.Load() + if cfg.Current().ActiveClientID != "7" { + t.Errorf("active client = %q, want 7 (the adopted live client)", cfg.Current().ActiveClientID) + } + cred, _ := os.ReadFile(credFile) + if !strings.Contains(string(cred), "TRACEBLOC_CLIENT_ADOPTED=1") || + !strings.Contains(string(cred), "TRACEBLOC_CLIENT_ID=uuid-live") || + strings.Contains(string(cred), "TRACEBLOC_CLIENT_PASSWORD") { + t.Errorf("adopt credential file wrong (want id+ADOPTED, no password):\n%s", cred) + } +} + +// TestClientCreate_R7_UIDReadFailsAdoptsLive: the kube-system UID read fails +// (e.g. RBAC on namespaces, a transient API error) while the cluster is still +// reachable enough to discover a live owned client — #158. The create must NOT +// mint over it; it adopts the live client as-is, with no backfill PATCH (there's +// no anchor to stamp). +func TestClientCreate_R7_UIDReadFailsAdoptsLive(t *testing.T) { + postCalled, patchCalled := false, false + withClientBackend(t, func(w http.ResponseWriter, r *http.Request) { + switch { + case r.Method == http.MethodGet && r.URL.Path == "/edge-device/": + _, _ = w.Write([]byte(`[{"id":7,"first_name":"box","username":"uuid-live","namespace":"ns-live","cluster_id":""}]`)) + case r.Method == http.MethodPatch && r.URL.Path == "/edge-device/7/": + patchCalled = true + t.Error("no anchor to backfill when the UID read failed — PATCH must NOT be called") + case r.Method == http.MethodPost && r.URL.Path == "/edge-device/": + postCalled = true + t.Error("must NOT mint over a live client when the UID read failed (#158)") + default: + t.Errorf("unexpected %s %s", r.Method, r.URL.Path) + } + }) + // UID read fails, but the live client is still discoverable. + stubClusterID(t, "", errors.New(`namespaces "kube-system" is forbidden`)) + stubInClusterClient(t, &cluster.InClusterClient{ClientID: "uuid-live", Namespace: "ns-live"}, nil) + + var out bytes.Buffer + if err := runClientCreate(context.Background(), ui.New(&out), nil, + clientCreateOpts{name: "box", location: "DE", yes: true}); err != nil { + t.Fatalf("create: %v", err) + } + if postCalled || patchCalled { + t.Fatal("expected a pure adopt (no mint, no backfill)") + } + cfg, _ := config.Load() + if cfg.Current().ActiveClientID != "7" { + t.Errorf("active client = %q, want 7 (adopted live client)", cfg.Current().ActiveClientID) + } +} + +// TestClientCreate_R7_AlreadyAnchored: the live client already carries this +// cluster's anchor → adopt directly, no PATCH, no mint. +func TestClientCreate_R7_AlreadyAnchored(t *testing.T) { + withClientBackend(t, func(w http.ResponseWriter, r *http.Request) { + switch { + case r.Method == http.MethodGet && r.URL.Path == "/edge-device/": + _, _ = w.Write([]byte(`[{"id":7,"first_name":"box","username":"uuid-live","namespace":"ns-live","cluster_id":"uid-9"}]`)) + default: + t.Errorf("unexpected %s %s (no PATCH/POST expected)", r.Method, r.URL.Path) + } + }) + stubClusterID(t, "uid-9", nil) + stubInClusterClient(t, &cluster.InClusterClient{ClientID: "uuid-live", Namespace: "ns-live"}, nil) + + var out bytes.Buffer + if err := runClientCreate(context.Background(), ui.New(&out), nil, + clientCreateOpts{name: "box", location: "DE", yes: true}); err != nil { + t.Fatalf("create: %v", err) + } + cfg, _ := config.Load() + if cfg.Current().ActiveClientID != "7" { + t.Errorf("active client = %q, want 7", cfg.Current().ActiveClientID) + } +} + +// TestClientCreate_R7_CrossAccountRefuse: a client is live here but it isn't in +// the signed-in account — refuse rather than mint over it or silently adopt. +func TestClientCreate_R7_CrossAccountRefuse(t *testing.T) { + withClientBackend(t, func(w http.ResponseWriter, r *http.Request) { + switch { + case r.Method == http.MethodGet && r.URL.Path == "/edge-device/": + _, _ = w.Write([]byte(`[]`)) // signed-in account owns no such client + default: + t.Errorf("unexpected %s %s (must refuse before PATCH/POST)", r.Method, r.URL.Path) + } + }) + stubClusterID(t, "uid-9", nil) + stubInClusterClient(t, &cluster.InClusterClient{ClientID: "uuid-foreign", Namespace: "ns"}, nil) + + err := runClientCreate(context.Background(), ui.New(&bytes.Buffer{}), nil, + clientCreateOpts{name: "box", location: "DE", yes: true}) + if err == nil || !strings.Contains(err.Error(), "different tracebloc account") { + t.Errorf("want cross-account refusal, got %v", err) + } +} + func TestClientCreate_RequiresLogin(t *testing.T) { t.Setenv("TRACEBLOC_CONFIG_DIR", t.TempDir()) // no config → not signed in err := runClientCreate(context.Background(), ui.New(&bytes.Buffer{}), nil, clientCreateOpts{name: "x", location: "DE", yes: true}) @@ -161,17 +338,14 @@ func TestClientCreate_Interactive(t *testing.T) { posted = true _ = json.NewDecoder(r.Body).Decode(&body) w.WriteHeader(http.StatusCreated) - _, _ = w.Write([]byte(`{"id":9,"first_name":"Lab One","username":"u-9","namespace":"lab-one","location":"DE"}`)) + _, _ = w.Write([]byte(`{"id":9,"first_name":"lab-01","username":"u-9","namespace":"lab-01"}`)) } }) + signInAs(t, "Lab", "lab@example.com") // auto-name base "lab" + // No name/location prompts anymore (cli#137): the name is auto-derived and + // location is optional, so an interactive run only reaches the confirm. confirmYes := true - pr := &fakePrompter{ - answers: map[string]string{ - "Client name": "Lab One", - "Location zone (e.g. DE)": "DE", - }, - confirm: &confirmYes, - } + pr := &fakePrompter{confirm: &confirmYes} var out bytes.Buffer if err := runClientCreate(context.Background(), ui.New(&out), pr, clientCreateOpts{}); err != nil { t.Fatalf("interactive create: %v", err) @@ -179,8 +353,11 @@ func TestClientCreate_Interactive(t *testing.T) { if !posted { t.Fatal("expected a POST after the user confirmed") } - if body.Name != "Lab One" || body.Namespace != "lab-one" || body.Location != "DE" { - t.Errorf("create body = %+v", body) + if body.Name != "lab-01" || body.Namespace != "lab-01" { + t.Errorf("auto-named create body = %+v, want name/namespace lab-01", body) + } + if body.Location != "" { + t.Errorf("location = %q, want empty (no --location given, none sent)", body.Location) } if !strings.Contains(out.String(), "Review") { t.Errorf("expected a review section before the confirm, got:\n%s", out.String()) @@ -195,14 +372,9 @@ func TestClientCreate_InteractiveCancel(t *testing.T) { } _, _ = w.Write([]byte(`[]`)) }) + signInAs(t, "Lab", "lab@example.com") confirmNo := false - pr := &fakePrompter{ - answers: map[string]string{ - "Client name": "Lab Two", - "Location zone (e.g. DE)": "US", - }, - confirm: &confirmNo, - } + pr := &fakePrompter{confirm: &confirmNo} var out bytes.Buffer if err := runClientCreate(context.Background(), ui.New(&out), pr, clientCreateOpts{}); err != nil { t.Fatalf("declining the confirm should be a clean exit, got: %v", err) @@ -553,7 +725,7 @@ func TestClientCreate_ReRunReviewShowsAdoptedNamespace(t *testing.T) { }) stubClusterID(t, "uid-1", nil) confirmYes := true - pr := &fakePrompter{answers: map[string]string{}, confirm: &confirmYes} + pr := &fakePrompter{confirm: &confirmYes} var out bytes.Buffer if err := runClientCreate(context.Background(), ui.New(&out), pr, clientCreateOpts{name: "Lab One", location: "DE"}); err != nil { @@ -563,3 +735,622 @@ func TestClientCreate_ReRunReviewShowsAdoptedNamespace(t *testing.T) { t.Errorf("review showed a bumped namespace — the cluster's own client wasn't excluded from collision detection:\n%s", out.String()) } } + +// TestClientCreate_AutoNameNoLocation is the cli#137 headline acceptance case: +// a non-interactive create with NO name and NO location flags still succeeds — +// the name is auto-generated from the signed-in identity and no location is sent. +func TestClientCreate_AutoNameNoLocation(t *testing.T) { + var body api.CreateClientRequest + rawBody := "" + withClientBackend(t, func(w http.ResponseWriter, r *http.Request) { + switch { + case r.Method == http.MethodGet && r.URL.Path == "/edge-device/": + _, _ = w.Write([]byte(`[]`)) // no existing clients on the account + case r.Method == http.MethodPost && r.URL.Path == "/edge-device/": + b, _ := io.ReadAll(r.Body) + rawBody = string(b) + _ = json.Unmarshal(b, &body) + w.WriteHeader(http.StatusCreated) + _, _ = w.Write([]byte(`{"id":5,"first_name":"lukas-01","username":"u-5","namespace":"lukas-01"}`)) + default: + t.Errorf("unexpected %s %s", r.Method, r.URL.Path) + } + }) + signInAs(t, "Lukas", "lukas@tracebloc.io") + + // pr == nil → non-interactive (the installer path). No flags at all. + if err := runClientCreate(context.Background(), ui.New(&bytes.Buffer{}), nil, + clientCreateOpts{yes: true}); err != nil { + t.Fatalf("zero-flag non-interactive create should succeed, got: %v", err) + } + if body.Name != "lukas-01" || body.Namespace != "lukas-01" { + t.Errorf("auto-name = %+v, want name/namespace lukas-01", body) + } + // "no location sent" must mean the key is absent from the JSON (omitempty), + // not just an empty string — the backend distinguishes unset from blank. + if strings.Contains(rawBody, "location") { + t.Errorf("request body carried a location key, want it omitted: %s", rawBody) + } +} + +// TestClientCreate_AutoNameNumbering: a second machine on the same account with +// the same first name numbers up (lukas-02), rather than stacking a slug -2 bump. +func TestClientCreate_AutoNameNumbering(t *testing.T) { + var body api.CreateClientRequest + withClientBackend(t, func(w http.ResponseWriter, r *http.Request) { + switch { + case r.Method == http.MethodGet && r.URL.Path == "/edge-device/": + // lukas-01 already exists on the account. + _, _ = w.Write([]byte(`[{"id":1,"first_name":"lukas-01","username":"u-1","namespace":"lukas-01"}]`)) + case r.Method == http.MethodPost && r.URL.Path == "/edge-device/": + _ = json.NewDecoder(r.Body).Decode(&body) + w.WriteHeader(http.StatusCreated) + _, _ = w.Write([]byte(`{"id":2,"first_name":"lukas-02","username":"u-2","namespace":"lukas-02"}`)) + } + }) + signInAs(t, "Lukas", "lukas@tracebloc.io") + if err := runClientCreate(context.Background(), ui.New(&bytes.Buffer{}), nil, + clientCreateOpts{yes: true}); err != nil { + t.Fatalf("create: %v", err) + } + if body.Name != "lukas-02" || body.Namespace != "lukas-02" { + t.Errorf("second machine = %+v, want lukas-02 (numbered, not a slug -2 bump)", body) + } +} + +// TestClientCreate_AutoNameEmailFallback: with no first name on the profile, the +// auto-name base falls back to the email local-part. +func TestClientCreate_AutoNameEmailFallback(t *testing.T) { + var body api.CreateClientRequest + withClientBackend(t, func(w http.ResponseWriter, r *http.Request) { + switch { + case r.Method == http.MethodGet && r.URL.Path == "/edge-device/": + _, _ = w.Write([]byte(`[]`)) + case r.Method == http.MethodPost && r.URL.Path == "/edge-device/": + _ = json.NewDecoder(r.Body).Decode(&body) + w.WriteHeader(http.StatusCreated) + _, _ = w.Write([]byte(`{"id":7,"first_name":"jane-doe-01","username":"u-7","namespace":"jane-doe-01"}`)) + } + }) + signInAs(t, "", "jane.doe@tracebloc.io") // no first name → local-part "jane.doe" → slug "jane-doe" + if err := runClientCreate(context.Background(), ui.New(&bytes.Buffer{}), nil, + clientCreateOpts{yes: true}); err != nil { + t.Fatalf("create: %v", err) + } + if body.Name != "jane-doe-01" { + t.Errorf("auto-name = %q, want jane-doe-01 (email local-part fallback)", body.Name) + } +} + +// TestClientCreate_FlagsStillHonored: explicit --name/--location are passed +// through verbatim and suppress the auto-name. +func TestClientCreate_FlagsStillHonored(t *testing.T) { + var body api.CreateClientRequest + withClientBackend(t, func(w http.ResponseWriter, r *http.Request) { + switch { + case r.Method == http.MethodGet && r.URL.Path == "/edge-device/": + _, _ = w.Write([]byte(`[]`)) + case r.Method == http.MethodPost && r.URL.Path == "/edge-device/": + _ = json.NewDecoder(r.Body).Decode(&body) + w.WriteHeader(http.StatusCreated) + _, _ = w.Write([]byte(`{"id":4,"first_name":"lab-one","username":"u-4","namespace":"lab-one","location":"US"}`)) + } + }) + signInAs(t, "Lukas", "lukas@tracebloc.io") // would auto-name lukas-01 if not overridden + if err := runClientCreate(context.Background(), ui.New(&bytes.Buffer{}), nil, + clientCreateOpts{name: "Lab One", location: "US", yes: true}); err != nil { + t.Fatalf("create: %v", err) + } + if body.Name != "Lab One" || body.Location != "US" { + t.Errorf("create body = %+v, want name 'Lab One' location US (flags verbatim)", body) + } +} + +// TestClientCreate_NonInteractiveNeedsConsent (review #1): a bare non-interactive +// run (no TTY, no --yes, no --credential-file) must NOT silently mint and print +// the credential to stdout — it fails closed with guidance before any POST. +func TestClientCreate_NonInteractiveNeedsConsent(t *testing.T) { + posted := false + withClientBackend(t, func(w http.ResponseWriter, r *http.Request) { + switch { + case r.Method == http.MethodGet && r.URL.Path == "/edge-device/": + _, _ = w.Write([]byte(`[]`)) + case r.Method == http.MethodPost && r.URL.Path == "/edge-device/": + posted = true + } + }) + signInAs(t, "Lukas", "lukas@tracebloc.io") + // pr == nil (non-interactive), yes == false, no credential file. + err := runClientCreate(context.Background(), ui.New(&bytes.Buffer{}), nil, clientCreateOpts{}) + if code := ExitCodeFromError(err); code != 1 { + t.Fatalf("want exit 1, got %d (err=%v)", code, err) + } + if err == nil || !strings.Contains(err.Error(), "refusing to provision non-interactively") { + t.Errorf("want a consent-required error, got: %v", err) + } + if posted { + t.Error("no client should be minted without --yes/--credential-file") + } +} + +// TestClientCreate_NonInteractiveWithYes: --yes alone is sufficient consent for a +// non-interactive mint (the confirm can't run, but the user opted in). +func TestClientCreate_NonInteractiveWithYes(t *testing.T) { + posted := false + withClientBackend(t, func(w http.ResponseWriter, r *http.Request) { + switch { + case r.Method == http.MethodGet && r.URL.Path == "/edge-device/": + _, _ = w.Write([]byte(`[]`)) + case r.Method == http.MethodPost && r.URL.Path == "/edge-device/": + posted = true + w.WriteHeader(http.StatusCreated) + _, _ = w.Write([]byte(`{"id":5,"first_name":"lukas-01","username":"u-5","namespace":"lukas-01"}`)) + } + }) + signInAs(t, "Lukas", "lukas@tracebloc.io") + if err := runClientCreate(context.Background(), ui.New(&bytes.Buffer{}), nil, clientCreateOpts{yes: true}); err != nil { + t.Fatalf("--yes should authorize a non-interactive mint, got: %v", err) + } + if !posted { + t.Error("expected a mint with --yes") + } +} + +// TestClientCreate_AutoNameFailsClosedOnListError (review #2): if the account's +// client list can't be read, auto-naming would number against an empty set and +// mint a deterministic duplicate. It must fail closed instead — never POST. +func TestClientCreate_AutoNameFailsClosedOnListError(t *testing.T) { + posted := false + withClientBackend(t, func(w http.ResponseWriter, r *http.Request) { + switch { + case r.Method == http.MethodGet && r.URL.Path == "/edge-device/": + w.WriteHeader(http.StatusBadGateway) // transient list failure + case r.Method == http.MethodPost && r.URL.Path == "/edge-device/": + posted = true + } + }) + signInAs(t, "Lukas", "lukas@tracebloc.io") + err := runClientCreate(context.Background(), ui.New(&bytes.Buffer{}), nil, clientCreateOpts{yes: true}) + if code := ExitCodeFromError(err); code != 1 { + t.Fatalf("want exit 1 on list failure, got %d (err=%v)", code, err) + } + if err == nil || !strings.Contains(err.Error(), "unique client name") { + t.Errorf("want a 'couldn't pick a unique name' error, got: %v", err) + } + if posted { + t.Error("must not mint (a duplicate) when the client list is unreadable") + } +} + +// TestClientCreate_AutoNameFailsClosedOnListError still lets an explicit --name +// through a list blip (the list is only best-effort for slug-collision avoidance). +func TestClientCreate_ExplicitNameToleratesListError(t *testing.T) { + posted := false + withClientBackend(t, func(w http.ResponseWriter, r *http.Request) { + switch { + case r.Method == http.MethodGet && r.URL.Path == "/edge-device/": + w.WriteHeader(http.StatusBadGateway) + case r.Method == http.MethodPost && r.URL.Path == "/edge-device/": + posted = true + w.WriteHeader(http.StatusCreated) + _, _ = w.Write([]byte(`{"id":5,"first_name":"lab","username":"u-5","namespace":"lab"}`)) + } + }) + signInAs(t, "Lukas", "lukas@tracebloc.io") + if err := runClientCreate(context.Background(), ui.New(&bytes.Buffer{}), nil, + clientCreateOpts{name: "lab", yes: true}); err != nil { + t.Fatalf("explicit --name should tolerate a list blip, got: %v", err) + } + if !posted { + t.Error("expected a mint with an explicit --name despite the list error") + } +} + +// TestClientCreate_AutoNameCapsAt63 (review #4): a very long first name must still +// produce name == namespace within the 63-char DNS label cap — no slug -NN bump. +func TestClientCreate_AutoNameCapsAt63(t *testing.T) { + var body api.CreateClientRequest + withClientBackend(t, func(w http.ResponseWriter, r *http.Request) { + switch { + case r.Method == http.MethodGet && r.URL.Path == "/edge-device/": + _, _ = w.Write([]byte(`[]`)) + case r.Method == http.MethodPost && r.URL.Path == "/edge-device/": + _ = json.NewDecoder(r.Body).Decode(&body) + w.WriteHeader(http.StatusCreated) + _, _ = w.Write([]byte(`{"id":5,"first_name":"x","username":"u-5","namespace":"x"}`)) + } + }) + signInAs(t, strings.Repeat("a", 70), "long@tracebloc.io") // slugifies to 63 a's + if err := runClientCreate(context.Background(), ui.New(&bytes.Buffer{}), nil, clientCreateOpts{yes: true}); err != nil { + t.Fatalf("create: %v", err) + } + if body.Name != body.Namespace { + t.Errorf("name != namespace for a long first name: %q vs %q", body.Name, body.Namespace) + } + if len(body.Name) > 63 { + t.Errorf("name exceeds the 63-char DNS label cap: %d chars (%q)", len(body.Name), body.Name) + } + if !strings.HasSuffix(body.Name, "-01") { + t.Errorf("expected a -01 suffix, got %q", body.Name) + } +} + +// TestClientCreate_AutoNameSurfacesUpgradeRequired (Bugbot #144-A): a 426 while +// listing clients for the auto-name must surface as an upgrade signal, not a +// "retry" reachability error — retrying an outdated CLI never succeeds. +func TestClientCreate_AutoNameSurfacesUpgradeRequired(t *testing.T) { + withClientBackend(t, func(w http.ResponseWriter, r *http.Request) { + if r.Method == http.MethodGet && r.URL.Path == "/edge-device/" { + w.WriteHeader(http.StatusUpgradeRequired) // 426 + _, _ = w.Write([]byte(`{"error":"upgrade_required","min_version":"1.2.3"}`)) + } + }) + signInAs(t, "Lukas", "lukas@tracebloc.io") + err := runClientCreate(context.Background(), ui.New(&bytes.Buffer{}), nil, clientCreateOpts{yes: true}) + if code := ExitCodeFromError(err); code != 1 { + t.Fatalf("want exit 1, got %d (err=%v)", code, err) + } + if err == nil || !strings.Contains(err.Error(), "too old") { + t.Errorf("want an upgrade-required message, got: %v", err) + } + if err != nil && strings.Contains(err.Error(), "couldn't reach the backend") { + t.Errorf("a 426 must not be framed as a transient reachability error: %v", err) + } +} + +// TestClientCreate_AutoNameReservesSluggedNames (Bugbot #144-B): a legacy client +// whose display name slugifies to the same handle (e.g. "Lukas 01" → lukas-01), +// even with a blank namespace, must reserve that handle so auto-naming skips it. +func TestClientCreate_AutoNameReservesSluggedNames(t *testing.T) { + var body api.CreateClientRequest + withClientBackend(t, func(w http.ResponseWriter, r *http.Request) { + switch { + case r.Method == http.MethodGet && r.URL.Path == "/edge-device/": + // Legacy client: raw display name, no namespace stored. + _, _ = w.Write([]byte(`[{"id":1,"first_name":"Lukas 01","username":"u-1","namespace":""}]`)) + case r.Method == http.MethodPost && r.URL.Path == "/edge-device/": + _ = json.NewDecoder(r.Body).Decode(&body) + w.WriteHeader(http.StatusCreated) + _, _ = w.Write([]byte(`{"id":2,"first_name":"lukas-02","username":"u-2","namespace":"lukas-02"}`)) + } + }) + signInAs(t, "Lukas", "lukas@tracebloc.io") + if err := runClientCreate(context.Background(), ui.New(&bytes.Buffer{}), nil, clientCreateOpts{yes: true}); err != nil { + t.Fatalf("create: %v", err) + } + if body.Name != "lukas-02" { + t.Errorf("auto-name = %q, want lukas-02 (lukas-01 reserved by legacy \"Lukas 01\")", body.Name) + } +} + +// TestClientCreate_NonInteractiveAdoptNeedsNoConsent (Bugbot follow-up): the +// non-interactive consent guard must NOT block an idempotent re-run on an +// already-anchored cluster — that path adopts (HTTP 200) and prints no +// credential, so it stays zero-friction without --yes/--credential-file. +func TestClientCreate_NonInteractiveAdoptNeedsNoConsent(t *testing.T) { + posted := false + withClientBackend(t, func(w http.ResponseWriter, r *http.Request) { + switch { + case r.Method == http.MethodGet && r.URL.Path == "/edge-device/": + // This cluster is already anchored to an existing client. + _, _ = w.Write([]byte(`[{"id":8,"first_name":"box","username":"u-8","namespace":"existing","cluster_id":"uid-1"}]`)) + case r.Method == http.MethodPost && r.URL.Path == "/edge-device/": + posted = true + w.WriteHeader(http.StatusOK) // 200 = adopted + _, _ = w.Write([]byte(`{"id":8,"first_name":"box","username":"u-8","namespace":"existing","cluster_id":"uid-1"}`)) + } + }) + stubClusterID(t, "uid-1", nil) + signInAs(t, "Lukas", "lukas@tracebloc.io") + // pr == nil, no --yes, no --credential-file: a fresh mint would be blocked, but + // this is an adopt, so it must proceed. + var out bytes.Buffer + if err := runClientCreate(context.Background(), ui.New(&out), nil, clientCreateOpts{}); err != nil { + t.Fatalf("non-interactive re-run on an anchored cluster should adopt, got: %v", err) + } + if !posted { + t.Error("expected the adopt POST") + } + if !strings.Contains(out.String(), "already registered") { + t.Errorf("expected an adopt message, got:\n%s", out.String()) + } + if strings.Contains(out.String(), "Machine credential") { + t.Errorf("adopt must not print a credential:\n%s", out.String()) + } +} + +// TestClientCreate_AutoNameReusesAnchoredClientName (Bugbot follow-up): a re-run +// without --name on an already-anchored cluster reuses the anchored client's +// name, not a freshly-numbered handle the backend would ignore on adopt. +func TestClientCreate_AutoNameReusesAnchoredClientName(t *testing.T) { + var body api.CreateClientRequest + withClientBackend(t, func(w http.ResponseWriter, r *http.Request) { + switch { + case r.Method == http.MethodGet && r.URL.Path == "/edge-device/": + // lukas-01 is already anchored to this cluster. + _, _ = w.Write([]byte(`[{"id":8,"first_name":"lukas-01","username":"u-8","namespace":"lukas-01","cluster_id":"uid-1"}]`)) + case r.Method == http.MethodPost && r.URL.Path == "/edge-device/": + _ = json.NewDecoder(r.Body).Decode(&body) + w.WriteHeader(http.StatusOK) // adopt + _, _ = w.Write([]byte(`{"id":8,"first_name":"lukas-01","username":"u-8","namespace":"lukas-01","cluster_id":"uid-1"}`)) + } + }) + stubClusterID(t, "uid-1", nil) + signInAs(t, "Lukas", "lukas@tracebloc.io") // would auto-name lukas-02 (lukas-01 taken) if not reused + if err := runClientCreate(context.Background(), ui.New(&bytes.Buffer{}), nil, clientCreateOpts{yes: true}); err != nil { + t.Fatalf("create: %v", err) + } + if body.Name != "lukas-01" { + t.Errorf("re-run should reuse the anchored name lukas-01, got %q (a fresh handle the backend would discard)", body.Name) + } +} + +// TestClientCreate_NonInteractiveListFailureExplainsCause (Bugbot follow-up): when +// the consent guard fires because the client list couldn't be read (willAdopt +// unknown), the error must name the real cause and note that a retry adopts — +// not just tell the user to pass --yes. +func TestClientCreate_NonInteractiveListFailureExplainsCause(t *testing.T) { + posted := false + withClientBackend(t, func(w http.ResponseWriter, r *http.Request) { + switch { + case r.Method == http.MethodGet && r.URL.Path == "/edge-device/": + w.WriteHeader(http.StatusBadGateway) // list fails → willAdopt unknown + case r.Method == http.MethodPost && r.URL.Path == "/edge-device/": + posted = true + } + }) + stubClusterID(t, "uid-1", nil) // a cluster that could well be adopt-only + signInAs(t, "Lukas", "lukas@tracebloc.io") + // --name given so the auto-name fast-fail is skipped and we reach the guard. + err := runClientCreate(context.Background(), ui.New(&bytes.Buffer{}), nil, clientCreateOpts{name: "lab"}) + if got := ExitCodeFromError(err); got != 1 { + t.Fatalf("exit code = %d, want 1", got) + } + if err == nil || !strings.Contains(err.Error(), "couldn't read the account's client list") { + t.Errorf("want a list-failure explanation, got: %v", err) + } + if posted { + t.Error("must not mint when the client list is unreadable") + } +} + +// setActiveClientID points the signed-in profile at client id `id` (the local +// "this machine enrolls as" pointer that `client status` reads). +func setActiveClientID(t *testing.T, id string) { + t.Helper() + cfg, err := config.Load() + if err != nil { + t.Fatal(err) + } + cfg.Current().ActiveClientID = id + if err := cfg.Save(); err != nil { + t.Fatal(err) + } +} + +// TestClientStatus_WaitOnline_Exit0: --wait exits 0 as soon as the backend +// reports the active client online, and says so. +func TestClientStatus_WaitOnline_Exit0(t *testing.T) { + withClientBackend(t, func(w http.ResponseWriter, r *http.Request) { + if r.Method == http.MethodGet && r.URL.Path == "/edge-device/" { + _, _ = w.Write([]byte(`[{"id":5,"first_name":"c","namespace":"c","status":1}]`)) // 1 = online + return + } + t.Errorf("unexpected %s %s", r.Method, r.URL.Path) + }) + setActiveClientID(t, "5") + var out bytes.Buffer + if err := runClientStatus(context.Background(), ui.New(&out), true, 5*time.Second); err != nil { + t.Fatalf("--wait should exit 0 when the client is online, got: %v", err) + } + if !strings.Contains(out.String(), "can see this client") { + t.Errorf("expected the confirmation line, got:\n%s", out.String()) + } +} + +// TestClientStatus_WaitTimeout_Exit1: --wait times out non-zero (with a +// plain-language line naming the last observed state) when the client never +// comes online. A 1ns timeout means the deadline has passed by the first check, +// so the loop never sleeps. +func TestClientStatus_WaitTimeout_Exit1(t *testing.T) { + withClientBackend(t, func(w http.ResponseWriter, r *http.Request) { + if r.Method == http.MethodGet && r.URL.Path == "/edge-device/" { + _, _ = w.Write([]byte(`[{"id":5,"first_name":"c","namespace":"c","status":0}]`)) // 0 = offline + } + }) + setActiveClientID(t, "5") + err := runClientStatus(context.Background(), ui.New(&bytes.Buffer{}), true, time.Nanosecond) + if got := ExitCodeFromError(err); got != 1 { + t.Fatalf("exit code = %d, want 1 on timeout", got) + } + if err == nil || !strings.Contains(err.Error(), "timed out") { + t.Errorf("timeout error should say so, got: %v", err) + } + if !strings.Contains(err.Error(), "offline") { + t.Errorf("timeout error should name the last state (offline), got: %v", err) + } +} + +// TestClientStatus_OneShot: without --wait, report the current state and exit 0. +func TestClientStatus_OneShot(t *testing.T) { + withClientBackend(t, func(w http.ResponseWriter, r *http.Request) { + if r.Method == http.MethodGet && r.URL.Path == "/edge-device/" { + _, _ = w.Write([]byte(`[{"id":5,"first_name":"c","namespace":"c","status":2}]`)) // 2 = pending + } + }) + setActiveClientID(t, "5") + var out bytes.Buffer + if err := runClientStatus(context.Background(), ui.New(&out), false, 0); err != nil { + t.Fatalf("one-shot status should exit 0, got: %v", err) + } + if !strings.Contains(out.String(), "pending") { + t.Errorf("expected the state label, got:\n%s", out.String()) + } +} + +// TestClientStatus_NoActiveClient: a machine with no active client is a clear +// error, not a hang or a false "offline". +func TestClientStatus_NoActiveClient(t *testing.T) { + withClientBackend(t, func(w http.ResponseWriter, r *http.Request) { + _, _ = w.Write([]byte(`[]`)) + }) + // withClientBackend leaves ActiveClientID empty. + err := runClientStatus(context.Background(), ui.New(&bytes.Buffer{}), false, 0) + if err == nil || !strings.Contains(err.Error(), "no active client") { + t.Errorf("want a 'no active client' error, got: %v", err) + } +} + +// TestClientStatus_WaitFailsFastOn426 (Bugbot #146-D): --wait must not retry a +// 426 until timeout — it fails fast with the upgrade signal. A long timeout +// proves we didn't poll to exhaustion. +func TestClientStatus_WaitFailsFastOn426(t *testing.T) { + withClientBackend(t, func(w http.ResponseWriter, r *http.Request) { + if r.Method == http.MethodGet && r.URL.Path == "/edge-device/" { + w.WriteHeader(http.StatusUpgradeRequired) // 426 + _, _ = w.Write([]byte(`{"error":"upgrade_required","min_version":"1.2.3"}`)) + } + }) + setActiveClientID(t, "5") + err := runClientStatus(context.Background(), ui.New(&bytes.Buffer{}), true, 10*time.Minute) + if got := ExitCodeFromError(err); got != 1 { + t.Fatalf("exit code = %d, want 1", got) + } + if err == nil || !strings.Contains(err.Error(), "too old") { + t.Errorf("want the upgrade message, got: %v", err) + } + if err != nil && strings.Contains(err.Error(), "timed out") { + t.Errorf("a 426 must fail fast, not time out: %v", err) + } +} + +// TestClientStatus_WaitFailsFastOnMissingClient (Bugbot #146-E): --wait must fail +// fast when the active client isn't in the account (deleted / wrong account), +// matching the one-shot path, rather than polling to the timeout. +func TestClientStatus_WaitFailsFastOnMissingClient(t *testing.T) { + withClientBackend(t, func(w http.ResponseWriter, r *http.Request) { + if r.Method == http.MethodGet && r.URL.Path == "/edge-device/" { + _, _ = w.Write([]byte(`[{"id":9,"first_name":"other","namespace":"other","status":1}]`)) // active id 5 absent + } + }) + setActiveClientID(t, "5") + // Long timeout: the test would hang if it didn't fail fast on the missing client. + err := runClientStatus(context.Background(), ui.New(&bytes.Buffer{}), true, 10*time.Minute) + if got := ExitCodeFromError(err); got != 1 { + t.Fatalf("exit code = %d, want 1", got) + } + if err == nil || !strings.Contains(err.Error(), "isn't in your account list") { + t.Errorf("want a 'not in account list' error, got: %v", err) + } +} + +// TestClientStatus_WaitTimeoutSurfacesListError (Bugbot #146-F): when every +// status check fails, the timeout message must name the real error, not a bare +// "unreachable". A 1ns timeout means the deadline passes on the first failure. +func TestClientStatus_WaitTimeoutSurfacesListError(t *testing.T) { + withClientBackend(t, func(w http.ResponseWriter, r *http.Request) { + if r.Method == http.MethodGet && r.URL.Path == "/edge-device/" { + w.WriteHeader(http.StatusInternalServerError) // persistent list failure + } + }) + setActiveClientID(t, "5") + err := runClientStatus(context.Background(), ui.New(&bytes.Buffer{}), true, time.Nanosecond) + if got := ExitCodeFromError(err); got != 1 { + t.Fatalf("exit code = %d, want 1", got) + } + if err == nil || !strings.Contains(err.Error(), "last status check failed") { + t.Errorf("timeout should surface the real list error, got: %v", err) + } + if err != nil && strings.Contains(err.Error(), "unreachable") { + t.Errorf("should not report a bare 'unreachable' when the error is known: %v", err) + } +} + +// TestClientStatus_WaitFailsFastOn401 (Lukas #2): a revoked/expired token (401) +// won't recover by waiting — --wait must fail fast pointing at sign-in, not burn +// the full timeout. A 10-minute timeout would hang the test if it didn't. +func TestClientStatus_WaitFailsFastOn401(t *testing.T) { + withClientBackend(t, func(w http.ResponseWriter, r *http.Request) { + if r.Method == http.MethodGet && r.URL.Path == "/edge-device/" { + w.WriteHeader(http.StatusUnauthorized) // dead credential + } + }) + setActiveClientID(t, "5") + err := runClientStatus(context.Background(), ui.New(&bytes.Buffer{}), true, 10*time.Minute) + if got := ExitCodeFromError(err); got != 1 { + t.Fatalf("exit code = %d, want 1", got) + } + if err == nil || !strings.Contains(err.Error(), "rejected your credentials") { + t.Errorf("want a credentials-rejected error pointing at login, got: %v", err) + } + if err != nil && strings.Contains(err.Error(), "timed out") { + t.Errorf("a 401 must fail fast, not time out: %v", err) + } +} + +// TestClientStatus_TimeoutWithoutWaitRejected (Lukas #4): --timeout without --wait +// is a silent no-op, so it's rejected rather than accepted misleadingly. +func TestClientStatus_TimeoutWithoutWaitRejected(t *testing.T) { + t.Setenv("TRACEBLOC_CONFIG_DIR", t.TempDir()) // defensive; the guard returns before config access + _, err := runCmd(t, "client", "status", "--timeout", "5s") + if got := ExitCodeFromError(err); got != 1 { + t.Fatalf("exit code = %d, want 1", got) + } + if err == nil || !strings.Contains(err.Error(), "no effect without --wait") { + t.Errorf("want a '--timeout needs --wait' error, got: %v", err) + } +} + +// TestClientStatus_WaitTimeoutClearsStaleError (Bugbot): an early transient list +// error must not mask the real last state at timeout — once a later poll succeeds +// (client present but offline), the timeout reports "offline", not the old error. +func TestClientStatus_WaitTimeoutClearsStaleError(t *testing.T) { + calls := 0 + withClientBackend(t, func(w http.ResponseWriter, r *http.Request) { + if r.Method == http.MethodGet && r.URL.Path == "/edge-device/" { + calls++ + if calls == 1 { + w.WriteHeader(http.StatusBadGateway) // one transient blip + return + } + _, _ = w.Write([]byte(`[{"id":5,"first_name":"c","namespace":"c","status":0}]`)) // then offline + } + }) + setActiveClientID(t, "5") + // Instant polling so many iterations fit inside the timeout — the first is the + // 502, all the rest are the successful offline poll that clears lastErr. + origAfter := pollAfter + pollAfter = func(time.Duration) <-chan time.Time { ch := make(chan time.Time, 1); ch <- time.Time{}; return ch } + t.Cleanup(func() { pollAfter = origAfter }) + + err := runClientStatus(context.Background(), ui.New(&bytes.Buffer{}), true, 100*time.Millisecond) + if got := ExitCodeFromError(err); got != 1 { + t.Fatalf("exit code = %d, want 1", got) + } + if err == nil || !strings.Contains(err.Error(), "last state: offline") { + t.Errorf("timeout should report the real last state (offline), got: %v", err) + } + if err != nil && strings.Contains(err.Error(), "last status check failed") { + t.Errorf("a stale transient error must be cleared after a later successful poll: %v", err) + } +} + +// TestClientStatus_WaitCtrlCIsSilent (review #4): cancelling the context (Ctrl-C) +// during --wait exits quietly with code 130 — not a bare "Error: context canceled". +func TestClientStatus_WaitCtrlCIsSilent(t *testing.T) { + withClientBackend(t, func(w http.ResponseWriter, r *http.Request) { + if r.Method == http.MethodGet && r.URL.Path == "/edge-device/" { + _, _ = w.Write([]byte(`[{"id":5,"first_name":"c","namespace":"c","status":0}]`)) // offline + } + }) + setActiveClientID(t, "5") + ctx, cancel := context.WithCancel(context.Background()) + cancel() // simulate Ctrl-C + err := runClientStatus(ctx, ui.New(&bytes.Buffer{}), true, 10*time.Second) + if got := ExitCodeFromError(err); got != 130 { + t.Fatalf("exit code = %d, want 130 on Ctrl-C", got) + } + if !IsSilentError(err) { + t.Errorf("Ctrl-C should exit silently (nil-inner exitError), got: %v", err) + } +} diff --git a/internal/cli/cluster.go b/internal/cli/cluster.go index db2cd7a..76a0952 100644 --- a/internal/cli/cluster.go +++ b/internal/cli/cluster.go @@ -4,6 +4,7 @@ import ( "context" "crypto/sha256" "encoding/hex" + "errors" "fmt" "time" @@ -28,8 +29,8 @@ func newClusterCmd() *cobra.Command { Long: `Commands for inspecting the Kubernetes cluster the CLI is configured to talk to. -Use ` + "`cluster info`" + ` to verify which cluster, namespace, and parent -tracebloc release the next ` + "`dataset push`" + ` will target. Useful as a +Use ` + "`cluster info`" + ` to verify which cluster, namespace, and +client the next ` + "`data ingest`" + ` will target. Useful as a pre-flight before doing anything destructive (e.g. ingesting into the wrong cluster).`, } @@ -64,14 +65,14 @@ func newClusterInfoCmd() *cobra.Command { cmd := &cobra.Command{ Use: "info", - Short: "Show the cluster, namespace, parent release, and ingestor SA token state", - Long: `Discovers the tracebloc parent client release in the configured + Short: "Show the cluster, namespace, client install, and ingestor token state", + Long: `Discovers the tracebloc client installed in the configured cluster + namespace and prints: • Which kubeconfig context the CLI used • The namespace it resolved to - • The parent release name + chart version + appVersion - • The jobs-manager Service the next dataset push would POST to + • The client's release name + chart version + appVersion + • The jobs-manager Service the next data ingest would POST to • The ingestor ServiceAccount the post-install hook would auth as • The cluster's configured INGESTOR_IMAGE_DIGEST default • Whether the user's kubeconfig can mint short-lived SA tokens @@ -84,7 +85,7 @@ token I expect" without leaking it to terminal scrollback. Exit codes: 0 cluster discovered + token mintable; CLI is ready - 4 cluster reachable but no tracebloc parent release found + 4 cluster reachable but no tracebloc client found 5 cluster reachable + release found but no usable SA token`, Args: cobra.NoArgs, RunE: func(cmd *cobra.Command, _ []string) error { @@ -103,10 +104,10 @@ Exit codes: cmd.Flags().StringVar(&contextOverride, "context", "", "name of the kubeconfig context to use (default: kubeconfig's current-context)") cmd.Flags().StringVarP(&nsOverride, "namespace", "n", "", - "namespace where the parent tracebloc/client release is installed (default: the context's namespace, or 'default')") + "namespace where your tracebloc client is installed (default: the context's namespace, or 'default')") cmd.Flags().StringVar(&ingestorSAName, "ingestor-sa", "", "override the ingestor ServiceAccount name (default: \"ingestor\", the chart default; "+ - "set this if you customized `ingestionAuthz.serviceAccountName` in the parent client chart)") + "set this if you customized `ingestionAuthz.serviceAccountName` in your client's install)") cmd.Flags().Int64Var(&tokenExpiry, "token-expiry-seconds", 600, "requested SA token expiration in seconds (default 600 = 10 min; ignored for static-secret fallback)") @@ -122,11 +123,17 @@ func runClusterInfo( ) error { p.Banner("tracebloc", "cluster diagnostics") - resolved, err := cluster.Load(cluster.KubeconfigOptions{ + // Bind the active client's namespace exactly like the data commands do, + // so this pre-flight targets what `data ingest` will actually target — + // and so the multi-client "set your active client" remediation works + // here too, not just on the data path. + opts := cluster.KubeconfigOptions{ Path: kubeconfigPath, Context: contextOverride, Namespace: nsOverride, - }) + } + binding := bindActiveClientNamespace(&opts) + resolved, err := cluster.Load(opts) if err != nil { // Kubeconfig errors are exit-code-3 territory (file/parse // problem, same conceptual class as `ingest validate`'s @@ -142,17 +149,27 @@ func runClusterInfo( p.Section("Kubeconfig") p.Field("context", resolved.Context) p.Field("server", resolved.ServerURL) - p.Field("namespace", resolved.Namespace) - // Discover the parent release. - release, err := cluster.DiscoverParentRelease(ctx, cs, resolved.Namespace) + // Discover the client's release — with the cluster-wide fallback scan + // when the namespace is just the kubeconfig default, so diagnostics find + // the client in its slug namespace instead of dead-ending on "default". + release, nsUsed, err := discoverRelease(ctx, p, cs, resolved.Namespace, binding.allowScan()) if err != nil { - // 4 = "cluster reachable, but no tracebloc release here." + // 4 = "cluster reachable, but no tracebloc client here." // Distinct from the kubeconfig error (3) so callers can - // branch: 3 means "fix your kubeconfig", 4 means "install - // the parent chart first". + // branch: 3 means "fix your kubeconfig", 4 means "no client + // installed on this cluster". A binding miss gets the §7.3 + // "runs elsewhere" explanation, same as the data commands. + if errors.Is(err, cluster.ErrNoParentRelease) { + return binding.explain(&exitError{code: 4, err: &noParentReleaseError{err}}) + } return &exitError{code: 4, err: err} } + resolved.Namespace = nsUsed + // Printed after discovery so it reflects the namespace the scan actually + // retargeted to — this pre-flight's whole job is to report what the next + // `data ingest` will target, so it must not show the pre-scan default. + p.Field("namespace", resolved.Namespace) // Apply the SA-name override here. Discovery doesn't read the // name from the cluster (see #7); customers with a non-default @@ -161,7 +178,7 @@ func runClusterInfo( release.IngestorSAName = ingestorSAOverride } - p.Section("Parent release") + p.Section("Client install") p.Field("name", release.ReleaseName) p.Field("chart version", release.ChartVersion) p.Field("app version", release.AppVersion) @@ -197,6 +214,6 @@ func runClusterInfo( } p.Newline() - p.Successf("Ready for `tracebloc dataset push`.") + p.Successf("Ready for `tracebloc data ingest`.") return nil } diff --git a/internal/cli/clustertarget.go b/internal/cli/clustertarget.go new file mode 100644 index 0000000..e4dee66 --- /dev/null +++ b/internal/cli/clustertarget.go @@ -0,0 +1,174 @@ +package cli + +import ( + "context" + "errors" + "fmt" + "strings" + + "k8s.io/client-go/kubernetes" + + "github.com/tracebloc/cli/internal/cluster" + "github.com/tracebloc/cli/internal/config" + "github.com/tracebloc/cli/internal/ui" +) + +// noParentReleaseError marks the exit-4 case where the reached cluster +// genuinely hosts no tracebloc release in the target namespace +// (cluster.ErrNoParentRelease) — as opposed to a present-but-PVC-missing +// release, an API/RBAC list failure, or an ambiguous multiple-release match. +// §7.3 uses it to turn an active-client binding miss into a clear "runs on +// another machine" message; the other failures keep their own diagnostics. +type noParentReleaseError struct{ err error } + +func (e *noParentReleaseError) Error() string { return e.err.Error() } +func (e *noParentReleaseError) Unwrap() error { return e.err } + +// clusterTarget bundles the cluster handles the data commands resolve from a +// kubeconfig before doing any work: the resolved config, a clientset, the +// parent tracebloc release, and — when asked — the shared data PVC. +type clusterTarget struct { + Resolved *cluster.ResolvedConfig + Clientset kubernetes.Interface + Release *cluster.ParentRelease + PVC *cluster.SharedPVC // nil unless needPVC was requested +} + +// resolveClusterTarget centralizes the identical Load → NewClientset → +// DiscoverParentRelease (→ DiscoverSharedPVC) sequence that `data ingest`, +// `data list`, and `data delete` each repeated, together with its exit-code +// contract: exit 3 for kubeconfig / clientset failures (can't reach a cluster +// at all), exit 4 for a missing tracebloc release or shared PVC (reached a +// cluster, but it isn't a tracebloc workspace). +// +// `cluster doctor` is deliberately NOT a caller — it has a different exit +// contract (2/3 escalation, with discovery reported as a check Result rather +// than a hard error). +func resolveClusterTarget(ctx context.Context, p *ui.Printer, opts cluster.KubeconfigOptions, b activeClientBinding, needPVC bool) (*clusterTarget, error) { + resolved, err := cluster.Load(opts) + if err != nil { + return nil, &exitError{code: 3, err: fmt.Errorf("loading kubeconfig: %w", err)} + } + cs, err := cluster.NewClientset(resolved) + if err != nil { + return nil, &exitError{code: 3, err: err} + } + // The cluster-wide fallback scan only engages when the target namespace is + // the kubeconfig's default — i.e. nobody chose it: not the user (explicit + // --namespace/--context) and not the active-client binding. A binding miss + // must NOT silently redirect to some other client (§7.5 — that could be a + // different machine's client); it keeps the §7.3 "runs elsewhere" message. + release, nsUsed, err := discoverRelease(ctx, p, cs, resolved.Namespace, b.allowScan()) + if err != nil { + // Only a genuine "namespace has no release" maps to the §7.3 + // "runs elsewhere" rewrite; an API/RBAC list failure or an + // ambiguous multiple-release match keeps its own message. + if errors.Is(err, cluster.ErrNoParentRelease) { + return nil, &exitError{code: 4, err: &noParentReleaseError{err}} + } + return nil, &exitError{code: 4, err: err} + } + // The scan may have retargeted discovery to the namespace that actually + // hosts the client; everything downstream (PVC discovery, dataset listing, + // prints) keys on Resolved.Namespace, so it must follow. + resolved.Namespace = nsUsed + t := &clusterTarget{Resolved: resolved, Clientset: cs, Release: release} + if needPVC { + pvc, err := cluster.DiscoverSharedPVC(ctx, cs, resolved.Namespace) + if err != nil { + return nil, &exitError{code: 4, err: err} + } + t.PVC = pvc + } + return t, nil +} + +// discoverRelease wraps DiscoverParentRelease with the cluster-wide fallback +// scan: when allowScan is set and the target namespace hosts no client, every +// namespace is scanned for one. Exactly one → target it, with a visible note +// (never a silent redirect); several → name them and ask the user to pick; +// none, or a scan failure (e.g. RBAC forbids the cluster-wide list) → the +// original discovery error stands. Returns the namespace actually used. +func discoverRelease(ctx context.Context, p *ui.Printer, cs kubernetes.Interface, namespace string, allowScan bool) (*cluster.ParentRelease, string, error) { + release, err := cluster.DiscoverParentRelease(ctx, cs, namespace) + if err == nil || !allowScan || !errors.Is(err, cluster.ErrNoParentRelease) { + return release, namespace, err + } + found, scanErr := cluster.FindClientNamespaces(ctx, cs) + if scanErr != nil || len(found) == 0 { + return nil, namespace, err + } + if len(found) > 1 { + return nil, namespace, fmt.Errorf( + "%w in namespace %q, but tracebloc clients are running in: %s. "+ + "Pass --namespace to pick one, or set your active client with `tracebloc client use`.", + cluster.ErrNoParentRelease, namespace, strings.Join(found, ", ")) + } + if p != nil { + p.Infof("No client in namespace %q — using the one in %q (override with --namespace).", namespace, found[0]) + } + release, err = cluster.DiscoverParentRelease(ctx, cs, found[0]) + return release, found[0], err +} + +// activeClientBinding records that a data command defaulted its target +// namespace to the active client's cached namespace (§7.3), so a subsequent +// "no release here" failure can be explained as "the active client runs +// elsewhere" rather than a bare discovery error. +type activeClientBinding struct { + applied bool + explicit bool // user pinned --namespace/--context themselves + name string + namespace string +} + +// bindActiveClientNamespace defaults opts.Namespace to the active client's +// cached namespace when the user overrode neither --namespace nor --context. +// It never fails: no config, no active client, or no cached namespace all +// leave opts untouched (unchanged current-context behavior), so this is +// backward compatible for anyone who hasn't run `client use`/`create`. +func bindActiveClientNamespace(opts *cluster.KubeconfigOptions) activeClientBinding { + if opts.Namespace != "" || opts.Context != "" { + return activeClientBinding{explicit: true} // user was explicit — don't second-guess + } + cfg, err := config.Load() + if err != nil { + return activeClientBinding{} + } + p := cfg.Current() + if p.ActiveClientNamespace == "" { + return activeClientBinding{} + } + opts.Namespace = p.ActiveClientNamespace + return activeClientBinding{applied: true, name: p.ActiveClientName, namespace: p.ActiveClientNamespace} +} + +// allowScan reports whether the cluster-wide fallback scan may engage: only +// when the target namespace is the kubeconfig's default — i.e. nobody chose +// it. An explicit --namespace/--context is never second-guessed, and a +// binding miss must NOT silently retarget to some other client (§7.5 — it +// could be a different machine's); it keeps the §7.3 "runs elsewhere" message. +func (b activeClientBinding) allowScan() bool { return !b.applied && !b.explicit } + +// explain rewrites a "no tracebloc release in namespace" failure (exit 4) into +// §7.3's "client runs on another machine" guidance when the target namespace +// came from the active-client binding: the cluster the kubeconfig reaches +// doesn't host that client. Non-binding errors (and PVC-missing, where the +// release *was* found) pass through unchanged. +func (b activeClientBinding) explain(err error) error { + if !b.applied { + return err + } + var npr *noParentReleaseError + if !errors.As(err, &npr) { + return err + } + handle := b.name + if handle == "" { + handle = b.namespace + } + return &exitError{code: 4, err: fmt.Errorf( + "active client %q runs on another machine — namespace %q isn't on the cluster your kubeconfig points at; "+ + "run this command there, `tracebloc client use` a client on this cluster, or override with --namespace/--context", + handle, b.namespace)} +} diff --git a/internal/cli/clustertarget_test.go b/internal/cli/clustertarget_test.go new file mode 100644 index 0000000..d62411d --- /dev/null +++ b/internal/cli/clustertarget_test.go @@ -0,0 +1,230 @@ +package cli + +import ( + "bytes" + "context" + "errors" + "strings" + "testing" + + appsv1 "k8s.io/api/apps/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/client-go/kubernetes/fake" + + "github.com/tracebloc/cli/internal/api" + "github.com/tracebloc/cli/internal/cluster" + "github.com/tracebloc/cli/internal/config" + "github.com/tracebloc/cli/internal/ui" +) + +func TestSetActiveClient_CachesNamespaceAndName(t *testing.T) { + p := &config.Profile{} + setActiveClient(p, &api.ProvisionedClient{ID: 7, Name: "Lab A", Namespace: "lab-a", Location: "FR"}) + if p.ActiveClientID != "7" || p.ActiveClientNamespace != "lab-a" || p.ActiveClientName != "Lab A" { + t.Errorf("profile after setActiveClient = %+v", p) + } +} + +func TestClientStateLabel(t *testing.T) { + cases := map[int]string{ + clientStatusOnline: "online", + clientStatusOffline: "offline", + clientStatusPending: "pending", + 99: "unknown", + } + for status, want := range cases { + if got := clientStateLabel(status); got != want { + t.Errorf("clientStateLabel(%d) = %q, want %q", status, got, want) + } + } +} + +// writeActiveClientConfig writes a signed-in dev profile whose active client +// carries the given namespace + name, into a temp config dir. +func writeActiveClientConfig(t *testing.T, namespace, name string) { + t.Helper() + t.Setenv("TRACEBLOC_CONFIG_DIR", t.TempDir()) + cfg := &config.Config{CurrentEnv: "dev", Profiles: map[string]*config.Profile{ + "dev": {Token: "tok", ActiveClientID: "5", ActiveClientNamespace: namespace, ActiveClientName: name}, + }} + if err := cfg.Save(); err != nil { + t.Fatal(err) + } +} + +func TestBindActiveClientNamespace_DefaultsFromActiveClient(t *testing.T) { + writeActiveClientConfig(t, "munich-radiology", "Munich Radiology") + opts := cluster.KubeconfigOptions{} + b := bindActiveClientNamespace(&opts) + if !b.applied { + t.Fatal("binding not applied") + } + if opts.Namespace != "munich-radiology" { + t.Errorf("opts.Namespace = %q, want munich-radiology", opts.Namespace) + } + if b.name != "Munich Radiology" || b.namespace != "munich-radiology" { + t.Errorf("binding = %+v", b) + } +} + +func TestBindActiveClientNamespace_ExplicitOverrideWins(t *testing.T) { + writeActiveClientConfig(t, "munich-radiology", "Munich Radiology") + + // --namespace set → don't touch it. + optsNS := cluster.KubeconfigOptions{Namespace: "chosen"} + if b := bindActiveClientNamespace(&optsNS); b.applied || optsNS.Namespace != "chosen" { + t.Errorf("with --namespace: applied=%v ns=%q, want false/chosen", b.applied, optsNS.Namespace) + } + + // --context set → user is steering the cluster; don't bind. + optsCtx := cluster.KubeconfigOptions{Context: "some-ctx"} + if b := bindActiveClientNamespace(&optsCtx); b.applied || optsCtx.Namespace != "" { + t.Errorf("with --context: applied=%v ns=%q, want false/empty", b.applied, optsCtx.Namespace) + } +} + +func TestBindActiveClientNamespace_NoActiveClient(t *testing.T) { + t.Setenv("TRACEBLOC_CONFIG_DIR", t.TempDir()) // no config written → nothing cached + opts := cluster.KubeconfigOptions{} + if b := bindActiveClientNamespace(&opts); b.applied || opts.Namespace != "" { + t.Errorf("no active client: applied=%v ns=%q, want false/empty", b.applied, opts.Namespace) + } +} + +func TestActiveClientBinding_Explain(t *testing.T) { + noRelease := &exitError{code: 4, err: &noParentReleaseError{errors.New("no release")}} + pvcMissing := &exitError{code: 4, err: errors.New("shared PVC not bound")} + + // Applied + "no release here" → rewritten to the §7.3 guidance. + bound := activeClientBinding{applied: true, name: "gpu-box-01", namespace: "gpu-box-01"} + got := bound.explain(noRelease) + if got == noRelease { + t.Fatal("expected a rewritten error") + } + if !strings.Contains(got.Error(), "another machine") || !strings.Contains(got.Error(), "gpu-box-01") { + t.Errorf("rewritten error = %q", got.Error()) + } + var ee *exitError + if !errors.As(got, &ee) || ee.Code() != 4 { + t.Errorf("rewritten error should stay exit 4, got %v", got) + } + + // Applied but a PVC failure (release WAS found) → pass through untouched. + if bound.explain(pvcMissing) != pvcMissing { + t.Error("PVC-missing error should not be rewritten") + } + + // Not applied → always pass through. + if (activeClientBinding{}).explain(noRelease) != noRelease { + t.Error("unbound explain should pass the error through") + } +} + +// jmDep builds a chart-labeled jobs-manager Deployment in the given namespace, +// for the fallback-scan tests (mirrors the cluster package's fixture). +func jmDep(namespace string) *appsv1.Deployment { + return &appsv1.Deployment{ + ObjectMeta: metav1.ObjectMeta{ + Name: "tracebloc-jobs-manager", + Namespace: namespace, + Labels: map[string]string{ + "app.kubernetes.io/name": "client", + "app.kubernetes.io/instance": "tracebloc", + "app.kubernetes.io/managed-by": "Helm", + "helm.sh/chart": "client-1.6.0", + }, + }, + } +} + +// The cluster-wide fallback scan (§7.3): a default-namespace miss must find +// the single client in its slug namespace and retarget — visibly, not +// silently — instead of dead-ending on "default". +func TestDiscoverRelease_ScanFindsSingleClientElsewhere(t *testing.T) { + cs := fake.NewSimpleClientset(jmDep("lukas-01")) + var buf bytes.Buffer + p := ui.New(&buf, ui.WithColor(false)) + release, nsUsed, err := discoverRelease(context.Background(), p, cs, "default", true) + if err != nil { + t.Fatalf("expected scan to find the client, got: %v", err) + } + if nsUsed != "lukas-01" { + t.Errorf("nsUsed = %q, want lukas-01", nsUsed) + } + if release == nil || release.ReleaseName != "tracebloc" { + t.Errorf("release = %+v", release) + } + // never a silent redirect + if !strings.Contains(buf.String(), "lukas-01") { + t.Errorf("expected a visible note about the redirect, got: %q", buf.String()) + } +} + +func TestDiscoverRelease_ScanMultipleNamespacesRefuses(t *testing.T) { + cs := fake.NewSimpleClientset(jmDep("alpha"), jmDep("beta")) + _, _, err := discoverRelease(context.Background(), nil, cs, "default", true) + if err == nil { + t.Fatal("expected an error for multiple client namespaces") + } + for _, want := range []string{"alpha", "beta", "--namespace", "client use"} { + if !strings.Contains(err.Error(), want) { + t.Errorf("error should mention %q, got: %s", want, err) + } + } + if !errors.Is(err, cluster.ErrNoParentRelease) { + t.Errorf("multi-namespace refusal should stay errors.Is-identifiable, got: %v", err) + } +} + +func TestDiscoverRelease_NoScanWhenExplicit(t *testing.T) { + // The client exists in lukas-01, but the caller pinned the namespace — + // the scan must NOT engage and the plain discovery error stands. + cs := fake.NewSimpleClientset(jmDep("lukas-01")) + _, nsUsed, err := discoverRelease(context.Background(), nil, cs, "default", false) + if err == nil { + t.Fatal("expected the namespace miss to stand when scan is disallowed") + } + if nsUsed != "default" { + t.Errorf("nsUsed = %q, want default (no retarget)", nsUsed) + } + if !errors.Is(err, cluster.ErrNoParentRelease) { + t.Errorf("expected ErrNoParentRelease, got: %v", err) + } +} + +func TestDiscoverRelease_ScanFindsNothingKeepsOriginalError(t *testing.T) { + cs := fake.NewSimpleClientset() // empty cluster + _, _, err := discoverRelease(context.Background(), nil, cs, "default", true) + if err == nil { + t.Fatal("expected an error on an empty cluster") + } + if !errors.Is(err, cluster.ErrNoParentRelease) { + t.Errorf("expected ErrNoParentRelease, got: %v", err) + } + // The no-client error must stay customer-actionable without Helm. + if strings.Contains(err.Error(), "helm") { + t.Errorf("error must not tell customers to run helm: %s", err) + } +} + +// The §7.5 contract at the caller level: the scan may engage ONLY when nobody +// chose the namespace — never for an explicit flag, never for a binding miss +// (which could silently retarget a different machine's client). +func TestActiveClientBinding_AllowScan(t *testing.T) { + cases := []struct { + name string + b activeClientBinding + want bool + }{ + {"kubeconfig default (nobody chose)", activeClientBinding{}, true}, + {"explicit --namespace/--context", activeClientBinding{explicit: true}, false}, + {"active-client binding applied", activeClientBinding{applied: true}, false}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + if got := c.b.allowScan(); got != c.want { + t.Errorf("allowScan() = %v, want %v", got, c.want) + } + }) + } +} diff --git a/internal/cli/data.go b/internal/cli/data.go index b91c8bb..310ecb2 100644 --- a/internal/cli/data.go +++ b/internal/cli/data.go @@ -6,6 +6,7 @@ import ( "errors" "fmt" "io" + "k8s.io/client-go/kubernetes" "os" "path/filepath" "strings" @@ -34,13 +35,14 @@ func newDataCmd() *cobra.Command { cmd := &cobra.Command{ Use: "data", Aliases: []string{"dataset"}, - Short: "Manage datasets in the parent client release", - Long: `Commands for staging and managing datasets on the cluster's -shared PVC. + Short: "Manage the datasets in your client", + Long: `Commands for staging and managing the datasets your client holds — +the data models train on. It stays on your infrastructure. -` + "`data ingest`" + ` stages a local dataset to the cluster's shared -PVC, submits the ingestion run to jobs-manager, and watches the -ingestor Job to completion (streaming logs + the final summary). +` + "`data ingest`" + ` stages a local dataset into your client's storage, +submits the ingestion run, and watches it to completion (streaming +logs + the final summary). ` + "`data validate`" + ` checks an ingest.yaml +locally first. ` + "`tracebloc cluster info`" + ` is the pre-flight you'd typically run before the first ingest.`, @@ -48,6 +50,7 @@ before the first ingest.`, cmd.AddCommand(newDataIngestCmd()) cmd.AddCommand(newDataListCmd()) cmd.AddCommand(newDataDeleteCmd()) + cmd.AddCommand(newIngestValidateCmd()) return cmd } @@ -96,6 +99,7 @@ func newDataIngestCmd() *cobra.Command { // Operations flags. dryRun bool + overwrite bool noInput bool outputJSON bool @@ -126,8 +130,8 @@ func newDataIngestCmd() *cobra.Command { cmd := &cobra.Command{ Use: "ingest ", Aliases: []string{"push"}, - Short: "Stage a local dataset to the cluster's shared PVC", - Long: `Stages a local dataset to the parent client release's shared PVC, + Short: "Stage a local dataset into your client's storage", + Long: `Stages a local dataset into your client's shared storage, submits an ingestion run to jobs-manager, and watches the ingestor Job to completion. Supports 9 task categories (image classification, object/keypoint detection, text classification, masked language @@ -142,7 +146,9 @@ Expected local layout (image_classification shown): 002.jpg ... -Accepted image extensions: .jpg, .jpeg, .png, .webp (case-insensitive). +Accepted image extensions: .jpg, .jpeg, or .png (case-insensitive). +All images in one dataset must share a single type — the cluster +validates the type it was told to expect. v0.1 caps the dataset at 1 GiB total + 500 MiB per file. Larger datasets need the v0.2 cloud-source story (S3/GCS/HTTPS sources) — @@ -153,11 +159,14 @@ Exit codes: 2 schema validation failed (synthesized spec rejected) or v0.1-unsupported category passed 3 local-layout or kubeconfig error - 4 cluster reachable but parent release / shared PVC missing + 4 cluster reachable but no tracebloc client / shared storage missing 5 ingestor SA token couldn't be obtained, or jobs-manager rejected the token (401/403) + 6 destination table already exists (re-run with --overwrite to + replace it, or pick a different --table) 7 pre-flight succeeded but staging the files failed - (Pod creation, image pull, exec stream, or remote tar error) + (Pod creation, image pull, exec stream, or remote tar error) — + or, with --overwrite, removing the old table failed 8 jobs-manager rejected the submit (4xx/5xx other than auth) 9 ingestion Job exited non-zero, or completed with row-level failures the summary panel reports`, @@ -200,6 +209,7 @@ Exit codes: TargetSizeFlag: targetSize, SchemaFlag: schemaFlag, DryRun: dryRun, + Overwrite: overwrite, IngestorSAName: ingestorSAName, StagePodImage: stagePodImage, Detach: detach, @@ -220,7 +230,7 @@ Exit codes: cmd.Flags().StringVar(&contextOverride, "context", "", "name of the kubeconfig context to use (default: kubeconfig's current-context)") cmd.Flags().StringVarP(&nsOverride, "namespace", "n", "", - "namespace where the parent tracebloc/client release is installed") + "namespace where your tracebloc client is installed") // Required spec flags. We DON'T mark them required-at-cobra-level // because cobra's "required flag" error message is terse and @@ -249,6 +259,8 @@ Exit codes: cmd.Flags().IntVar(&numberOfKeypoints, "number-of-keypoints", 0, "keypoint_detection only: number of keypoints per sample (required; e.g. 17 for COCO pose)") + cmd.Flags().BoolVar(&overwrite, "overwrite", false, + "replace the destination table if it already exists: its current table + files are removed first (same as `tracebloc data delete`), then the new data is ingested. Not combinable with --idempotency-key") cmd.Flags().BoolVar(&dryRun, "dry-run", false, "validate + discover + walk, but don't create any cluster resources") cmd.Flags().BoolVar(&noInput, "no-input", false, @@ -257,7 +269,7 @@ Exit codes: "emit a machine-readable JSON result on stdout (human output → stderr; implies --no-input)") cmd.Flags().StringVar(&ingestorSAName, "ingestor-sa", "", "override the ingestor ServiceAccount name (default: \"ingestor\"); "+ - "set this if you customized ingestionAuthz.serviceAccountName in the parent client chart") + "set this if you customized ingestionAuthz.serviceAccountName in your client's install") cmd.Flags().StringVar(&stagePodImage, "stage-pod-image", "", "override the ephemeral stage Pod's image (default: digest-pinned alpine 3.20 baked into the CLI). "+ "Pin by digest in your override too — tag-only refs drift silently.") @@ -289,6 +301,7 @@ type runDataIngestArgs struct { TargetSizeFlag string // raw --target-size; resolved after Discover (image) SchemaFlag string // raw --schema; resolved or inferred after Discover (tabular) DryRun bool + Overwrite bool IngestorSAName string StagePodImage string @@ -370,13 +383,23 @@ func runDataIngest(ctx context.Context, out, errOut io.Writer, a runDataIngestAr // does, so a first-time user understands it before any prompts. // Routed through a.Printer, so --output-json keeps it on stderr and // --plain/non-TTY degrade cleanly. (#31) + // --overwrite + a reused --idempotency-key is a data-loss trap: the + // teardown removes the existing data, then jobs-manager treats the + // duplicate key as a REPLAY and attaches to the previous run instead of + // ingesting anything — old data gone, new data never loaded, exit 0 from + // the old Job's status. Refuse the combination outright. + if a.Overwrite && a.IdempotencyKey != "" { + return &exitError{code: 2, err: errors.New( + "--overwrite can't be combined with --idempotency-key: a reused key makes the cluster replay the previous run instead of ingesting the new data — after --overwrite's removal that would report success while loading nothing. Drop one of the two (a fresh per-run key is the default).")} + } + a.Printer.Banner("tracebloc", "data ingest") a.Printer.Para(strings.TrimSpace(` This uploads a dataset from your machine into your tracebloc workspace so models can be trained on it. Your files are sent to the Kubernetes cluster your workspace was installed on — tracebloc checks them and loads them into a table your training runs read from. Your data stays on that cluster the whole time; -contributors train against it without ever seeing the raw files.`)) +other collaborators train against it without ever seeing the raw files.`)) a.Printer.Hintf("Learn more: https://docs.tracebloc.io") // 0. Guided mode: prompt for any missing core inputs before @@ -479,6 +502,17 @@ contributors train against it without ever seeing the raw files.`)) // synthesized spec carries the right fields before validation. switch { case push.IsTabular(a.Spec.Category): + // P3 (cli#71): a BOM'd tabular CSV is doomed in-cluster AND would + // corrupt InferSchema's own header read below — reject before + // either. The rest of the content preflight runs after the spec + // schema validation (mirroring the in-cluster order). + if perr := push.CheckTabularBOM(layout.LabelsCSV); perr != nil { + return &exitError{code: 3, err: perr} + } + if perr := push.CheckHasDataRows(layout.LabelsCSV); perr != nil { + return &exitError{code: 3, err: perr} + } + // Column schema: an explicit --schema wins; otherwise infer // INT/FLOAT/VARCHAR types from the CSV so the customer doesn't // hand-write one for the common case. @@ -540,6 +574,15 @@ contributors train against it without ever seeing the raw files.`)) "resolution mismatch.\n", derr) } } + // Extension: every image must share one type, and the spec tells + // the cluster which one to validate against (file_options.extension). + // Without this the ingestor checked its .jpeg convention default and + // rejected .jpg/.png datasets AFTER the full upload (cli#68). + ext, exterr := push.DetectExtension(layout.Images) + if exterr != nil { + return &exitError{code: 3, err: exterr} + } + a.Spec.Extension = ext default: // Text family: no extra per-category resolution. The label (for // text_classification) comes straight from --label-column; @@ -583,6 +626,15 @@ contributors train against it without ever seeing the raw files.`)) return &exitError{code: 2, err: errors.New("synthesized spec failed schema validation; check the flag values above")} } + // P3 content preflight (backend#828, cli#69/#71/#72/#73): preview the + // ingestor's own validators locally — AFTER the spec schema validation, + // mirroring the in-cluster order (jsonschema first, then validators), + // and BEFORE any cluster work. Each check names the rule it previews; + // parity is pinned by internal/push/parity_golden_test.go. + if perr := runLocalPreflight(a, layout, errOut); perr != nil { + return perr + } + printLocalSummary(a.Printer, layout, spec) // 5. Cluster discovery — same kubeconfig path as `cluster info`. @@ -591,38 +643,45 @@ contributors train against it without ever seeing the raw files.`)) // consistent across pre-flight commands. a.Printer.Step(2, 4, "Connect to your workspace's cluster") a.Printer.Hintf("Using your kubeconfig to find the tracebloc release in your workspace and the shared storage your dataset will live on.") - resolved, err := cluster.Load(cluster.KubeconfigOptions{ - Path: a.Kubeconfig, - Context: a.Context, - Namespace: a.Namespace, - }) - if err != nil { - return &exitError{code: 3, err: fmt.Errorf("loading kubeconfig: %w", err)} - } - cs, err := cluster.NewClientset(resolved) + // 6. PVC discovery (needPVC) confirms the chart's shared-data PVC is + // Bound before we waste time provisioning a Pod that can't mount it. + opts := cluster.KubeconfigOptions{Path: a.Kubeconfig, Context: a.Context, Namespace: a.Namespace} + binding := bindActiveClientNamespace(&opts) + target, err := resolveClusterTarget(ctx, a.Printer, opts, binding, true) if err != nil { - return &exitError{code: 3, err: err} - } - release, err := cluster.DiscoverParentRelease(ctx, cs, resolved.Namespace) - if err != nil { - return &exitError{code: 4, err: err} + return binding.explain(err) } + resolved, cs, release, pvc := target.Resolved, target.Clientset, target.Release, target.PVC if a.IngestorSAName != "" { release.IngestorSAName = a.IngestorSAName } - // 6. PVC discovery — confirms the chart's shared-data PVC is - // Bound before we waste time provisioning a Pod that can't - // mount it. - pvc, err := cluster.DiscoverSharedPVC(ctx, cs, resolved.Namespace) - if err != nil { - return &exitError{code: 4, err: err} - } - // 7. Show what we found on the cluster — the customer's last look // before any bytes move. printClusterSummary(a.Printer, release, pvc) + // 8a. Destination guard (cli#70, P4-lite): re-ingesting an existing + // table used to stage EVERYTHING and then fail the in-cluster Job + // on the ingestor's duplicate check — a full upload burned to learn + // the table exists. One cheap read heads that off. The check fails + // open (dim note) — the ingestor still refuses duplicates, so a + // broken check can't cause silent data loss. + existingTable, checkNote := destTableExists(ctx, cs, resolved, a.Spec.Table) + if checkNote != "" { + a.Printer.Hintf("%s", checkNote) + } + tableExists := existingTable != "" + if tableExists && !a.Overwrite { + return &exitError{code: 6, err: fmt.Errorf( + "table %q already exists in this client. Re-ingesting the same table doesn't merge or replace — "+ + "the run would fail after uploading everything. Re-run with --overwrite to replace it, "+ + "or pick a different --table. (`tracebloc data delete %s` also removes it.)", + existingTable, existingTable)} + } + if tableExists && a.Overwrite { + a.Printer.Warnf("Table %q already exists — --overwrite replaces it (table + files).", existingTable) + } + // 8. Dry-run stop. Acknowledged success, plus a reminder of the // live-only steps (stage + ingest) the customer just skipped. if a.DryRun { @@ -636,6 +695,36 @@ contributors train against it without ever seeing the raw files.`)) return nil } + // 8b. --overwrite: remove the existing table + files before staging — + // the same teardown `data delete` runs, so the semantics match. + if tableExists && a.Overwrite { + // Tear down the MATCHED name, not the flag's spelling — table names + // are case-sensitive on Linux MySQL and PVC paths always are, so + // acting on a differently-cased --table would silently no-op the + // DROP/rm and then "succeed". + a.Printer.Infof("Removing the existing %q first…", existingTable) + plan := push.PlanTeardown(existingTable) + if _, terr := push.Teardown(ctx, cs, &push.SPDYExecutor{Config: resolved.RestConfig, Client: cs}, resolved.Namespace, plan, push.PodSpecOptions{ + Namespace: resolved.Namespace, + PVCClaimName: pvc.ClaimName, + PVCMountPath: pvc.MountPath, + Table: existingTable, + ServiceAccountName: release.IngestorSAName, + Image: a.StagePodImage, + }); terr != nil { + // The teardown drops the table before removing files, so a + // partial failure can leave files the DB-backed guard can no + // longer see — a plain re-run would upload everything and then + // hit them in-cluster. data delete first is the real recovery. + return &exitError{code: 7, err: fmt.Errorf( + "replacing table %q failed partway — its removal may be incomplete, and a plain re-run "+ + "would hit the leftovers after uploading everything. Run `tracebloc data delete %s` "+ + "first, then re-run this ingest. Nothing new was staged. (%w)", + existingTable, existingTable, terr)} + } + a.Printer.Successf("Removed the old %q — ingesting the new data.", existingTable) + } + // 9. Stage the files: create ephemeral Pod → wait Ready → tar // stream → cleanup. The deferred cleanup inside push.Stage // runs on success and failure (including ctx cancellation @@ -827,7 +916,15 @@ func printLocalSummary(p *ui.Printer, layout *push.LocalLayout, spec map[string] } default: p.Field("labels.csv", layout.LabelsCSV) - p.Field("images", fmt.Sprintf("%d files", len(layout.Images))) + imagesVal := fmt.Sprintf("%d files", len(layout.Images)) + if ext, _ := spec["spec"].(map[string]any); ext != nil { + if fo, _ := ext["file_options"].(map[string]any); fo != nil { + if e, _ := fo["extension"].(string); e != "" { + imagesVal = fmt.Sprintf("%d files (%s)", len(layout.Images), e) + } + } + } + p.Field("images", imagesVal) if anns := layout.Sidecars["annotations"]; len(anns) > 0 { p.Field("annotations", fmt.Sprintf("%d files", len(anns))) } @@ -940,3 +1037,47 @@ func writePushErrorJSON(w io.Writer, sp push.SpecArgs, e error, code int) { } _, _ = fmt.Fprintln(w, string(b)) } + +// listDatasetsFn is a test seam over push.ListDatasets. +var listDatasetsFn = push.ListDatasets + +// destTableExists reports whether the destination table already holds an +// ingested dataset, via the same query `data list` uses. It fails OPEN: a +// broken check returns (false, note) so the ingest proceeds — the in-cluster +// duplicate check still backstops it — but the note tells the user the guard +// didn't run rather than silently skipping it. +// The first return is the EXISTING table's exact name ("" when absent): +// matching is case-insensitive (mysql's catalog may be), but any teardown +// must act on the real spelling — DROP/rm against the flag's casing would +// silently no-op on case-sensitive systems and then claim success. +func destTableExists(ctx context.Context, cs kubernetes.Interface, resolved *cluster.ResolvedConfig, table string) (string, string) { + names, err := listDatasetsFn(ctx, cs, resolved.RestConfig, resolved.Namespace) + if err != nil { + return "", fmt.Sprintf("(couldn't check whether %q already exists — continuing; the cluster still refuses duplicates: %v)", table, err) + } + for _, n := range names { + if strings.EqualFold(n, table) { + return n, "" + } + } + return "", "" +} + +// runLocalPreflight maps push.PreflightDataset — THE shared preview +// dispatch, also exercised verbatim by the parity harness — onto the CLI's +// conventions: notes print dim to errOut, a BadFlag problem exits 2 (fix a +// flag), anything else exits 3 (fix the data). +func runLocalPreflight(a runDataIngestArgs, layout *push.LocalLayout, errOut io.Writer) error { + notes, problem := push.PreflightDataset(a.Spec, layout) + for _, n := range notes { + _, _ = fmt.Fprintln(errOut, n) + } + if problem == nil { + return nil + } + code := 3 + if problem.BadFlag { + code = 2 + } + return &exitError{code: code, err: problem.Err} +} diff --git a/internal/cli/data_delete.go b/internal/cli/data_delete.go index 974630a..e06831a 100644 --- a/internal/cli/data_delete.go +++ b/internal/cli/data_delete.go @@ -57,7 +57,7 @@ Exit codes: 0 artifacts removed (or --dry-run, or the user declined) 2 invalid table name 3 kubeconfig error, or refused (no confirmation off a terminal) - 4 cluster reachable but parent release / shared PVC missing + 4 cluster reachable but no tracebloc client / shared storage missing 7 teardown failed mid-flight (table drop or PVC rm errored)`, Args: cobra.ExactArgs(1), RunE: func(cmd *cobra.Command, args []string) error { @@ -84,7 +84,7 @@ Exit codes: cmd.Flags().StringVar(&contextOverride, "context", "", "name of the kubeconfig context to use (default: kubeconfig's current-context)") cmd.Flags().StringVarP(&nsOverride, "namespace", "n", "", - "namespace where the parent tracebloc/client release is installed") + "namespace where your tracebloc client is installed") cmd.Flags().BoolVar(&dryRun, "dry-run", false, "show what would be deleted without deleting anything") cmd.Flags().BoolVarP(&yes, "yes", "y", false, @@ -110,31 +110,17 @@ undone — re-ingesting the data is the only way back.`) return &exitError{code: 2, err: fmt.Errorf("invalid table name %q: %w", a.Table, err)} } - // 2. Resolve cluster + clientset (kubeconfig errors = exit 3). - resolved, err := cluster.Load(cluster.KubeconfigOptions{ - Path: a.Kubeconfig, - Context: a.Context, - Namespace: a.Namespace, - }) - if err != nil { - return &exitError{code: 3, err: fmt.Errorf("loading kubeconfig: %w", err)} - } - cs, err := cluster.NewClientset(resolved) - if err != nil { - return &exitError{code: 3, err: err} - } - - // 3. Confirm the parent release + shared PVC exist (exit 4 if not) — + // 2. Resolve cluster + clientset (kubeconfig errors = exit 3), then + // confirm the parent release + shared PVC exist (exit 4 if not) — // both "is this the right cluster?" context and a guard against // running teardown against a cluster with no tracebloc install. - release, err := cluster.DiscoverParentRelease(ctx, cs, resolved.Namespace) - if err != nil { - return &exitError{code: 4, err: err} - } - pvc, err := cluster.DiscoverSharedPVC(ctx, cs, resolved.Namespace) + opts := cluster.KubeconfigOptions{Path: a.Kubeconfig, Context: a.Context, Namespace: a.Namespace} + binding := bindActiveClientNamespace(&opts) + target, err := resolveClusterTarget(ctx, a.Printer, opts, binding, true) if err != nil { - return &exitError{code: 4, err: err} + return binding.explain(err) } + resolved, cs, release, pvc := target.Resolved, target.Clientset, target.Release, target.PVC // 4. Show exactly what will be deleted — the customer's last look // before destructive, unrecoverable work. diff --git a/internal/cli/data_list.go b/internal/cli/data_list.go index 0d6029a..f76f342 100644 --- a/internal/cli/data_list.go +++ b/internal/cli/data_list.go @@ -42,7 +42,7 @@ func newDataListCmd() *cobra.Command { cmd := &cobra.Command{ Use: "list", Short: "List datasets ingested in the cluster", - Long: `Lists the datasets ingested into the parent client release — + Long: `Lists the datasets ingested into your client — the tables in ` + push.IngestionDatabase + ` on the cluster. With no flags it uses your current kubeconfig context and its namespace; @@ -53,7 +53,7 @@ https://ai.tracebloc.io/metadata. Exit codes: 0 listed successfully (including an empty list) 3 kubeconfig error - 4 cluster reachable but no parent release in the namespace + 4 cluster reachable but no tracebloc client in the namespace 7 couldn't query the cluster for datasets`, Args: cobra.NoArgs, RunE: func(cmd *cobra.Command, _ []string) error { @@ -81,7 +81,7 @@ Exit codes: cmd.Flags().StringVar(&contextOverride, "context", "", "name of the kubeconfig context to use (default: kubeconfig's current-context)") cmd.Flags().StringVarP(&nsOverride, "namespace", "n", "", - "namespace where the parent tracebloc/client release is installed") + "namespace where your tracebloc client is installed") cmd.Flags().BoolVar(&outputJSON, "output-json", false, "emit the dataset list as JSON on stdout (human output → stderr)") @@ -111,22 +111,13 @@ func runDataList(ctx context.Context, a runDataListArgs) (err error) { p := a.Printer p.Banner("tracebloc", "datasets in the cluster") - resolved, err := cluster.Load(cluster.KubeconfigOptions{ - Path: a.Kubeconfig, - Context: a.Context, - Namespace: a.Namespace, - }) + opts := cluster.KubeconfigOptions{Path: a.Kubeconfig, Context: a.Context, Namespace: a.Namespace} + binding := bindActiveClientNamespace(&opts) + target, err := resolveClusterTarget(ctx, p, opts, binding, false) if err != nil { - return &exitError{code: 3, err: fmt.Errorf("loading kubeconfig: %w", err)} - } - cs, err := cluster.NewClientset(resolved) - if err != nil { - return &exitError{code: 3, err: err} - } - release, err := cluster.DiscoverParentRelease(ctx, cs, resolved.Namespace) - if err != nil { - return &exitError{code: 4, err: err} + return binding.explain(err) } + resolved, cs, release := target.Resolved, target.Clientset, target.Release tables, err := push.ListDatasets(ctx, cs, resolved.RestConfig, resolved.Namespace) if err != nil { diff --git a/internal/cli/data_test.go b/internal/cli/data_test.go index 299849c..703191e 100644 --- a/internal/cli/data_test.go +++ b/internal/cli/data_test.go @@ -1,7 +1,17 @@ package cli import ( + "github.com/tracebloc/cli/internal/push" + "github.com/tracebloc/cli/internal/ui" + "image" + "image/png" + "k8s.io/client-go/kubernetes" + "k8s.io/client-go/rest" + "bytes" + "context" + "errors" + "github.com/tracebloc/cli/internal/cluster" "os" "path/filepath" "strings" @@ -16,16 +26,25 @@ func imgcLayout(t *testing.T) string { t.Helper() root := t.TempDir() if err := os.WriteFile(filepath.Join(root, "labels.csv"), - []byte("image_id,label\n001.jpg,cat\n"), 0o644); err != nil { + []byte("image_id,label\n001.jpg,cat\n002.jpg,dog\n"), 0o644); err != nil { t.Fatalf("write labels.csv: %v", err) } imagesDir := filepath.Join(root, "images") if err := os.MkdirAll(imagesDir, 0o755); err != nil { t.Fatalf("mkdir images: %v", err) } - if err := os.WriteFile(filepath.Join(imagesDir, "001.jpg"), - make([]byte, 100), 0o644); err != nil { - t.Fatalf("write image: %v", err) + // Real decodable images with two classes: the P3 preflight decodes + // EVERY image and requires >=2 label classes, so opaque stubs would + // short-circuit any test that needs to get past preflight (a + // kubeconfig test failed vacuously on exactly that). + var buf bytes.Buffer + if err := png.Encode(&buf, image.NewRGBA(image.Rect(0, 0, 4, 4))); err != nil { + t.Fatal(err) + } + for _, name := range []string{"001.jpg", "002.jpg"} { + if err := os.WriteFile(filepath.Join(imagesDir, name), buf.Bytes(), 0o644); err != nil { + t.Fatalf("write %s: %v", name, err) + } } return root } @@ -314,6 +333,18 @@ func TestAliasResolution(t *testing.T) { args: []string{"data", "delete", "--help"}, want: "Removes the in-cluster artifacts", }, + { + // `ingest validate` moved under data (top-level `ingest` is a + // hidden deprecated alias) — both paths must keep resolving. + name: "data validate canonical", + args: []string{"data", "validate", "--help"}, + want: "validates it against the bundled", + }, + { + name: "ingest validate alias resolves", + args: []string{"ingest", "validate", "--help"}, + want: "validates it against the bundled", + }, } for _, c := range cases { t.Run(c.name, func(t *testing.T) { @@ -332,3 +363,54 @@ func TestAliasResolution(t *testing.T) { }) } } + +// destTableExists backs the cli#70 P4-lite guard: an existing destination +// table must be caught BEFORE staging (a re-ingest used to burn the full +// upload and then fail in-cluster), and a broken check must fail OPEN with +// a visible note — never block the ingest, never pretend it ran. +func TestDestTableExists(t *testing.T) { + resolved := &cluster.ResolvedConfig{Namespace: "ns"} + + restore := listDatasetsFn + defer func() { listDatasetsFn = restore }() + + listDatasetsFn = func(_ context.Context, _ kubernetes.Interface, _ *rest.Config, _ string) ([]string, error) { + return []string{"other", "MyTable"}, nil + } + matched, note := destTableExists(context.Background(), nil, resolved, "mytable") + if matched != "MyTable" || note != "" { + t.Errorf("case-insensitive match must return the EXISTING spelling (teardown acts on it): matched=%q note=%q, want MyTable/empty", matched, note) + } + + matched, note = destTableExists(context.Background(), nil, resolved, "fresh_table") + if matched != "" || note != "" { + t.Errorf("absent table: matched=%q note=%q, want empty/empty", matched, note) + } + + listDatasetsFn = func(_ context.Context, _ kubernetes.Interface, _ *rest.Config, _ string) ([]string, error) { + return nil, errors.New("mysql pod not found") + } + matched, note = destTableExists(context.Background(), nil, resolved, "t") + if matched != "" { + t.Error("a broken check must fail open (no match), not closed") + } + if !strings.Contains(note, "couldn't check") || !strings.Contains(note, "mysql pod not found") { + t.Errorf("fail-open note = %q, want it to say the check didn't run and why", note) + } +} + +// The images summary line surfaces the detected extension — the visible +// half of the cli#68 fix (the spec half is pinned in internal/push). +func TestPrintLocalSummary_ShowsDetectedExtension(t *testing.T) { + var buf bytes.Buffer + p := ui.New(&buf, ui.WithColor(false)) + layout := &push.LocalLayout{Root: "/d", LabelsCSV: "/d/labels.csv", Images: []string{"/d/images/a.png"}} + spec := map[string]any{ + "table": "t", "category": "image_classification", "intent": "train", + "spec": map[string]any{"file_options": map[string]any{"extension": ".png"}}, + } + printLocalSummary(p, layout, spec) + if !strings.Contains(buf.String(), "1 files (.png)") { + t.Errorf("summary missing detected extension:\n%s", buf.String()) + } +} diff --git a/internal/cli/doctor.go b/internal/cli/doctor.go index 9fae112..1bbce5a 100644 --- a/internal/cli/doctor.go +++ b/internal/cli/doctor.go @@ -66,7 +66,7 @@ Exit codes: cmd.Flags().StringVar(&contextOverride, "context", "", "name of the kubeconfig context to use (default: kubeconfig's current-context)") cmd.Flags().StringVarP(&nsOverride, "namespace", "n", "", - "namespace where the parent tracebloc/client release is installed (default: the context's namespace, or 'default')") + "namespace where your tracebloc client is installed (default: the context's namespace, or 'default')") return cmd } diff --git a/internal/cli/ingest.go b/internal/cli/ingest.go index b49d692..174ec25 100644 --- a/internal/cli/ingest.go +++ b/internal/cli/ingest.go @@ -3,28 +3,35 @@ package cli import ( "fmt" "os" + "strings" "github.com/spf13/cobra" "github.com/tracebloc/cli/internal/schema" ) -// newIngestCmd implements the `tracebloc ingest` subtree. Today it -// has only one verb — `validate` — which runs the schema check -// locally without touching the cluster. Future verbs (status, retry, -// cancel) hang off this same parent in later phases. +// newIngestCmd implements the `tracebloc ingest` subtree — kept as a HIDDEN +// alias for one deprecation cycle: its only verb, `validate`, moved to +// `tracebloc data validate`, so the top level no longer carries two different +// things named "ingest" (`ingest validate` vs `data ingest`). Scripts using +// the old path keep working; help and docs point at the new one. func newIngestCmd() *cobra.Command { cmd := &cobra.Command{ - Use: "ingest", - Short: "Inspect and manage ingestion configurations", - Long: `Commands for working with ingest.yaml files and ingestion runs. - -Today only ` + "`validate`" + ` is implemented — it runs the same schema check -the cluster's jobs-manager runs, but locally and instantly. Use it to -catch typos and missing fields before submitting an ingestion. - -Future verbs (status, retry, cancel) will land alongside the -push/list/show commands in later phases.`, + Use: "ingest", + Short: "Deprecated alias for `tracebloc data validate`", + Hidden: true, + // A bare path here is almost always someone meaning `data ingest` + // (the home screen advertises it) — redirect instead of dumping a + // confusing deprecation usage block. + RunE: func(cmd *cobra.Command, args []string) error { + if len(args) > 0 { + return fmt.Errorf( + "`tracebloc ingest` doesn't stage datasets — did you mean:\n"+ + " tracebloc data ingest %s", + strings.Join(args, " ")) + } + return cmd.Help() + }, } cmd.AddCommand(newIngestValidateCmd()) @@ -49,9 +56,7 @@ ingest.v1.json schema (synced from tracebloc/data-ingestors). Prints violations in the same JSON-pointer-prefixed format the cluster's jobs-manager uses, and exits non-zero if any are found. -Useful as a pre-flight before ` + "`tracebloc dataset push`" + ` lands in a -future phase; for now, customers running the Helm chart can validate -their ` + "`ingest.yaml`" + ` before invoking ` + "`helm install`" + `, getting +Useful as a pre-flight before running ` + "`tracebloc data ingest`" + ` — millisecond local feedback instead of a multi-second cluster round trip. diff --git a/internal/cli/root.go b/internal/cli/root.go index 2b7868d..56ce963 100644 --- a/internal/cli/root.go +++ b/internal/cli/root.go @@ -41,32 +41,26 @@ func NewRootCmd(info BuildInfo) *cobra.Command { root := &cobra.Command{ Use: "tracebloc", - Short: "tracebloc — interactive data ingestion for your cluster", - Long: `tracebloc is the customer-facing CLI for the tracebloc declarative -ingestion path. It wraps the same POST /internal/submit-ingestion-run -protocol the tracebloc/ingestor Helm chart uses, so any cluster running -the parent tracebloc/client chart can be targeted directly from a -developer's workstation. - -The dominant workflow: - - tracebloc data ingest ./my-data \ - --table cats_dogs_train \ - --category image_classification \ - --intent train \ - --label-column label - -The CLI handles cluster discovery (via kubeconfig), staging the data -on the cluster's shared PVC, submitting the ingestion request, -watching the resulting Job, and reporting the outcome. Customers never -touch Helm, never edit YAML, never run kubectl cp manually. - -This binary implements the full v0.1 ingestion path: ` + "`data ingest`" + ` -(the dominant workflow above), ` + "`ingest validate`" + ` for a local -schema check, ` + "`cluster info`" + ` for discovery diagnostics, plus -` + "`version`" + ` and ` + "`completion`" + `. See -https://github.com/tracebloc/client/issues/147 for the v0.1 roadmap and -what's planned next.`, + Short: "tracebloc — connect this machine to tracebloc and manage its data", + Long: `The tracebloc CLI connects machines to tracebloc as clients and +manages the datasets that models train on. Your data stays on your +infrastructure — models from other collaborators come to it, once you +approve them. + +Two kinds of commands: + + Your account (sign in first): login, logout, auth, client + This machine's client: data, cluster + +A typical first session: + + tracebloc login # sign in or create your account (browser) + tracebloc data ingest ./my-data # stage a dataset into your client + tracebloc data list # see what's in the cluster + +The CLI finds your cluster through your kubeconfig, stages data onto +the cluster's shared storage, and reports progress as it goes. No +Helm, no YAML, no kubectl needed.`, // Silence cobra's auto-printed errors + usage on every error; // we already print structured errors in handlers, and the @@ -107,13 +101,14 @@ what's planned next.`, return cmd.Help() // an arg that wasn't a known subcommand } p := printerFor(cmd) - p.Banner("tracebloc", "interactive data ingestion for your cluster") + p.Banner("tracebloc", "connect this machine to tracebloc and manage its data") p.Section("Get started") - p.Infof("tracebloc data ingest — stage + ingest a dataset interactively (or use --help to see flags)") - p.Infof("tracebloc data list — list datasets ingested in the cluster") - p.Infof("tracebloc data delete — delete an ingested dataset (its table + files)") - p.Infof("tracebloc cluster info — check the CLI can reach your cluster") - p.Infof("tracebloc ingest validate f.yaml — validate an ingest.yaml locally") + p.Infof("tracebloc login — sign in to tracebloc (browser)") + p.Infof("tracebloc data ingest ./data — stage a dataset into your client") + p.Infof("tracebloc data list — datasets in the cluster") + p.Infof("tracebloc data delete
— delete an ingested dataset") + p.Infof("tracebloc client list — your clients and their status") + p.Infof("tracebloc cluster doctor — diagnose connection issues") p.Newline() p.Hintf("Add --help to any command for the full flag list.") return nil diff --git a/internal/cli/root_test.go b/internal/cli/root_test.go index 218be56..64711ba 100644 --- a/internal/cli/root_test.go +++ b/internal/cli/root_test.go @@ -41,7 +41,7 @@ func TestRootCmd_HomeScreen(t *testing.T) { if err := root.Execute(); err != nil { t.Fatalf("bare root failed: %v\n%s", err, out.String()) } - for _, want := range []string{"tracebloc", "data ingest", "data list", "data delete", "cluster info"} { + for _, want := range []string{"tracebloc", "login", "data ingest", "data list", "data delete", "cluster doctor"} { if !strings.Contains(out.String(), want) { t.Errorf("home screen missing %q:\n%s", want, out.String()) } diff --git a/internal/cli/verbose_install_test.go b/internal/cli/verbose_install_test.go index 2c1f241..78ec84f 100644 --- a/internal/cli/verbose_install_test.go +++ b/internal/cli/verbose_install_test.go @@ -137,9 +137,17 @@ func TestClientCreate_CancelLogsCancelledNotDone(t *testing.T) { return "", errors.New("no cluster (test)") } t.Cleanup(func() { readClusterID = origCID }) + // The R7 adopt-backfill now runs even when the UID read fails (#158), so it + // calls readInClusterClient regardless — stub it to "no live client" so this + // test doesn't touch the developer's real kubeconfig. + origLive := readInClusterClient + readInClusterClient = func(context.Context, cluster.KubeconfigOptions) (*cluster.InClusterClient, error) { + return nil, nil + } + t.Cleanup(func() { readInClusterClient = origLive }) confirmNo := false - pr := &fakePrompter{answers: map[string]string{}, confirm: &confirmNo} + pr := &fakePrompter{confirm: &confirmNo} var out bytes.Buffer if err := runClientCreate(context.Background(), ui.New(&out), pr, clientCreateOpts{name: "Lab", location: "DE"}); err != nil { @@ -161,17 +169,17 @@ func TestClientCreate_CancelLogsCancelledNotDone(t *testing.T) { } } -// TestClientCreate_ResumeCommandIncludesPromptedValues pins the Bugbot fix: when -// name/location come from interactive prompts (not flags), a failed provision's -// resume command must still include them — opts alone would omit them. -func TestClientCreate_ResumeCommandIncludesPromptedValues(t *testing.T) { +// TestClientCreate_ResumeCommandPinsAutoName pins that a failed provision's resume +// command carries the AUTO-DERIVED name (cli#137), not a bare re-invocation — so a +// retry reproduces the same client name instead of re-deriving/renumbering it. +func TestClientCreate_ResumeCommandPinsAutoName(t *testing.T) { t.Setenv("TRACEBLOC_CONFIG_DIR", t.TempDir()) if err := (&config.Config{CurrentEnv: "dev", Profiles: map[string]*config.Profile{ - "dev": {Token: "tok"}, + "dev": {Token: "tok", FirstName: "Lukas"}, }}).Save(); err != nil { t.Fatal(err) } - // list ok; the provision POST 500s after the user confirms. + // list ok; the provision POST 500s. srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { if r.Method == http.MethodGet { _, _ = w.Write([]byte(`[]`)) @@ -189,18 +197,13 @@ func TestClientCreate_ResumeCommandIncludesPromptedValues(t *testing.T) { } t.Cleanup(func() { readClusterID = origCID }) - confirmYes := true - pr := &fakePrompter{answers: map[string]string{ - "Client name": "Prompted Lab", - "Location zone (e.g. DE)": "FR", - }, confirm: &confirmYes} var out bytes.Buffer - // No name/location flags — both come from the prompts. - if err := runClientCreate(context.Background(), ui.New(&out), pr, clientCreateOpts{}); err == nil { + // No flags at all — the name is auto-derived to lukas-01. + if err := runClientCreate(context.Background(), ui.New(&out), nil, clientCreateOpts{yes: true}); err == nil { t.Fatal("expected the provision to fail (POST 500)") } - if !strings.Contains(out.String(), "--name 'Prompted Lab' --location FR") { - t.Errorf("resume command should carry the PROMPTED name + location, got:\n%s", out.String()) + if !strings.Contains(out.String(), "--name lukas-01") { + t.Errorf("resume command should pin the auto-derived name, got:\n%s", out.String()) } } diff --git a/internal/cluster/discover.go b/internal/cluster/discover.go index b1718f9..3f6ad69 100644 --- a/internal/cluster/discover.go +++ b/internal/cluster/discover.go @@ -2,7 +2,9 @@ package cluster import ( "context" + "errors" "fmt" + "sort" "strings" appsv1 "k8s.io/api/apps/v1" @@ -10,6 +12,12 @@ import ( "k8s.io/client-go/kubernetes" ) +// ErrNoParentRelease is the sentinel for DiscoverParentRelease's "the namespace +// has no tracebloc release" case — distinct from an API/RBAC list failure or an +// ambiguous multiple-release match. Callers use errors.Is to tell "this cluster +// doesn't host the release" apart from "couldn't determine the release." +var ErrNoParentRelease = errors.New("no tracebloc client found") + // ParentRelease describes the tracebloc parent client chart release // discovered in the customer's cluster. The information comes from // the helm-managed Deployment's labels (and a few of its env vars), @@ -113,12 +121,16 @@ func DiscoverParentRelease(ctx context.Context, cs kubernetes.Interface, namespa switch len(jmDeps) { case 0: + // Customer-actionable, no Helm: the CLI's own contract is that + // customers never touch Helm, so the remediation is the flag, + // the installer, or the doctor — not a helm invocation. return nil, fmt.Errorf( - "no tracebloc parent client release found in namespace %q "+ - "(no chart-managed Deployment named *-jobs-manager). "+ - "Install with `helm install tracebloc/client --namespace %s` first, "+ - "or pass --namespace to point at the namespace where it's running.", - namespace, namespace, + "%w in namespace %q. "+ + "If your client runs in another namespace, pass --namespace; "+ + "if this cluster has no tracebloc client yet, run the installer: "+ + "bash <(curl -fsSL https://tracebloc.io/i.sh). "+ + "Diagnose with `tracebloc cluster doctor`.", + ErrNoParentRelease, namespace, ) case 1: // happy path @@ -128,9 +140,9 @@ func DiscoverParentRelease(ctx context.Context, cs kubernetes.Interface, namespa names = append(names, d.Name) } return nil, fmt.Errorf( - "found %d tracebloc parent releases in namespace %q (%s); "+ + "found %d tracebloc clients in namespace %q (%s); "+ "this CLI doesn't yet support disambiguating between multiple. "+ - "Pass --namespace to target a namespace with exactly one release.", + "Pass --namespace to target a namespace with exactly one client.", len(jmDeps), namespace, strings.Join(names, ", "), ) } @@ -181,6 +193,90 @@ func DiscoverParentRelease(ctx context.Context, cs kubernetes.Interface, namespa return release, nil } +// FindClientNamespaces scans every namespace the kubeconfig user may list for +// jobs-manager Deployments (the same selector + name filter DiscoverParentRelease +// uses) and returns the sorted, de-duplicated namespaces hosting one. It backs +// the fallback that makes `data list`/`cluster info` work out of the box when +// the client lives in its slug namespace rather than the kubeconfig's default +// (§7.3): a miss in the default namespace triggers this scan instead of a dead +// end. An RBAC-restricted user (cluster-wide list forbidden) gets the error +// back; callers treat that as "scan unavailable" and keep the original message. +func FindClientNamespaces(ctx context.Context, cs kubernetes.Interface) ([]string, error) { + deps, err := cs.AppsV1().Deployments(metav1.NamespaceAll).List(ctx, metav1.ListOptions{ + LabelSelector: "app.kubernetes.io/name=client,app.kubernetes.io/managed-by=Helm", + }) + if err != nil { + return nil, fmt.Errorf("scanning the cluster for tracebloc clients: %w", err) + } + seen := make(map[string]bool) + var namespaces []string + for _, d := range deps.Items { + if d.Name != "jobs-manager" && !strings.HasSuffix(d.Name, "-jobs-manager") { + continue + } + if !seen[d.Namespace] { + seen[d.Namespace] = true + namespaces = append(namespaces, d.Namespace) + } + } + sort.Strings(namespaces) + return namespaces, nil +} + +// InClusterClient identifies a tracebloc client already installed on the cluster: +// its CLIENT_ID (the UUID auth username the pod authenticates with) and the +// namespace its release occupies. +type InClusterClient struct { + ClientID string + Namespace string +} + +// clientChartSelector matches the chart-managed resources of a tracebloc client +// release (the same selector DiscoverParentRelease uses on Deployments). +const clientChartSelector = "app.kubernetes.io/name=client,app.kubernetes.io/managed-by=Helm" + +// DiscoverInClusterClientID finds a tracebloc client already installed on the +// cluster, if any, and returns its live CLIENT_ID + namespace (RFC-0001 §7.2 +// step 1). It locates the namespace hosting the client release (its jobs-manager +// Deployment), then reads CLIENT_ID from the chart's `-secrets` Secret +// there — scoping to that namespace avoids the node-agents mirror secret, which +// carries the same CLIENT_ID under the same labels. +// +// This anchors R7 adopt-backfill: a live client whose backend cluster_id is null +// must be adopted (and its anchor backfilled), never re-minted. Best-effort — it +// returns (nil, nil) when nothing is installed or the cluster can't be read +// (unreachable / restricted RBAC), so callers fall back to a plain create. +func DiscoverInClusterClientID(ctx context.Context, cs kubernetes.Interface) (*InClusterClient, error) { + deps, err := cs.AppsV1().Deployments(metav1.NamespaceAll).List(ctx, metav1.ListOptions{ + LabelSelector: clientChartSelector, + }) + if err != nil { + return nil, nil // best-effort: treat an unreadable cluster as "nothing installed" + } + ns := "" + for _, d := range deps.Items { + if d.Name == "jobs-manager" || strings.HasSuffix(d.Name, "-jobs-manager") { + ns = d.Namespace + break + } + } + if ns == "" { + return nil, nil // no client release on this cluster + } + secrets, err := cs.CoreV1().Secrets(ns).List(ctx, metav1.ListOptions{ + LabelSelector: clientChartSelector, + }) + if err != nil { + return nil, nil + } + for _, s := range secrets.Items { + if v, ok := s.Data["CLIENT_ID"]; ok && len(v) > 0 { + return &InClusterClient{ClientID: string(v), Namespace: ns}, nil + } + } + return nil, nil +} + // pickJobsManagerService probes for the chart's jobs-manager // Service. The chart's helper templates have used both names over // chart history: diff --git a/internal/cluster/discover_test.go b/internal/cluster/discover_test.go index 5c0921c..23e2d8e 100644 --- a/internal/cluster/discover_test.go +++ b/internal/cluster/discover_test.go @@ -2,6 +2,7 @@ package cluster import ( "context" + "errors" "strings" "testing" @@ -90,6 +91,57 @@ func TestDiscoverParentRelease_HappyPath(t *testing.T) { } } +// clientSecret builds the chart's `-secrets` Secret carrying CLIENT_ID +// (the live client's UUID username). extraLabels lets a test mimic the +// node-agents mirror, which shares the labels + CLIENT_ID in another namespace. +func clientSecret(release, namespace, clientID string, extraLabels map[string]string) *corev1.Secret { + labels := map[string]string{ + "app.kubernetes.io/name": "client", + "app.kubernetes.io/instance": release, + "app.kubernetes.io/managed-by": "Helm", + } + for k, v := range extraLabels { + labels[k] = v + } + return &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{Name: release + "-secrets", Namespace: namespace, Labels: labels}, + Data: map[string][]byte{"CLIENT_ID": []byte(clientID)}, + } +} + +func TestDiscoverInClusterClientID_HappyPath(t *testing.T) { + const ns = "tracebloc" + cs := fake.NewClientset( + jobsManagerDeployment("tracebloc", ns, "client-1.3.5", "1.3.5", "sha256:x"), + clientSecret("tracebloc", ns, "uuid-live", nil), + // node-agents mirror: same labels + CLIENT_ID, DIFFERENT namespace. The read + // is scoped to the jobs-manager namespace, so this must be ignored. + clientSecret("tracebloc", "node-agents", "uuid-live", map[string]string{"app": "resource-monitor"}), + ) + got, err := DiscoverInClusterClientID(context.Background(), cs) + if err != nil { + t.Fatal(err) + } + if got == nil || got.ClientID != "uuid-live" || got.Namespace != ns { + t.Errorf("got %+v, want {ClientID:uuid-live Namespace:%s}", got, ns) + } +} + +func TestDiscoverInClusterClientID_NoRelease(t *testing.T) { + got, err := DiscoverInClusterClientID(context.Background(), fake.NewClientset()) + if err != nil || got != nil { + t.Errorf("empty cluster: want (nil,nil), got (%+v,%v)", got, err) + } +} + +func TestDiscoverInClusterClientID_ReleaseButNoSecret(t *testing.T) { + cs := fake.NewClientset(jobsManagerDeployment("tracebloc", "tracebloc", "client-1.3.5", "1.3.5", "d")) + got, err := DiscoverInClusterClientID(context.Background(), cs) + if err != nil || got != nil { + t.Errorf("release but no secret: want (nil,nil), got (%+v,%v)", got, err) + } +} + func TestDiscoverParentRelease_NoReleaseFound(t *testing.T) { cs := fake.NewClientset() // empty cluster @@ -100,11 +152,16 @@ func TestDiscoverParentRelease_NoReleaseFound(t *testing.T) { // The error message has to be customer-actionable. Pin the // key remediation phrase so a future refactor that loses it // (or worse, replaces it with a stack trace) fails this test. - for _, want := range []string{"no tracebloc parent client release found", "helm install"} { + for _, want := range []string{"no tracebloc client found", "--namespace", "https://tracebloc.io/i.sh", "cluster doctor"} { if !strings.Contains(err.Error(), want) { t.Errorf("expected error to mention %q, got: %s", want, err) } } + // The sentinel gates the §7.3 "runs elsewhere" rewrite (cli#128) — a + // genuine not-found must be errors.Is-identifiable. + if !errors.Is(err, ErrNoParentRelease) { + t.Errorf("not-found error should match ErrNoParentRelease, got: %v", err) + } } func TestDiscoverParentRelease_MultipleReleases(t *testing.T) { @@ -125,6 +182,11 @@ func TestDiscoverParentRelease_MultipleReleases(t *testing.T) { t.Errorf("expected error to mention %q, got: %s", want, err) } } + // An ambiguous match is NOT "no release" — it must not trip the §7.3 + // rewrite (the cluster does host tracebloc, just more than one). + if errors.Is(err, ErrNoParentRelease) { + t.Error("multiple-release error must not match ErrNoParentRelease") + } } // Regression for the real-cluster discovery bug. Pre-fix, the @@ -222,3 +284,49 @@ func TestChartVersionFromLabel(t *testing.T) { }) } } + +// FindClientNamespaces backs the cluster-wide fallback scan (§7.3): a miss in +// the kubeconfig's default namespace must find the client in its slug +// namespace instead of dead-ending. These pin the scan's filtering + ordering. +func TestFindClientNamespaces(t *testing.T) { + cs := fake.NewSimpleClientset( + jobsManagerDeployment("tracebloc", "lukas-01", "client-1.6.0", "1.6.0", ""), + jobsManagerDeployment("tracebloc", "zeta-ns", "client-1.6.0", "1.6.0", ""), + // a chart-labeled sibling that is NOT a jobs-manager must not count + siblingDeployment("mysql-client", "other-ns"), + ) + got, err := FindClientNamespaces(context.Background(), cs) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + want := []string{"lukas-01", "zeta-ns"} + if len(got) != len(want) || got[0] != want[0] || got[1] != want[1] { + t.Errorf("expected sorted namespaces %v, got %v", want, got) + } +} + +func TestFindClientNamespaces_NoneFound(t *testing.T) { + cs := fake.NewSimpleClientset(siblingDeployment("mysql-client", "somewhere")) + got, err := FindClientNamespaces(context.Background(), cs) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if len(got) != 0 { + t.Errorf("expected no namespaces, got %v", got) + } +} + +// siblingDeployment builds a chart-labeled Deployment that is not a +// jobs-manager — the scan must ignore it. +func siblingDeployment(name, namespace string) *appsv1.Deployment { + return &appsv1.Deployment{ + ObjectMeta: metav1.ObjectMeta{ + Name: name, + Namespace: namespace, + Labels: map[string]string{ + "app.kubernetes.io/name": "client", + "app.kubernetes.io/managed-by": "Helm", + }, + }, + } +} diff --git a/internal/cluster/identity.go b/internal/cluster/identity.go index f33f14c..b031491 100644 --- a/internal/cluster/identity.go +++ b/internal/cluster/identity.go @@ -35,6 +35,24 @@ func ClusterID(ctx context.Context, opts KubeconfigOptions) (string, error) { return clusterIDFrom(ctx, cs) } +// DiscoverInClusterClient loads the target cluster from opts and discovers a live +// tracebloc client already installed on it (see DiscoverInClusterClientID — +// RFC-0001 §7.2 step 1, the anchor for R7 adopt-backfill). Mirrors ClusterID's +// best-effort, time-bounded read so `client create` never blocks on it: a load / +// connect failure returns the error and callers fall back to a plain create. +func DiscoverInClusterClient(ctx context.Context, opts KubeconfigOptions) (*InClusterClient, error) { + rc, err := Load(opts) + if err != nil { + return nil, err + } + rc.RestConfig.Timeout = clusterIDReadTimeout + cs, err := NewClientset(rc) + if err != nil { + return nil, err + } + return DiscoverInClusterClientID(ctx, cs) +} + // clusterIDFrom reads the kube-system UID from a clientset. Split out so it can be // exercised with a fake clientset without a real cluster. func clusterIDFrom(ctx context.Context, cs kubernetes.Interface) (string, error) { diff --git a/internal/config/config.go b/internal/config/config.go index da62557..0ce0b81 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -39,9 +39,18 @@ const defaultEnv = "prod" // partially-configured profile stays small and forward-compatible. type Profile struct { Email string `json:"email,omitempty"` // who is signed in (display only) + FirstName string `json:"first_name,omitempty"` // signed-in user's first name; auto-names clients (cli#137) Token string `json:"token,omitempty"` // user token from device login ExpiresAt string `json:"expires_at,omitempty"` // token expiry (RFC 3339), when known ActiveClientID string `json:"active_client_id,omitempty"` // client this machine enrolls as, for THIS env + + // ActiveClientNamespace + ActiveClientName cache the active client's k8s + // namespace and display name at `client use`/`create` time, so the data + // commands can bind to the active client's cluster (RFC-0001 §7.3) without + // a backend round-trip — they run cluster-local and may be offline. Empty + // when no client is active or for pre-v2 configs that predate the cache. + ActiveClientNamespace string `json:"active_client_namespace,omitempty"` + ActiveClientName string `json:"active_client_name,omitempty"` } // Config is the on-disk CLI state: env-scoped profiles plus the current env. diff --git a/internal/doctor/doctor.go b/internal/doctor/doctor.go index d69ce9c..43d8b0f 100644 --- a/internal/doctor/doctor.go +++ b/internal/doctor/doctor.go @@ -123,10 +123,14 @@ func Run(ctx context.Context, cs kubernetes.Interface, opts Options) []Result { func checkReachable(release *cluster.ParentRelease, err error, ns string) Result { const name = "Cluster reachable" if err != nil { + // The discovery error's remediation tail points at cluster doctor — + // which is what's running. Strip it so doctor never tells the user + // to run doctor. + detail := strings.TrimSuffix(strings.TrimSpace(err.Error()), "Diagnose with `tracebloc cluster doctor`.") return Result{ Name: name, Status: StatusFail, - Detail: err.Error(), + Detail: strings.TrimSpace(detail), Remedy: "Check your kubeconfig/context and that the tracebloc client chart is installed here: kubectl get deploy -n " + ns, } } @@ -418,7 +422,7 @@ func checkImagePull(ctx context.Context, cs kubernetes.Interface, ns string, rel Name: name, Status: StatusWarn, Detail: "couldn't read jobs-manager to resolve image pull secrets — skipping", - Remedy: "Check the parent client release is installed in " + ns + ".", + Remedy: "Check a tracebloc client is installed in " + ns + ".", } } secrets := dep.Spec.Template.Spec.ImagePullSecrets diff --git a/internal/push/detect.go b/internal/push/detect.go index 9a2b838..896d45c 100644 --- a/internal/push/detect.go +++ b/internal/push/detect.go @@ -9,9 +9,9 @@ import ( // Register the stdlib image decoders so image.DecodeConfig can // read the headers of the formats the image_classification layout - // accepts. webp is NOT in the stdlib — DetectImageSize returns an - // error for it and the caller falls back to requiring - // --target-size. (.jpg/.jpeg both decode via image/jpeg.) + // accepts (.jpg/.jpeg both decode via image/jpeg). Formats without + // a registered decoder never reach this function anymore — Discover + // skips anything outside the ingestor's accept-set (cli#68). _ "image/gif" _ "image/jpeg" _ "image/png" @@ -22,9 +22,10 @@ import ( // read the pixel data, so it's cheap even for large images). // // Supports the stdlib-registered formats (jpeg, png, gif). Returns an -// error for formats without a registered decoder (notably webp); the -// caller treats that as "couldn't auto-detect" and falls back to the -// ingestor default, advising --target-size. +// error for formats without a registered decoder; the caller treats +// that as "couldn't auto-detect" and falls back to the ingestor +// default, advising --target-size. (Since Discover only yields the +// ingestor's accept-set — .jpg/.jpeg/.png — that path is defensive.) func DetectImageSize(path string) (width, height int, err error) { f, err := os.Open(path) if err != nil { diff --git a/internal/push/orphan.go b/internal/push/orphan.go index 65ab4c0..483793a 100644 --- a/internal/push/orphan.go +++ b/internal/push/orphan.go @@ -147,7 +147,7 @@ func FormatOrphansWarning(orphans []Orphan) string { } var s string s += fmt.Sprintf("WARNING: %d orphan stage Pod%s detected in this namespace — likely "+ - "leftover from a previously crashed `dataset push`:\n", + "leftover from a previously crashed `data ingest`:\n", len(orphans), pluralS(len(orphans))) names := make([]string, 0, len(orphans)) for _, o := range orphans { diff --git a/internal/push/parity_golden_test.go b/internal/push/parity_golden_test.go new file mode 100644 index 0000000..fda5a72 --- /dev/null +++ b/internal/push/parity_golden_test.go @@ -0,0 +1,126 @@ +package push + +import ( + "encoding/json" + "os" + "path/filepath" + "sort" + "testing" +) + +// The validator-parity harness (backend#828 P3). Two assertions per case: +// +// 1. the Go preflight's verdict matches the manifest's cli_verdict — +// pins the CLI side; +// 2. the COMMITTED goldens (generated from the real data-ingestors +// validators by scripts/gen-validator-goldens.py) match the +// manifest's ingestor_verdict — so when the ingestor's rules change, +// regenerating the goldens fails this test until the manifest (and, +// where needed, the Go preview) is consciously updated. +// +// Deliberate divergences (the CLI previewing read-/transfer-time failures +// the ingestor's preflight can't see) are explicit in the manifest, never +// silent. + +type parityCase struct { + Name string `json:"name"` + Category string `json:"category"` + CSV string `json:"csv"` + LabelColumn string `json:"label_column"` + Extension string `json:"extension"` + TargetSize []int `json:"target_size"` + CLIVerdict string `json:"cli_verdict"` + IngestorVerdict string `json:"ingestor_verdict"` + Note string `json:"note"` +} + +func TestValidatorParity(t *testing.T) { + var manifest struct { + Cases []parityCase `json:"cases"` + } + mustLoad(t, filepath.Join("testdata", "parity", "cases.json"), &manifest) + + var goldens struct { + Verdicts map[string]struct { + Verdict string `json:"verdict"` + Errors []string `json:"errors"` + } `json:"verdicts"` + } + mustLoad(t, filepath.Join("testdata", "parity", "goldens.json"), &goldens) + + for _, c := range manifest.Cases { + t.Run(c.Name, func(t *testing.T) { + golden, ok := goldens.Verdicts[c.Name] + if !ok { + t.Fatalf("no golden for %s — run scripts/gen-validator-goldens.py", c.Name) + } + if golden.Verdict != c.IngestorVerdict { + t.Errorf("the REAL ingestor validators say %q but the manifest expects %q — "+ + "the ingestor's rules changed; update cases.json (and the Go preview if needed). "+ + "Golden errors: %v", golden.Verdict, c.IngestorVerdict, golden.Errors) + } + got := runGoPreflight(t, c) + if got != c.CLIVerdict { + t.Errorf("Go preflight = %q, manifest expects %q (note: %s)", got, c.CLIVerdict, c.Note) + } + }) + } +} + +// runGoPreflight runs THE production dispatch (push.PreflightDataset) over +// the case — the same code path runDataIngest executes, so a check deleted +// or rewired in production fails parity here. +func runGoPreflight(t *testing.T, c parityCase) string { + t.Helper() + dir := filepath.Join("testdata", "parity", "cases", c.Name) + layout := &LocalLayout{ + Root: dir, + LabelsCSV: filepath.Join(dir, c.CSV), + Images: listImages(t, dir), + } + spec := SpecArgs{ + Category: c.Category, + LabelColumn: c.LabelColumn, + Extension: c.Extension, + TargetSize: c.TargetSize, + } + if IsTabular(c.Category) { + // Mirror runDataIngest: the schema is inferred from the CSV before + // the preflight runs (the schema-columns preview needs it). + if sch, _, _, err := InferSchema(layout.LabelsCSV); err == nil { + spec.Schema = sch + } + } + _, problem := PreflightDataset(spec, layout) + if problem != nil { + return "reject" + } + return "accept" +} + +func listImages(t *testing.T, dir string) []string { + t.Helper() + entries, err := os.ReadDir(filepath.Join(dir, "images")) + if err != nil { + return nil // tabular/text cases have no images/ + } + var out []string + for _, e := range entries { + if !e.IsDir() { + out = append(out, filepath.Join(dir, "images", e.Name())) + } + } + sort.Strings(out) + return out +} + +func mustLoad(t *testing.T, path string, into any) { + t.Helper() + b, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + if err := json.Unmarshal(b, into); err != nil { + t.Fatalf("%s: %v", path, err) + } +} diff --git a/internal/push/preflight.go b/internal/push/preflight.go new file mode 100644 index 0000000..0e67e7e --- /dev/null +++ b/internal/push/preflight.go @@ -0,0 +1,724 @@ +// Package push — local preflight previews of the in-cluster ingestor's +// validators (backend#828 P3; closes cli#69/#71/#72/#73). +// +// THE CONTRACT: every check in this file previews a NAMED rule in +// tracebloc/data-ingestors, so "preflight passed" means "the ingestor's +// validation will pass" — the customer finds out BEFORE the upload, not +// after. The CLI never invents rules of its own here: stricter-than-ingestor +// checks would reject datasets the cluster accepts, looser ones burn +// uploads. Parity is pinned by internal/push/parity_golden_test.go against +// goldens generated from the real Python validators +// (scripts/gen-validator-goldens.py); regenerate them whenever the +// ingestor's rules change. +package push + +import ( + "bufio" + "bytes" + "encoding/csv" + "errors" + "fmt" + "image" + "io" + "os" + "path/filepath" + "sort" + "strconv" + "strings" + "unicode/utf8" +) + +// utf8BOM is the byte-order mark Excel's "CSV UTF-8" export prepends. +var utf8BOM = []byte{0xEF, 0xBB, 0xBF} + +// HasBOM reports whether the file starts with a UTF-8 BOM. +func HasBOM(path string) (bool, error) { + f, err := os.Open(path) + if err != nil { + return false, err + } + defer func() { _ = f.Close() }() + head := make([]byte, 3) + n, err := io.ReadFull(f, head) + if err != nil && n < 3 { + return false, nil // shorter than a BOM — trivially no BOM + } + return bytes.Equal(head, utf8BOM), nil +} + +// ReadCSVHeader returns the first record of the CSV with a UTF-8 BOM (if +// any) stripped and surrounding whitespace trimmed from each name. +// +// BOM parity (cli#71): the ingestor reads CSVs with pandas, which strips +// the BOM even under encoding="utf-8" — so for the pandas-backed checks +// (label column, row counts) a BOM'd file behaves as if it had none, and +// this reader must match or the CLI would falsely reject what the cluster +// accepts. The one in-cluster path that does NOT strip it is the tabular +// schema probe — see CheckTabularBOM. +func ReadCSVHeader(path string) ([]string, error) { + f, err := os.Open(path) + if err != nil { + return nil, err + } + defer func() { _ = f.Close() }() + br := bufio.NewReader(f) + if head, _ := br.Peek(3); bytes.Equal(head, utf8BOM) { + _, _ = br.Discard(3) + } + r := csv.NewReader(br) + header, err := r.Read() + if err != nil { + if errors.Is(err, io.EOF) { + return nil, fmt.Errorf("%s is empty — no header row", filepath.Base(path)) + } + return nil, fmt.Errorf("reading %s header: %w", filepath.Base(path), err) + } + for i := range header { + header[i] = strings.TrimSpace(header[i]) + } + return header, nil +} + +// CheckTabularBOM previews the in-cluster tabular schema probe +// (data-ingestors data_validator.py _validate_csv_streaming): unlike the +// pandas paths, that probe reads the header with Python's stdlib csv over +// plain utf-8, so a BOM glues U+FEFF onto the first column name and the +// schema check FALSELY rejects the file after the full upload +// ("Schema columns not present in CSV: "). Until that +// ingestor bug is fixed and deployed, a BOM'd tabular CSV is doomed +// in-cluster — reject it here, before the upload, with the actual fix. +func CheckTabularBOM(path string) error { + bom, err := HasBOM(path) + if err != nil { + return fmt.Errorf("checking %s for a byte-order mark: %w", filepath.Base(path), err) + } + if !bom { + return nil + } + return fmt.Errorf( + "%s starts with a UTF-8 byte-order mark (Excel's \"CSV UTF-8\" export adds it), "+ + "which the cluster's schema check can't read — the ingestion would fail after "+ + "uploading everything. Re-save it without the mark (in Excel choose plain \"CSV\"; "+ + "or: tail -c +4 %s > fixed.csv) and re-run.", + filepath.Base(path), filepath.Base(path)) +} + +// CheckLabelColumn previews the ingestor's LabelColumnValidator +// (label_column_validator.py): the configured label column must exist in +// the CSV header — matched exactly first, then case-insensitively with +// surrounding whitespace stripped (the ingestor's _match_column rule; it +// must stay this loose or the CLI would reject datasets the cluster +// accepts). +func CheckLabelColumn(header []string, labelColumn, csvName string) error { + for _, c := range header { + if c == labelColumn { + return nil + } + } + want := strings.ToLower(strings.TrimSpace(labelColumn)) + for _, c := range header { + if strings.ToLower(strings.TrimSpace(c)) == want { + return nil + } + } + return fmt.Errorf( + "label column %q isn't in %s's header (columns: %s). Pass --label-column with one of "+ + "the existing columns, or add a %q column to the CSV.", + labelColumn, csvName, strings.Join(header, ", "), labelColumn) +} + +// CheckDuplicateHeaders previews the ingestor's duplicate-header probes +// (data_validator.py preflight + csv_ingestor.py read time): duplicate +// column names — compared stripped but case-SENSITIVE, exactly as the +// ingestor compares them — are rejected there, and the read-time copy +// fires after table creation, leaving an orphaned empty table. Catch it +// locally instead. (InferSchema's map keying would also silently collapse +// the duplicate — cli#73a.) +func CheckDuplicateHeaders(header []string, csvName string) error { + seen := make(map[string]bool, len(header)) + var dups []string + for _, c := range header { + if seen[c] { + dups = append(dups, c) + } + seen[c] = true + } + if len(dups) == 0 { + return nil + } + sort.Strings(dups) + return fmt.Errorf( + "%s has duplicate column name(s): %s. Each column must be unique — the cluster "+ + "rejects duplicates, and the schema would map onto the wrong column. Rename them and re-run.", + csvName, strings.Join(dups, ", ")) +} + +// CheckHasDataRows previews the ingestor's IngestableRecordsValidator +// (ingestable_records_validator.py _check_has_rows, run for every +// category): a header-only CSV has zero ingestable records and is +// rejected in-cluster before any table is created. +func CheckHasDataRows(path string) error { + f, err := os.Open(path) + if err != nil { + return fmt.Errorf("reading %s: %w", filepath.Base(path), err) + } + defer func() { _ = f.Close() }() + br := bufio.NewReader(f) + if head, _ := br.Peek(3); bytes.Equal(head, utf8BOM) { + _, _ = br.Discard(3) + } + r := csv.NewReader(br) + r.FieldsPerRecord = -1 // row-shape problems are someone else's diagnostic + if _, err := r.Read(); err != nil { + if errors.Is(err, io.EOF) { + return fmt.Errorf("%s is empty — add a header and at least one data row, then re-run", filepath.Base(path)) + } + return fmt.Errorf("reading %s: %w", filepath.Base(path), err) + } + if _, err := r.Read(); err != nil { + if errors.Is(err, io.EOF) { + return fmt.Errorf( + "%s has a header but no data rows (0 ingestable records). Add at least one data row and re-run.", + filepath.Base(path)) + } + return fmt.Errorf("reading %s: %w", filepath.Base(path), err) + } + return nil +} + +// ValidateImages previews the ingestor's ImageResolutionValidator +// (image_validator.py): it opens EVERY image (header-only decode — cheap) +// and rejects zero-byte files, undecodable files, and any image whose +// resolution differs from the expected size (exact equality, zero +// tolerance — the ingestor validates, it does not resize). Previously the +// CLI decoded only the first image, so a single odd-sized or corrupt file +// failed in-cluster after the full upload (cli#72b/c). +// +// expectedW/expectedH of 0 skips the resolution comparison (the caller +// couldn't establish a target size — the ingestor would then auto-detect +// from its first file, which the CLI's detection already mirrors). +func ValidateImages(images []string, expectedW, expectedH int) error { + const maxListed = 5 + var broken, mismatched []string + for _, path := range images { + name := filepath.Base(path) + f, err := os.Open(path) + if err != nil { + broken = append(broken, fmt.Sprintf("%s (unreadable: %v)", name, err)) + continue + } + cfg, _, err := image.DecodeConfig(f) + _ = f.Close() + if err != nil { + if st, serr := os.Stat(path); serr == nil && st.Size() == 0 { + broken = append(broken, name+" (empty file, 0 bytes)") + } else { + broken = append(broken, name+" (not a valid image — corrupt or unsupported format)") + } + continue + } + if expectedW > 0 && expectedH > 0 && (cfg.Width != expectedW || cfg.Height != expectedH) { + mismatched = append(mismatched, + fmt.Sprintf("%s (%dx%d)", name, cfg.Width, cfg.Height)) + } + } + if len(broken) > 0 { + return fmt.Errorf( + "%d image(s) can't be ingested: %s. The cluster rejects these after the upload — "+ + "fix or remove them and re-run.", + len(broken), TruncateList(broken, maxListed)) + } + if len(mismatched) > 0 { + return fmt.Errorf( + "%d image(s) don't match the %dx%d resolution: %s. The cluster validates the size, "+ + "it does not resize — make them uniform, or pass --target-size to match your data.", + len(mismatched), expectedW, expectedH, TruncateList(mismatched, maxListed)) + } + return nil +} + +// CrossCheckLabels previews the transfer-time fate of each labels.csv row +// for image_classification (file_transfer.py _find_src): a row whose image +// file doesn't exist under images/ is dropped as a failed record — the run +// then "completes with failures" (exit 9) after the full upload. The +// filename column may omit the extension; the ingestor appends the +// dataset's extension in that case, and this check mirrors that. Files +// with no CSV row are NOT an error (the ingestor never checks that +// direction for image_classification) — the caller may surface them as a +// note. +// +// filenameColumn is the CSV's first column (the ingestor reads filenames +// positionally from the id column of labels.csv). +func CrossCheckLabels(csvPath string, images []string, extension string) (missing []string, orphans []string, err error) { + f, err := os.Open(csvPath) + if err != nil { + return nil, nil, fmt.Errorf("reading %s: %w", filepath.Base(csvPath), err) + } + defer func() { _ = f.Close() }() + br := bufio.NewReader(f) + if head, _ := br.Peek(3); bytes.Equal(head, utf8BOM) { + _, _ = br.Discard(3) + } + r := csv.NewReader(br) + r.FieldsPerRecord = -1 + + present := make(map[string]bool, len(images)) + for _, img := range images { + present[filepath.Base(img)] = true + } + referenced := make(map[string]bool) + + if _, err := r.Read(); err != nil { // header + if errors.Is(err, io.EOF) { + return nil, nil, nil // emptiness is CheckHasDataRows' diagnostic + } + return nil, nil, fmt.Errorf("reading %s: %w", filepath.Base(csvPath), err) + } + for { + rec, err := r.Read() + if errors.Is(err, io.EOF) { + break + } + if err != nil { + return nil, nil, fmt.Errorf("reading %s: %w", filepath.Base(csvPath), err) + } + if len(rec) == 0 { + continue + } + name := strings.TrimSpace(rec[0]) + if name == "" { + continue + } + // Mirror the ingestor's extension handling (_has_extension): the + // dataset extension is appended unless the name already ends in a + // KNOWN media extension — a dotted stem like "img.2024" is not an + // extension to the ingestor, and must not be one here either. + if !hasKnownExtension(name) && extension != "" { + name += extension + } + referenced[name] = true + if !present[name] { + missing = append(missing, name) + } + } + for name := range present { + if !referenced[name] { + orphans = append(orphans, name) + } + } + sort.Strings(missing) + sort.Strings(orphans) + return missing, orphans, nil +} + +// CheckAnnotationPairing previews the ingestor's FilePairingValidator +// (file_pairing_validator.py) for object_detection: every image must have +// an annotation with the same filename stem and vice versa — a mismatch in +// either direction fails in-cluster after the upload. +func CheckAnnotationPairing(images, annotations []string) error { + const maxListed = 5 + stems := func(paths []string) map[string]bool { + m := make(map[string]bool, len(paths)) + for _, p := range paths { + base := filepath.Base(p) + m[strings.TrimSuffix(base, filepath.Ext(base))] = true + } + return m + } + imgStems, annStems := stems(images), stems(annotations) + var noAnn, noImg []string + for s := range imgStems { + if !annStems[s] { + noAnn = append(noAnn, s) + } + } + for s := range annStems { + if !imgStems[s] { + noImg = append(noImg, s) + } + } + if len(noAnn) == 0 && len(noImg) == 0 { + return nil + } + sort.Strings(noAnn) + sort.Strings(noImg) + var parts []string + if len(noAnn) > 0 { + parts = append(parts, fmt.Sprintf("%d image(s) without an annotation (%s)", + len(noAnn), TruncateList(noAnn, maxListed))) + } + if len(noImg) > 0 { + parts = append(parts, fmt.Sprintf("%d annotation(s) without an image (%s)", + len(noImg), TruncateList(noImg, maxListed))) + } + return fmt.Errorf( + "images/ and annotations/ don't pair up: %s. Every image needs a same-named .xml "+ + "annotation (and vice versa) — the cluster rejects mismatches after the upload.", + strings.Join(parts, "; ")) +} + +// TruncateList joins up to max items, appending "… and N more" past that. +func TruncateList(items []string, max int) string { + if len(items) <= max { + return strings.Join(items, ", ") + } + return fmt.Sprintf("%s, … and %d more", strings.Join(items[:max], ", "), len(items)-max) +} + +// CheckLabelDiversity previews the ingestor's LabelDiversityValidator +// (label_diversity_validator.py, wired for every classification category): +// a classification dataset needs at least 2 distinct label values +// (whitespace-stripped) — a single class can't train a classifier, and the +// in-cluster rejection otherwise lands after the full upload. Mirrors the +// validator's benign-skip when the label column isn't found (that's +// CheckLabelColumn's diagnostic, not this one's). +// +// dropNASentinels and collapseNumeric mirror the ingestor's per-column +// read (LabelDiversityValidator._label_read_kwargs): for a SCHEMA-TYPED +// tabular label pandas drops NA-sentinel values (na_values) and, for +// NUMERIC types only, numeric inference collapses "1"/"1.0" — but a +// string-family type (VARCHAR/CHAR/TEXT/STRING) is pinned to dtype=str, so +// numeric-looking labels stay distinct (data-ingestors #252). Image/text +// labels are read untyped with keep_default_na=False (both flags false), so +// even an empty string is a real class and every distinct trimmed string +// counts. The caller derives the two flags from the label's schema type. +func CheckLabelDiversity(csvPath, labelColumn string, dropNASentinels, collapseNumeric bool) error { + f, err := os.Open(csvPath) + if err != nil { + return nil // unreadable file is another check's diagnostic + } + defer func() { _ = f.Close() }() + br := bufio.NewReader(f) + if head, _ := br.Peek(3); bytes.Equal(head, utf8BOM) { + _, _ = br.Discard(3) + } + r := csv.NewReader(br) + r.FieldsPerRecord = -1 + header, err := r.Read() + if err != nil { + return nil + } + col := -1 + for i, c := range header { + if strings.TrimSpace(c) == labelColumn { + col = i + break + } + } + if col == -1 { + want := strings.ToLower(strings.TrimSpace(labelColumn)) + for i, c := range header { + if strings.ToLower(strings.TrimSpace(c)) == want { + col = i + break + } + } + } + if col == -1 { + return nil // benign-skip, like the ingestor + } + distinct := map[string]bool{} + for { + rec, err := r.Read() + if errors.Is(err, io.EOF) { + break + } + if err != nil || len(rec) <= col { + continue + } + v := strings.TrimSpace(rec[col]) + if dropNASentinels { + if _, isNA := naSentinels[v]; isNA { + continue + } + } + if collapseNumeric { + // Numeric inference collapses "1" and "1.0" into one value + // in-cluster; normalize the same way before counting. + if f, err := strconv.ParseFloat(v, 64); err == nil { + v = strconv.FormatFloat(f, 'g', -1, 64) + } + } + distinct[v] = true + if len(distinct) >= 2 { + return nil + } + } + if len(distinct) >= 2 { + return nil + } + return fmt.Errorf( + "the label column %q has %d distinct value(s) — a classification dataset needs at "+ + "least 2 classes. The cluster rejects this after the upload; check the labels and re-run.", + labelColumn, len(distinct)) +} + +// knownMediaExtensions mirrors the ingestor's FileExtension.get_all_extensions +// (utils/constants.py): the ONLY suffixes file_transfer._has_extension treats +// as extensions. Anything else — img.2024, photo.v2 — gets the dataset +// extension appended by the ingestor, and CrossCheckLabels must mirror that +// or it rejects dotted stems the cluster resolves fine. +var knownMediaExtensions = map[string]struct{}{ + ".jpeg": {}, ".jpg": {}, ".png": {}, ".xml": {}, ".txt": {}, ".text": {}, +} + +// hasKnownExtension previews file_transfer._has_extension: true only when the +// name's final dot-suffix is one of the ingestor's known extensions +// (case-insensitive). +func hasKnownExtension(name string) bool { + _, ok := knownMediaExtensions[strings.ToLower(filepath.Ext(name))] + return ok +} + +// CheckCSVEncoding previews the ingestor's preflight.check_csv_encoding — +// the FIRST gate validate_data runs in-cluster: the CSV must be valid UTF-8 +// and free of NUL bytes, or the whole run aborts (after the upload). +func CheckCSVEncoding(path string) error { + f, err := os.Open(path) + if err != nil { + return fmt.Errorf("reading %s: %w", filepath.Base(path), err) + } + defer func() { _ = f.Close() }() + data, err := io.ReadAll(io.LimitReader(f, MaxSingleFileBytes+1)) + if err != nil { + return fmt.Errorf("reading %s: %w", filepath.Base(path), err) + } + if bytes.IndexByte(data, 0x00) >= 0 { + return fmt.Errorf( + "%s contains a NUL byte — the file is corrupt or not really a CSV. The cluster "+ + "rejects it after the upload; re-export the file and re-run.", + filepath.Base(path)) + } + if !utf8.Valid(data) { + return fmt.Errorf( + "%s isn't valid UTF-8 (likely a Latin-1/Windows-1252 export). The cluster rejects "+ + "non-UTF-8 CSVs after the upload — re-save it as UTF-8 and re-run.", + filepath.Base(path)) + } + return nil +} + +// naSentinels mirrors the ingestor's coercion.NA_SENTINELS: values pandas +// drops to NaN for SCHEMA-TYPED columns (tabular labels), and therefore +// values the in-cluster LabelDiversityValidator does not count as classes. +var naSentinels = map[string]struct{}{ + "": {}, "NA": {}, "N/A": {}, "n/a": {}, "NULL": {}, "null": {}, + "None": {}, "none": {}, "NaN": {}, "nan": {}, "": {}, "#N/A": {}, +} + +// labelSchemaType resolves the label column's declared SQL type from the +// schema, matched case- and whitespace-insensitively — mirrors the ingestor's +// LabelDiversityValidator._schema_type_for. ok is false when the label isn't +// a schema column (an untyped read, in-cluster). +func labelSchemaType(schema map[string]string, labelColumn string) (sqlType string, ok bool) { + if t, found := schema[labelColumn]; found { + return t, true + } + target := strings.ToLower(strings.TrimSpace(labelColumn)) + for k, v := range schema { + if strings.ToLower(strings.TrimSpace(k)) == target { + return v, true + } + } + return "", false +} + +// isStringSQLType reports whether an SQL type declaration is a string family +// (VARCHAR/CHAR/TEXT/STRING) — the types the ingestor pins to dtype=str, +// which suppresses pandas numeric inference on the label column. Mirrors the +// base-type check in LabelDiversityValidator._label_read_kwargs. +func isStringSQLType(sqlType string) bool { + base := strings.ToUpper(strings.TrimSpace(sqlType)) + if i := strings.IndexByte(base, '('); i >= 0 { + base = strings.TrimSpace(base[:i]) + } + switch base { + case "VARCHAR", "CHAR", "TEXT", "STRING": + return true + } + return false +} + +// CheckSchemaColumns previews DataValidator's missing-schema-column probe +// ("Schema columns not present in CSV: …"): every schema column must appear +// in the header, compared stripped but case-SENSITIVE — exactly the probe's +// set difference. +func CheckSchemaColumns(header []string, schema map[string]string, csvName string) error { + present := make(map[string]bool, len(header)) + for _, c := range header { + present[strings.TrimSpace(c)] = true + } + var missing []string + for col := range schema { + if !present[strings.TrimSpace(col)] { + missing = append(missing, col) + } + } + if len(missing) == 0 { + return nil + } + sort.Strings(missing) + return fmt.Errorf( + "--schema names column(s) that aren't in %s: %s. The cluster rejects this after the "+ + "upload — fix the schema or the CSV header, then re-run.", + csvName, strings.Join(missing, ", ")) +} + +// PreflightProblem is a preflight rejection. BadFlag marks problems whose +// fix is a flag value (the CLI maps those to exit 2); everything else is a +// data problem (exit 3). +type PreflightProblem struct { + Err error + BadFlag bool +} + +// PreflightDataset is THE preflight dispatch — the single place that decides +// which previews run for which category, shared verbatim by the CLI +// (runLocalPreflight) and the parity harness (parity_golden_test.go), so the +// two cannot drift: deleting or rewiring a check here fails the parity test. +// Returned notes are advisory (the CLI prints them dim); a non-nil problem +// is the rejection. +// +// Ordering mirrors the in-cluster pipeline: encoding gate first (the +// ingestor's check_csv_encoding runs before any validator), then the +// validator previews. +func PreflightDataset(spec SpecArgs, layout *LocalLayout) (notes []string, problem *PreflightProblem) { + dataProblem := func(err error) *PreflightProblem { + if err == nil { + return nil + } + return &PreflightProblem{Err: err} + } + + switch { + case IsTabular(spec.Category): + if err := CheckTabularBOM(layout.LabelsCSV); err != nil { + return nil, dataProblem(err) + } + if err := CheckCSVEncoding(layout.LabelsCSV); err != nil { + return nil, dataProblem(err) + } + header, err := ReadCSVHeader(layout.LabelsCSV) + if err != nil { + return nil, dataProblem(err) + } + if err := CheckDuplicateHeaders(header, "the data CSV"); err != nil { + return nil, dataProblem(err) + } + if err := CheckHasDataRows(layout.LabelsCSV); err != nil { + return nil, dataProblem(err) + } + if err := CheckSchemaColumns(header, spec.Schema, "the data CSV"); err != nil { + return nil, dataProblem(err) + } + // A bogus --label-column otherwise fails in-cluster only at READ + // time — after the table was created — leaving an orphaned table. + if err := CheckLabelColumn(header, spec.LabelColumn, "the data CSV"); err != nil { + return nil, &PreflightProblem{Err: err, BadFlag: true} + } + if spec.Category == "tabular_classification" { + // The label is a schema-typed column: the ingestor drops NA + // sentinels for it, and collapses numeric-looking values ONLY + // for numeric types — a VARCHAR label is pinned to dtype=str, + // keeping "1"/"1.0" distinct (data-ingestors #252). Derive both + // flags from the label's declared type so the preview doesn't + // wrongly collapse a string label and reject a diverse dataset. + sqlType, inSchema := labelSchemaType(spec.Schema, spec.LabelColumn) + dropNA := inSchema + collapseNumeric := !(inSchema && isStringSQLType(sqlType)) + if err := CheckLabelDiversity(layout.LabelsCSV, spec.LabelColumn, dropNA, collapseNumeric); err != nil { + return nil, dataProblem(err) + } + } + + case IsImage(spec.Category): + if err := CheckCSVEncoding(layout.LabelsCSV); err != nil { + return nil, dataProblem(err) + } + // Every image, header-only decode (cheap). Previously only the + // first image was ever opened, so one corrupt or odd-sized file + // failed in-cluster after the full upload (cli#72). + expW, expH := 0, 0 + if len(spec.TargetSize) == 2 { + expW, expH = spec.TargetSize[0], spec.TargetSize[1] + } + if err := ValidateImages(layout.Images, expW, expH); err != nil { + return nil, dataProblem(err) + } + if err := CheckHasDataRows(layout.LabelsCSV); err != nil { + return nil, dataProblem(err) + } + header, err := ReadCSVHeader(layout.LabelsCSV) + if err != nil { + return nil, dataProblem(err) + } + if err := CheckDuplicateHeaders(header, "labels.csv"); err != nil { + return nil, dataProblem(err) + } + // LabelDiversityValidator runs in-cluster for the WHOLE image + // family (is_classification covers object_detection + keypoint + // too); it benign-skips when no label column resolves, and so + // does the preview. Image labels are read untyped, so no NA drop + // and no numeric collapse. + if err := CheckLabelDiversity(layout.LabelsCSV, spec.LabelColumn, false, false); err != nil { + return nil, dataProblem(err) + } + switch spec.Category { + case "image_classification": + if err := CheckLabelColumn(header, spec.LabelColumn, "labels.csv"); err != nil { + return nil, &PreflightProblem{Err: err, BadFlag: true} + } + // A row whose image is missing becomes a failed record + // in-cluster ("completed with failures", exit 9) — after the + // upload. Extra files are only a note (the ingestor never + // checks that direction). + missing, orphanFiles, err := CrossCheckLabels(layout.LabelsCSV, layout.Images, spec.Extension) + if err != nil { + return nil, dataProblem(err) + } + if len(missing) > 0 { + return nil, dataProblem(fmt.Errorf( + "%d labels.csv row(s) reference images that aren't in images/: %s. Those records "+ + "would fail after the upload — fix the rows or add the files, then re-run.", + len(missing), TruncateList(missing, 5))) + } + if len(orphanFiles) > 0 { + notes = append(notes, fmt.Sprintf( + "Note: %d file(s) in images/ have no labels.csv row and won't be part of the dataset: %s", + len(orphanFiles), TruncateList(orphanFiles, 5))) + } + case "object_detection": + // images↔annotations stem pairing (FilePairingValidator preview). + if err := CheckAnnotationPairing(layout.Images, layout.Sidecars["annotations"]); err != nil { + return nil, dataProblem(err) + } + } + + default: + // Text family. + if err := CheckCSVEncoding(layout.LabelsCSV); err != nil { + return nil, dataProblem(err) + } + if err := CheckHasDataRows(layout.LabelsCSV); err != nil { + return nil, dataProblem(err) + } + header, err := ReadCSVHeader(layout.LabelsCSV) + if err != nil { + return nil, dataProblem(err) + } + if err := CheckDuplicateHeaders(header, "labels.csv"); err != nil { + return nil, dataProblem(err) + } + if spec.Category == "text_classification" { + if err := CheckLabelColumn(header, spec.LabelColumn, "labels.csv"); err != nil { + return nil, &PreflightProblem{Err: err, BadFlag: true} + } + // Text labels are read untyped (like image), so no NA drop and + // no numeric collapse. + if err := CheckLabelDiversity(layout.LabelsCSV, spec.LabelColumn, false, false); err != nil { + return nil, dataProblem(err) + } + } + } + return notes, nil +} diff --git a/internal/push/preflight_test.go b/internal/push/preflight_test.go new file mode 100644 index 0000000..ed546f4 --- /dev/null +++ b/internal/push/preflight_test.go @@ -0,0 +1,254 @@ +package push + +import ( + "image" + "image/png" + "os" + "path/filepath" + "strings" + "testing" +) + +func writeTmp(t *testing.T, name string, body []byte) string { + t.Helper() + p := filepath.Join(t.TempDir(), name) + if err := os.WriteFile(p, body, 0o644); err != nil { + t.Fatal(err) + } + return p +} + +var bom = []byte{0xEF, 0xBB, 0xBF} + +func TestReadCSVHeader_StripsBOMAndTrims(t *testing.T) { + // Parity (cli#71): pandas strips the BOM in-cluster, so the local + // reader must too — otherwise the CLI would reject label columns the + // cluster accepts. + p := writeTmp(t, "labels.csv", append(bom, []byte(" filename , label \nx.jpg,cat\n")...)) + h, err := ReadCSVHeader(p) + if err != nil { + t.Fatal(err) + } + if h[0] != "filename" || h[1] != "label" { + t.Errorf("header = %v, want BOM-stripped + trimmed [filename label]", h) + } +} + +func TestCheckTabularBOM(t *testing.T) { + // Parity (cli#71): the in-cluster tabular schema probe does NOT strip + // the BOM and falsely rejects — the CLI must reject locally, before + // the upload, with the actual fix. + withBOM := writeTmp(t, "d.csv", append(bom, []byte("age,income\n1,2\n")...)) + if err := CheckTabularBOM(withBOM); err == nil { + t.Fatal("BOM'd tabular CSV must be rejected (in-cluster schema probe would falsely reject it post-upload)") + } else if !strings.Contains(err.Error(), "byte-order mark") || !strings.Contains(err.Error(), "Re-save") { + t.Errorf("BOM error must explain + remediate, got: %v", err) + } + clean := writeTmp(t, "c.csv", []byte("age,income\n1,2\n")) + if err := CheckTabularBOM(clean); err != nil { + t.Errorf("clean CSV rejected: %v", err) + } +} + +func TestCheckLabelColumn_MatchesLikeTheIngestor(t *testing.T) { + // Parity (cli#69): exact first, then case-insensitive + trimmed — + // the ingestor's _match_column rule. Stricter matching would reject + // datasets the cluster accepts. + header := []string{"filename", " Label "} + if err := CheckLabelColumn(header, "label", "labels.csv"); err != nil { + t.Errorf("case-insensitive+trimmed match must pass (ingestor accepts it): %v", err) + } + if err := CheckLabelColumn(header, "target", "labels.csv"); err == nil { + t.Fatal("absent label column must be rejected") + } else { + for _, want := range []string{`"target"`, "filename", "--label-column"} { + if !strings.Contains(err.Error(), want) { + t.Errorf("error should mention %q, got: %v", want, err) + } + } + } +} + +func TestCheckDuplicateHeaders_CaseSensitiveLikeTheIngestor(t *testing.T) { + // Parity (cli#73a): the ingestor compares stripped but case-SENSITIVE. + if err := CheckDuplicateHeaders([]string{"a", "A"}, "the data CSV"); err != nil { + t.Errorf("'a' vs 'A' are NOT duplicates to the ingestor: %v", err) + } + err := CheckDuplicateHeaders([]string{"age", "income", "age"}, "the data CSV") + if err == nil { + t.Fatal("duplicate headers must be rejected") + } + if !strings.Contains(err.Error(), "age") || !strings.Contains(err.Error(), "Rename") { + t.Errorf("dup error must name the column + remediate: %v", err) + } +} + +func TestCheckHasDataRows(t *testing.T) { + ok := writeTmp(t, "ok.csv", []byte("a,b\n1,2\n")) + if err := CheckHasDataRows(ok); err != nil { + t.Errorf("CSV with rows rejected: %v", err) + } + headerOnly := writeTmp(t, "h.csv", []byte("a,b\n")) + if err := CheckHasDataRows(headerOnly); err == nil { + t.Fatal("header-only CSV must be rejected (cli#73b — 0 ingestable records)") + } else if !strings.Contains(err.Error(), "no data rows") { + t.Errorf("unexpected message: %v", err) + } + empty := writeTmp(t, "e.csv", nil) + if err := CheckHasDataRows(empty); err == nil { + t.Fatal("empty CSV must be rejected") + } +} + +func pngBytes(t *testing.T, w, h int) []byte { + t.Helper() + var sb strings.Builder + if err := png.Encode(&nopWriter{&sb}, image.NewRGBA(image.Rect(0, 0, w, h))); err != nil { + t.Fatal(err) + } + return []byte(sb.String()) +} + +type nopWriter struct{ b *strings.Builder } + +func (w *nopWriter) Write(p []byte) (int, error) { return w.b.Write(p) } + +func TestValidateImages(t *testing.T) { + dir := t.TempDir() + write := func(name string, body []byte) string { + p := filepath.Join(dir, name) + if err := os.WriteFile(p, body, 0o644); err != nil { + t.Fatal(err) + } + return p + } + good := write("good.png", pngBytes(t, 8, 8)) + odd := write("odd.png", pngBytes(t, 4, 4)) + zero := write("zero.png", nil) + corrupt := write("corrupt.png", []byte("not an image at all")) + + if err := ValidateImages([]string{good}, 8, 8); err != nil { + t.Errorf("valid image rejected: %v", err) + } + if err := ValidateImages([]string{good, zero}, 8, 8); err == nil { + t.Fatal("zero-byte image must be rejected (cli#72b)") + } else if !strings.Contains(err.Error(), "0 bytes") { + t.Errorf("zero-byte diagnosis missing: %v", err) + } + if err := ValidateImages([]string{good, corrupt}, 8, 8); err == nil { + t.Fatal("corrupt image must be rejected (cli#72b)") + } + if err := ValidateImages([]string{good, odd}, 8, 8); err == nil { + t.Fatal("resolution mismatch must be rejected (cli#72c — the ingestor validates, it does not resize)") + } else if !strings.Contains(err.Error(), "4x4") || !strings.Contains(err.Error(), "8x8") { + t.Errorf("mismatch error must show both sizes: %v", err) + } + // 0x0 expectation skips the resolution comparison entirely. + if err := ValidateImages([]string{good, odd}, 0, 0); err != nil { + t.Errorf("no expected size → no resolution rejection: %v", err) + } +} + +func TestCrossCheckLabels(t *testing.T) { + dir := t.TempDir() + imgs := filepath.Join(dir, "images") + if err := os.MkdirAll(imgs, 0o755); err != nil { + t.Fatal(err) + } + for _, n := range []string{"a.jpg", "b.jpg", "extra.jpg"} { + if err := os.WriteFile(filepath.Join(imgs, n), []byte("x"), 0o644); err != nil { + t.Fatal(err) + } + } + // One row exact, one extensionless (the ingestor appends the dataset + // extension — the check must mirror that), one missing. + csvPath := filepath.Join(dir, "labels.csv") + if err := os.WriteFile(csvPath, + []byte("image_id,label\na.jpg,cat\nb,dog\nghost.jpg,cat\n"), 0o644); err != nil { + t.Fatal(err) + } + images := []string{filepath.Join(imgs, "a.jpg"), filepath.Join(imgs, "b.jpg"), filepath.Join(imgs, "extra.jpg")} + missing, orphans, err := CrossCheckLabels(csvPath, images, ".jpg") + if err != nil { + t.Fatal(err) + } + if len(missing) != 1 || missing[0] != "ghost.jpg" { + t.Errorf("missing = %v, want [ghost.jpg] (extensionless 'b' must resolve to b.jpg)", missing) + } + if len(orphans) != 1 || orphans[0] != "extra.jpg" { + t.Errorf("orphans = %v, want [extra.jpg]", orphans) + } +} + +func TestCheckAnnotationPairing(t *testing.T) { + imgs := []string{"images/a.jpg", "images/b.jpg"} + anns := []string{"annotations/a.xml", "annotations/c.xml"} + err := CheckAnnotationPairing(imgs, anns) + if err == nil { + t.Fatal("stem mismatch must be rejected (FilePairingValidator preview)") + } + for _, want := range []string{"b", "c", "don't pair up"} { + if !strings.Contains(err.Error(), want) { + t.Errorf("pairing error should mention %q: %v", want, err) + } + } + if err := CheckAnnotationPairing(imgs, []string{"annotations/a.xml", "annotations/b.xml"}); err != nil { + t.Errorf("matched stems rejected: %v", err) + } +} + +func TestCheckLabelDiversity(t *testing.T) { + // Parity: LabelDiversityValidator — classification needs >=2 distinct + // (stripped) label values. Discovered by the parity harness's first + // run; previously an in-cluster-only, post-upload failure. + two := writeTmp(t, "two.csv", []byte("id,label\na,cat\nb, cat \nc,dog\n")) + if err := CheckLabelDiversity(two, "label", false, false); err != nil { + t.Errorf("2 distinct labels rejected: %v", err) + } + one := writeTmp(t, "one.csv", []byte("id,label\na,cat\nb,cat\n")) + if err := CheckLabelDiversity(one, "label", false, false); err == nil { + t.Fatal("single-class dataset must be rejected") + } else if !strings.Contains(err.Error(), "at least 2 classes") { + t.Errorf("unexpected message: %v", err) + } + // benign-skip when the column is absent (that's CheckLabelColumn's job) + if err := CheckLabelDiversity(one, "nope", false, false); err != nil { + t.Errorf("missing column must benign-skip like the ingestor: %v", err) + } + + // Schema-type sensitivity (data-ingestors #252): "1" and "1.0" are two + // distinct classes for a string-typed (VARCHAR) label — the ingestor + // pins dtype=str and does NOT collapse them — but one class for a + // numeric label, where pandas numeric inference merges them. + numeric := writeTmp(t, "numeric.csv", []byte("id,label\na,1\nb,1.0\n")) + if err := CheckLabelDiversity(numeric, "label", true /*dropNA*/, false /*collapseNumeric*/); err != nil { + t.Errorf("VARCHAR label '1'/'1.0' must stay 2 classes (no numeric collapse): %v", err) + } + if err := CheckLabelDiversity(numeric, "label", true /*dropNA*/, true /*collapseNumeric*/); err == nil { + t.Error("numeric label '1'/'1.0' must collapse to a single class and be rejected") + } +} + +func TestCheckLabelDiversitySchemaTypeDispatch(t *testing.T) { + // PreflightDataset must derive the collapse flag from the label's schema + // type: a numeric-looking VARCHAR label is accepted (2 classes), the same + // data typed FLOAT is rejected (collapses to 1). Locks the fix at the + // dispatch level, not just the leaf function. + dir := t.TempDir() + if err := os.WriteFile(filepath.Join(dir, "data.csv"), []byte("feat,label\nx,1\ny,1.0\n"), 0o644); err != nil { + t.Fatal(err) + } + layout := &LocalLayout{Root: dir, LabelsCSV: filepath.Join(dir, "data.csv")} + + strSpec := SpecArgs{Category: "tabular_classification", LabelColumn: "label", + Schema: map[string]string{"feat": "VARCHAR(255)", "label": "VARCHAR(10)"}} + if _, problem := PreflightDataset(strSpec, layout); problem != nil { + t.Errorf("VARCHAR label should keep '1'/'1.0' distinct and pass: %v", problem.Err) + } + + numSpec := SpecArgs{Category: "tabular_classification", LabelColumn: "label", + Schema: map[string]string{"feat": "VARCHAR(255)", "label": "FLOAT"}} + if _, problem := PreflightDataset(numSpec, layout); problem == nil { + t.Error("FLOAT label should collapse '1'/'1.0' and be rejected") + } +} diff --git a/internal/push/spec.go b/internal/push/spec.go index a28d7e8..1d485c1 100644 --- a/internal/push/spec.go +++ b/internal/push/spec.go @@ -144,9 +144,14 @@ type SpecArgs struct { // default apply. Populated by the CLI from --target-size or by // auto-detecting the first image. (Image categories only.) // - // NOTE: stored [W, H] here, but EMITTED as [height, width] by - // Build — that's the order the ingest.v1 schema documents and the - // ingestor reads. buildImage does the swap. (Bugbot, PR #22.) + // NOTE: stored [W, H] here AND emitted as [width, height] by + // buildImage — no swap. That's the order the ingest.v1 schema + // documents ("matches PIL.Image.size and what + // ImageResolutionValidator expects") and the order the validator + // compares against verbatim (PIL's img.size is (W, H)). An earlier + // [H,W] revision (PR #22 review note) was mistaken and made every + // non-square dataset fail in-cluster; #147 reverted it. Do not + // re-introduce a swap. (See the inline comment in buildImage.) TargetSize []int // Schema is the column→SQL-type map for tabular / time-series @@ -175,6 +180,13 @@ type SpecArgs struct { // it from there, so this needs no top-level schema field. 0 ⇒ // unset (ignored for non-keypoint categories). NumberOfKeypoints int + + // Extension is the single image file extension every file in the + // dataset shares (detected by DetectExtension), emitted as + // spec.file_options.extension so the ingestor's FileTypeValidator + // checks against what was actually staged instead of its .jpeg + // convention default (cli#68). + Extension string } // Build produces the ingest.v1.json-conforming spec map. The @@ -256,25 +268,37 @@ func (a SpecArgs) buildImage(spec map[string]any, prefix string) { spec["annotations"] = path.Join(prefix, "annotations") + "/" } + // file_options carries the per-file conventions the ingestor's + // validators read: the detected extension (all categories in the + // image family — FileTypeValidator checks images against it) and, + // for the non-keypoint categories, the resolution override. + fileOptions := map[string]any{} + if a.Extension != "" { + fileOptions["extension"] = a.Extension + } + if a.Category == "keypoint_detection" { if len(a.TargetSize) == 2 { - // Schema documents target_size as [height, width]; TargetSize - // is stored [W, H], so swap on emit. (Bugbot, PR #22.) - spec["target_size"] = []int{a.TargetSize[1], a.TargetSize[0]} + // Emitted as [width, height] — the schema's own description + // says so ("matches PIL.Image.size and what + // ImageResolutionValidator expects"), and the validator + // compares PIL's (W,H) against this list verbatim (verified + // empirically: an 8×4 image passes [8,4], fails [4,8]). An + // earlier revision swapped to [H,W] here on a mistaken review + // note, which made EVERY non-square dataset fail in-cluster + // after the full upload. + spec["target_size"] = []int{a.TargetSize[0], a.TargetSize[1]} } if a.NumberOfKeypoints > 0 { spec["number_of_keypoints"] = a.NumberOfKeypoints } - return + } else if len(a.TargetSize) == 2 { + // [width, height] — same contract as the keypoint branch above. + fileOptions["target_size"] = []int{a.TargetSize[0], a.TargetSize[1]} } - if len(a.TargetSize) == 2 { - spec["spec"] = map[string]any{ - "file_options": map[string]any{ - // [height, width] per the schema; TargetSize is [W, H]. - "target_size": []int{a.TargetSize[1], a.TargetSize[0]}, - }, - } + if len(fileOptions) > 0 { + spec["spec"] = map[string]any{"file_options": fileOptions} } } diff --git a/internal/push/spec_test.go b/internal/push/spec_test.go index 09c6a4c..72d8a79 100644 --- a/internal/push/spec_test.go +++ b/internal/push/spec_test.go @@ -197,36 +197,29 @@ func TestBuild_NoTargetSize_OmitsSpecBlock(t *testing.T) { } } -// TestBuild_TargetSize_EmittedHeightWidth locks the [height, width] -// emit order (Bugbot, PR #22). TargetSize is stored [W, H] but the -// ingest.v1 schema + ingestor read target_size as [height, width]. -// Square sizes can't catch a swap, so this uses a NON-square -// resolution: [W, H] = [640, 480] must emit [480, 640]. -func TestBuild_TargetSize_EmittedHeightWidth(t *testing.T) { - // image_classification → under spec.file_options - icSpec := SpecArgs{ +// TestBuild_TargetSize_EmittedWidthHeight locks the [width, height] +// emission order: the ingest.v1 schema's own description says target_size +// "matches PIL.Image.size" (which is (W,H)) and ImageResolutionValidator +// compares that tuple against the list verbatim — verified empirically +// against the real validator (8×4 image passes [8,4], fails [4,8]). The +// parity harness pins this end-to-end with the non-square cases. +func TestBuild_TargetSize_EmittedWidthHeight(t *testing.T) { + // TargetSize is stored [W, H] = [640, 480] (a landscape image). + imgSpec := SpecArgs{ Table: "t", Category: "image_classification", Intent: "train", - LabelColumn: "label", TargetSize: []int{640, 480}, // [W, H] + LabelColumn: "label", TargetSize: []int{640, 480}, }.Build() - sb, ok := icSpec["spec"].(map[string]any) - if !ok { - t.Fatalf("image: no spec block: %#v", icSpec["spec"]) - } - fo, ok := sb["file_options"].(map[string]any) - if !ok { - t.Fatalf("image: no file_options: %#v", sb["file_options"]) - } - if ts, ok := fo["target_size"].([]int); !ok || len(ts) != 2 || ts[0] != 480 || ts[1] != 640 { - t.Errorf("image target_size = %#v, want [480 640] (height, width)", fo["target_size"]) + fo := imgSpec["spec"].(map[string]any)["file_options"].(map[string]any) + if ts, ok := fo["target_size"].([]int); !ok || len(ts) != 2 || ts[0] != 640 || ts[1] != 480 { + t.Errorf("image target_size = %#v, want [640 480] (width, height)", fo["target_size"]) } - // keypoint_detection → top-level kpSpec := SpecArgs{ Table: "t", Category: "keypoint_detection", Intent: "train", - LabelColumn: "image_label", TargetSize: []int{640, 480}, NumberOfKeypoints: 9, + LabelColumn: "label", TargetSize: []int{640, 480}, NumberOfKeypoints: 17, }.Build() - if ts, ok := kpSpec["target_size"].([]int); !ok || len(ts) != 2 || ts[0] != 480 || ts[1] != 640 { - t.Errorf("keypoint top-level target_size = %#v, want [480 640] (height, width)", kpSpec["target_size"]) + if ts, ok := kpSpec["target_size"].([]int); !ok || len(ts) != 2 || ts[0] != 640 || ts[1] != 480 { + t.Errorf("keypoint top-level target_size = %#v, want [640 480] (width, height)", kpSpec["target_size"]) } } @@ -446,3 +439,68 @@ func TestStagedPrefix_PanicsOnUnsafeName(t *testing.T) { }) } } + +// The cli#68 fix: Build must carry the detected extension into +// spec.file_options so the ingestor's FileTypeValidator checks the type +// that was actually staged — and the result must still schema-validate +// (the enum only allows what the ingestor's FileExtension enum allows). +func TestBuild_EmitsDetectedExtension_PassesSchema(t *testing.T) { + for _, tc := range []struct { + name string + args SpecArgs + }{ + {"image_classification with target size", SpecArgs{ + Table: "t1", Category: "image_classification", Intent: "train", + LabelColumn: "label", TargetSize: []int{512, 512}, Extension: ".png", + }}, + {"image_classification extension only", SpecArgs{ + Table: "t2", Category: "image_classification", Intent: "train", + LabelColumn: "label", Extension: ".jpg", + }}, + {"keypoint_detection keeps top-level fields", SpecArgs{ + Table: "t3", Category: "keypoint_detection", Intent: "train", + LabelColumn: "label", TargetSize: []int{256, 256}, + NumberOfKeypoints: 17, Extension: ".jpeg", + }}, + } { + t.Run(tc.name, func(t *testing.T) { + spec := tc.args.Build() + inner, _ := spec["spec"].(map[string]any) + if inner == nil { + t.Fatal("spec.file_options missing entirely") + } + fo, _ := inner["file_options"].(map[string]any) + if fo == nil || fo["extension"] != tc.args.Extension { + t.Fatalf("file_options.extension = %v, want %q (file_options=%v)", + fo["extension"], tc.args.Extension, fo) + } + if tc.args.Category == "keypoint_detection" { + if spec["number_of_keypoints"] != 17 { + t.Errorf("keypoint top-level fields must survive: %v", spec) + } + if _, hasTS := spec["target_size"]; !hasTS { + t.Error("keypoint target_size must stay present top-level") + } + if _, hasTS := fo["target_size"]; hasTS { + t.Errorf("keypoint target_size must stay top-level, not in file_options") + } + } + + specBytes, err := yaml.Marshal(spec) + if err != nil { + t.Fatalf("marshal: %v", err) + } + v, err := schema.NewV1Validator() + if err != nil { + t.Fatalf("NewV1Validator: %v", err) + } + _, errs, parseErr := v.ValidateYAML(specBytes) + if parseErr != nil { + t.Fatalf("parse error on our own output: %v\n%s", parseErr, specBytes) + } + if len(errs) != 0 { + t.Fatalf("schema rejects the emitted extension spec: %v\n%s", errs, specBytes) + } + }) + } +} diff --git a/internal/push/testdata/parity/cases.json b/internal/push/testdata/parity/cases.json new file mode 100644 index 0000000..fdc4e5d --- /dev/null +++ b/internal/push/testdata/parity/cases.json @@ -0,0 +1,286 @@ +{ + "_comment": "Drives BOTH sides of the validator-parity harness (backend#828 P3). cli_verdict = what the Go preflight must decide; ingestor_verdict = what the REAL Python validators decide (pinned by goldens.json, regenerated via scripts/gen-validator-goldens.py). The two may deliberately diverge only where the note explains why (e.g. failures the ingestor only surfaces at transfer time, which the CLI previews anyway).", + "cases": [ + { + "name": "tabular-ok", + "category": "tabular_classification", + "csv": "data.csv", + "label_column": "label", + "cli_verdict": "accept", + "ingestor_verdict": "accept" + }, + { + "name": "tabular-dup-header", + "category": "tabular_classification", + "csv": "data.csv", + "label_column": "label", + "cli_verdict": "reject", + "ingestor_verdict": "reject", + "note": "DataValidator preflight probe + read-time twin (cli#73a)" + }, + { + "name": "tabular-bom", + "category": "tabular_classification", + "csv": "data.csv", + "label_column": "label", + "cli_verdict": "reject", + "ingestor_verdict": "reject", + "note": "the stdlib header probe does not strip the BOM \u2014 false in-cluster rejection the CLI must preview (cli#71)" + }, + { + "name": "tabular-header-only", + "category": "tabular_classification", + "csv": "data.csv", + "label_column": "label", + "cli_verdict": "reject", + "ingestor_verdict": "reject", + "note": "IngestableRecordsValidator (cli#73b) (residual masking: a header-only CSV also has 0 label classes, so diversity rejects too \u2014 unavoidable for this shape)" + }, + { + "name": "tabular-label-missing", + "category": "tabular_classification", + "csv": "data.csv", + "label_column": "label", + "cli_verdict": "reject", + "ingestor_verdict": "accept", + "note": "DELIBERATE divergence: the ingestor's preflight validators pass \u2014 the failure only fires at READ time (full-schema check, after table creation, orphaning it). The CLI previews the read-time rule (cli#69)." + }, + { + "name": "imgc-ok", + "category": "image_classification", + "csv": "labels.csv", + "label_column": "label", + "extension": ".jpg", + "target_size": [ + 8, + 8 + ], + "cli_verdict": "accept", + "ingestor_verdict": "accept" + }, + { + "name": "imgc-bom-labels", + "category": "image_classification", + "csv": "labels.csv", + "label_column": "label", + "extension": ".jpg", + "target_size": [ + 8, + 8 + ], + "cli_verdict": "accept", + "ingestor_verdict": "accept", + "note": "pandas strips the BOM in-cluster \u2014 the CLI must NOT reject what the cluster accepts (cli#71 parity)" + }, + { + "name": "imgc-label-missing", + "category": "image_classification", + "csv": "labels.csv", + "label_column": "label", + "extension": ".jpg", + "target_size": [ + 8, + 8 + ], + "cli_verdict": "reject", + "ingestor_verdict": "reject", + "note": "LabelColumnValidator (cli#69)" + }, + { + "name": "imgc-label-case", + "category": "image_classification", + "csv": "labels.csv", + "label_column": "label", + "extension": ".jpg", + "target_size": [ + 8, + 8 + ], + "cli_verdict": "accept", + "ingestor_verdict": "accept", + "note": "the ingestor's _match_column is case-insensitive + trimmed \u2014 the CLI must be exactly as loose" + }, + { + "name": "imgc-zero-byte", + "category": "image_classification", + "csv": "labels.csv", + "label_column": "label", + "extension": ".jpg", + "target_size": [ + 8, + 8 + ], + "cli_verdict": "reject", + "ingestor_verdict": "reject", + "note": "ImageResolutionValidator: empty file (cli#72b)" + }, + { + "name": "imgc-corrupt", + "category": "image_classification", + "csv": "labels.csv", + "label_column": "label", + "extension": ".jpg", + "target_size": [ + 8, + 8 + ], + "cli_verdict": "reject", + "ingestor_verdict": "reject", + "note": "ImageResolutionValidator: not a valid image (cli#72b)" + }, + { + "name": "imgc-res-mismatch", + "category": "image_classification", + "csv": "labels.csv", + "label_column": "label", + "extension": ".jpg", + "target_size": [ + 8, + 8 + ], + "cli_verdict": "reject", + "ingestor_verdict": "reject", + "note": "ImageResolutionValidator: exact equality, no resize (cli#72c)" + }, + { + "name": "imgc-header-only", + "category": "image_classification", + "csv": "labels.csv", + "label_column": "label", + "extension": ".jpg", + "target_size": [ + 8, + 8 + ], + "cli_verdict": "reject", + "ingestor_verdict": "reject", + "note": "IngestableRecordsValidator (cli#73b) (residual masking: a header-only CSV also has 0 label classes, so diversity rejects too \u2014 unavoidable for this shape)" + }, + { + "name": "imgc-missing-file", + "category": "image_classification", + "csv": "labels.csv", + "label_column": "label", + "extension": ".jpg", + "target_size": [ + 8, + 8 + ], + "cli_verdict": "reject", + "ingestor_verdict": "accept", + "note": "DELIBERATE divergence: the ingestor's preflight passes \u2014 the row becomes a failed record at TRANSFER time ('completed with failures', exit 9, post-upload). The CLI previews the transfer fate (cli#72a)." + }, + { + "name": "imgc-label-uniform", + "category": "image_classification", + "csv": "labels.csv", + "label_column": "label", + "extension": ".jpg", + "target_size": [ + 8, + 8 + ], + "cli_verdict": "reject", + "ingestor_verdict": "reject", + "note": "LabelDiversityValidator: classification needs >=2 distinct labels \u2014 discovered BY this harness's first run (was not in any ticket)" + }, + { + "name": "imgc-nonsquare", + "category": "image_classification", + "csv": "labels.csv", + "label_column": "label", + "extension": ".jpg", + "target_size": [ + 8, + 4 + ], + "cli_verdict": "accept", + "ingestor_verdict": "accept", + "note": "non-square [W,H]=[8,4]: pins the target_size ORIENTATION end-to-end \u2014 an [H,W] swap on emit (the pre-P3 bug) flips this to reject in-cluster" + }, + { + "name": "imgc-nonsquare-swapped", + "category": "image_classification", + "csv": "labels.csv", + "label_column": "label", + "extension": ".jpg", + "target_size": [ + 4, + 8 + ], + "cli_verdict": "reject", + "ingestor_verdict": "reject", + "note": "the same 8\u00d74 images with target [4,8]: both sides must reject \u2014 proves the comparison is orientation-sensitive, not shape-normalized" + }, + { + "name": "imgc-dup-header", + "category": "image_classification", + "csv": "labels.csv", + "label_column": "label", + "extension": ".jpg", + "target_size": [ + 8, + 8 + ], + "cli_verdict": "reject", + "ingestor_verdict": "accept", + "note": "DELIBERATE divergence: image categories have no DataValidator, so the ingestor's preflight accepts dup headers \u2014 the rejection fires at READ time (after table creation, orphaning it). The CLI previews the read-time rule." + }, + { + "name": "imgc-empty-label", + "category": "image_classification", + "csv": "labels.csv", + "label_column": "label", + "extension": ".jpg", + "target_size": [ + 8, + 8 + ], + "cli_verdict": "accept", + "ingestor_verdict": "accept", + "note": "empty-string label IS a class for image categories (keep_default_na=False): the CLI must not be stricter" + }, + { + "name": "imgc-dotted-stem", + "category": "image_classification", + "csv": "labels.csv", + "label_column": "label", + "extension": ".jpg", + "target_size": [ + 8, + 8 + ], + "cli_verdict": "accept", + "ingestor_verdict": "accept", + "note": "row 'photo.2024' resolves to photo.2024.jpg via _has_extension (dotted stems are not extensions) \u2014 the CLI's cross-check must mirror, not reject" + }, + { + "name": "tabular-na-labels", + "category": "tabular_classification", + "csv": "data.csv", + "label_column": "label", + "cli_verdict": "reject", + "ingestor_verdict": "reject", + "note": "'NA' drops to NaN for schema-typed labels (coercion NA_SENTINELS) \u2192 1 distinct class in-cluster; the CLI mirrors the sentinel drop" + }, + { + "name": "tabular-nonutf8", + "category": "tabular_classification", + "csv": "data.csv", + "label_column": "label", + "cli_verdict": "reject", + "ingestor_verdict": "reject", + "note": "check_csv_encoding runs before ANY validator in-cluster; Latin-1 exports must fail locally, not post-upload" + }, + { + "name": "text-clf-ok", + "category": "text_classification", + "csv": "labels.csv", + "label_column": "label", + "extension": ".txt", + "cli_verdict": "accept", + "ingestor_verdict": "accept", + "note": "pins the text-family dispatch" + } + ] +} diff --git a/internal/push/testdata/parity/cases/imgc-bom-labels/images/a.jpg b/internal/push/testdata/parity/cases/imgc-bom-labels/images/a.jpg new file mode 100644 index 0000000..2c5a1e7 Binary files /dev/null and b/internal/push/testdata/parity/cases/imgc-bom-labels/images/a.jpg differ diff --git a/internal/push/testdata/parity/cases/imgc-bom-labels/images/b.jpg b/internal/push/testdata/parity/cases/imgc-bom-labels/images/b.jpg new file mode 100644 index 0000000..2c5a1e7 Binary files /dev/null and b/internal/push/testdata/parity/cases/imgc-bom-labels/images/b.jpg differ diff --git a/internal/push/testdata/parity/cases/imgc-bom-labels/labels.csv b/internal/push/testdata/parity/cases/imgc-bom-labels/labels.csv new file mode 100644 index 0000000..6b226c6 --- /dev/null +++ b/internal/push/testdata/parity/cases/imgc-bom-labels/labels.csv @@ -0,0 +1,3 @@ +image_id,label +a.jpg,cat +b.jpg,dog diff --git a/internal/push/testdata/parity/cases/imgc-corrupt/images/a.jpg b/internal/push/testdata/parity/cases/imgc-corrupt/images/a.jpg new file mode 100644 index 0000000..2c5a1e7 Binary files /dev/null and b/internal/push/testdata/parity/cases/imgc-corrupt/images/a.jpg differ diff --git a/internal/push/testdata/parity/cases/imgc-corrupt/images/c.jpg b/internal/push/testdata/parity/cases/imgc-corrupt/images/c.jpg new file mode 100644 index 0000000..c14f8d5 --- /dev/null +++ b/internal/push/testdata/parity/cases/imgc-corrupt/images/c.jpg @@ -0,0 +1 @@ +garbage-not-an-image \ No newline at end of file diff --git a/internal/push/testdata/parity/cases/imgc-corrupt/labels.csv b/internal/push/testdata/parity/cases/imgc-corrupt/labels.csv new file mode 100644 index 0000000..0f87dcf --- /dev/null +++ b/internal/push/testdata/parity/cases/imgc-corrupt/labels.csv @@ -0,0 +1,3 @@ +image_id,label +a.jpg,cat +c.jpg,dog diff --git a/internal/push/testdata/parity/cases/imgc-dotted-stem/images/b.jpg b/internal/push/testdata/parity/cases/imgc-dotted-stem/images/b.jpg new file mode 100644 index 0000000..c6b07d9 Binary files /dev/null and b/internal/push/testdata/parity/cases/imgc-dotted-stem/images/b.jpg differ diff --git a/internal/push/testdata/parity/cases/imgc-dotted-stem/images/photo.2024.jpg b/internal/push/testdata/parity/cases/imgc-dotted-stem/images/photo.2024.jpg new file mode 100644 index 0000000..c6b07d9 Binary files /dev/null and b/internal/push/testdata/parity/cases/imgc-dotted-stem/images/photo.2024.jpg differ diff --git a/internal/push/testdata/parity/cases/imgc-dotted-stem/labels.csv b/internal/push/testdata/parity/cases/imgc-dotted-stem/labels.csv new file mode 100644 index 0000000..aefc6e8 --- /dev/null +++ b/internal/push/testdata/parity/cases/imgc-dotted-stem/labels.csv @@ -0,0 +1,3 @@ +image_id,label +photo.2024,cat +b.jpg,dog diff --git a/internal/push/testdata/parity/cases/imgc-dup-header/images/a.jpg b/internal/push/testdata/parity/cases/imgc-dup-header/images/a.jpg new file mode 100644 index 0000000..c6b07d9 Binary files /dev/null and b/internal/push/testdata/parity/cases/imgc-dup-header/images/a.jpg differ diff --git a/internal/push/testdata/parity/cases/imgc-dup-header/images/b.jpg b/internal/push/testdata/parity/cases/imgc-dup-header/images/b.jpg new file mode 100644 index 0000000..c6b07d9 Binary files /dev/null and b/internal/push/testdata/parity/cases/imgc-dup-header/images/b.jpg differ diff --git a/internal/push/testdata/parity/cases/imgc-dup-header/labels.csv b/internal/push/testdata/parity/cases/imgc-dup-header/labels.csv new file mode 100644 index 0000000..a55a8f7 --- /dev/null +++ b/internal/push/testdata/parity/cases/imgc-dup-header/labels.csv @@ -0,0 +1,3 @@ +image_id,label,label +a.jpg,cat,x +b.jpg,dog,y diff --git a/internal/push/testdata/parity/cases/imgc-empty-label/images/a.jpg b/internal/push/testdata/parity/cases/imgc-empty-label/images/a.jpg new file mode 100644 index 0000000..c6b07d9 Binary files /dev/null and b/internal/push/testdata/parity/cases/imgc-empty-label/images/a.jpg differ diff --git a/internal/push/testdata/parity/cases/imgc-empty-label/images/b.jpg b/internal/push/testdata/parity/cases/imgc-empty-label/images/b.jpg new file mode 100644 index 0000000..c6b07d9 Binary files /dev/null and b/internal/push/testdata/parity/cases/imgc-empty-label/images/b.jpg differ diff --git a/internal/push/testdata/parity/cases/imgc-empty-label/labels.csv b/internal/push/testdata/parity/cases/imgc-empty-label/labels.csv new file mode 100644 index 0000000..d6de19c --- /dev/null +++ b/internal/push/testdata/parity/cases/imgc-empty-label/labels.csv @@ -0,0 +1,3 @@ +image_id,label +a.jpg,A +b.jpg, diff --git a/internal/push/testdata/parity/cases/imgc-header-only/images/a.jpg b/internal/push/testdata/parity/cases/imgc-header-only/images/a.jpg new file mode 100644 index 0000000..2c5a1e7 Binary files /dev/null and b/internal/push/testdata/parity/cases/imgc-header-only/images/a.jpg differ diff --git a/internal/push/testdata/parity/cases/imgc-header-only/labels.csv b/internal/push/testdata/parity/cases/imgc-header-only/labels.csv new file mode 100644 index 0000000..6c4b18c --- /dev/null +++ b/internal/push/testdata/parity/cases/imgc-header-only/labels.csv @@ -0,0 +1 @@ +image_id,label diff --git a/internal/push/testdata/parity/cases/imgc-label-case/images/a.jpg b/internal/push/testdata/parity/cases/imgc-label-case/images/a.jpg new file mode 100644 index 0000000..2c5a1e7 Binary files /dev/null and b/internal/push/testdata/parity/cases/imgc-label-case/images/a.jpg differ diff --git a/internal/push/testdata/parity/cases/imgc-label-case/images/b.jpg b/internal/push/testdata/parity/cases/imgc-label-case/images/b.jpg new file mode 100644 index 0000000..2c5a1e7 Binary files /dev/null and b/internal/push/testdata/parity/cases/imgc-label-case/images/b.jpg differ diff --git a/internal/push/testdata/parity/cases/imgc-label-case/labels.csv b/internal/push/testdata/parity/cases/imgc-label-case/labels.csv new file mode 100644 index 0000000..0bf13b2 --- /dev/null +++ b/internal/push/testdata/parity/cases/imgc-label-case/labels.csv @@ -0,0 +1,3 @@ +image_id, Label +a.jpg,cat +b.jpg,dog diff --git a/internal/push/testdata/parity/cases/imgc-label-missing/images/a.jpg b/internal/push/testdata/parity/cases/imgc-label-missing/images/a.jpg new file mode 100644 index 0000000..2c5a1e7 Binary files /dev/null and b/internal/push/testdata/parity/cases/imgc-label-missing/images/a.jpg differ diff --git a/internal/push/testdata/parity/cases/imgc-label-missing/labels.csv b/internal/push/testdata/parity/cases/imgc-label-missing/labels.csv new file mode 100644 index 0000000..96a2f73 --- /dev/null +++ b/internal/push/testdata/parity/cases/imgc-label-missing/labels.csv @@ -0,0 +1,2 @@ +image_id,target +a.jpg,cat diff --git a/internal/push/testdata/parity/cases/imgc-label-uniform/images/a.jpg b/internal/push/testdata/parity/cases/imgc-label-uniform/images/a.jpg new file mode 100644 index 0000000..2c5a1e7 Binary files /dev/null and b/internal/push/testdata/parity/cases/imgc-label-uniform/images/a.jpg differ diff --git a/internal/push/testdata/parity/cases/imgc-label-uniform/images/b.jpg b/internal/push/testdata/parity/cases/imgc-label-uniform/images/b.jpg new file mode 100644 index 0000000..2c5a1e7 Binary files /dev/null and b/internal/push/testdata/parity/cases/imgc-label-uniform/images/b.jpg differ diff --git a/internal/push/testdata/parity/cases/imgc-label-uniform/labels.csv b/internal/push/testdata/parity/cases/imgc-label-uniform/labels.csv new file mode 100644 index 0000000..30cd31a --- /dev/null +++ b/internal/push/testdata/parity/cases/imgc-label-uniform/labels.csv @@ -0,0 +1,3 @@ +image_id,label +a.jpg,cat +b.jpg,cat diff --git a/internal/push/testdata/parity/cases/imgc-missing-file/images/a.jpg b/internal/push/testdata/parity/cases/imgc-missing-file/images/a.jpg new file mode 100644 index 0000000..2c5a1e7 Binary files /dev/null and b/internal/push/testdata/parity/cases/imgc-missing-file/images/a.jpg differ diff --git a/internal/push/testdata/parity/cases/imgc-missing-file/labels.csv b/internal/push/testdata/parity/cases/imgc-missing-file/labels.csv new file mode 100644 index 0000000..34fda8e --- /dev/null +++ b/internal/push/testdata/parity/cases/imgc-missing-file/labels.csv @@ -0,0 +1,3 @@ +image_id,label +a.jpg,cat +ghost.jpg,dog diff --git a/internal/push/testdata/parity/cases/imgc-nonsquare-swapped/images/a.jpg b/internal/push/testdata/parity/cases/imgc-nonsquare-swapped/images/a.jpg new file mode 100644 index 0000000..4b5275b Binary files /dev/null and b/internal/push/testdata/parity/cases/imgc-nonsquare-swapped/images/a.jpg differ diff --git a/internal/push/testdata/parity/cases/imgc-nonsquare-swapped/images/b.jpg b/internal/push/testdata/parity/cases/imgc-nonsquare-swapped/images/b.jpg new file mode 100644 index 0000000..4b5275b Binary files /dev/null and b/internal/push/testdata/parity/cases/imgc-nonsquare-swapped/images/b.jpg differ diff --git a/internal/push/testdata/parity/cases/imgc-nonsquare-swapped/labels.csv b/internal/push/testdata/parity/cases/imgc-nonsquare-swapped/labels.csv new file mode 100644 index 0000000..aa98ba5 --- /dev/null +++ b/internal/push/testdata/parity/cases/imgc-nonsquare-swapped/labels.csv @@ -0,0 +1,3 @@ +image_id,label +a.jpg,cat +b.jpg,dog diff --git a/internal/push/testdata/parity/cases/imgc-nonsquare/images/a.jpg b/internal/push/testdata/parity/cases/imgc-nonsquare/images/a.jpg new file mode 100644 index 0000000..4b5275b Binary files /dev/null and b/internal/push/testdata/parity/cases/imgc-nonsquare/images/a.jpg differ diff --git a/internal/push/testdata/parity/cases/imgc-nonsquare/images/b.jpg b/internal/push/testdata/parity/cases/imgc-nonsquare/images/b.jpg new file mode 100644 index 0000000..4b5275b Binary files /dev/null and b/internal/push/testdata/parity/cases/imgc-nonsquare/images/b.jpg differ diff --git a/internal/push/testdata/parity/cases/imgc-nonsquare/labels.csv b/internal/push/testdata/parity/cases/imgc-nonsquare/labels.csv new file mode 100644 index 0000000..aa98ba5 --- /dev/null +++ b/internal/push/testdata/parity/cases/imgc-nonsquare/labels.csv @@ -0,0 +1,3 @@ +image_id,label +a.jpg,cat +b.jpg,dog diff --git a/internal/push/testdata/parity/cases/imgc-ok/images/a.jpg b/internal/push/testdata/parity/cases/imgc-ok/images/a.jpg new file mode 100644 index 0000000..2c5a1e7 Binary files /dev/null and b/internal/push/testdata/parity/cases/imgc-ok/images/a.jpg differ diff --git a/internal/push/testdata/parity/cases/imgc-ok/images/b.jpg b/internal/push/testdata/parity/cases/imgc-ok/images/b.jpg new file mode 100644 index 0000000..2c5a1e7 Binary files /dev/null and b/internal/push/testdata/parity/cases/imgc-ok/images/b.jpg differ diff --git a/internal/push/testdata/parity/cases/imgc-ok/labels.csv b/internal/push/testdata/parity/cases/imgc-ok/labels.csv new file mode 100644 index 0000000..aa98ba5 --- /dev/null +++ b/internal/push/testdata/parity/cases/imgc-ok/labels.csv @@ -0,0 +1,3 @@ +image_id,label +a.jpg,cat +b.jpg,dog diff --git a/internal/push/testdata/parity/cases/imgc-res-mismatch/images/a.jpg b/internal/push/testdata/parity/cases/imgc-res-mismatch/images/a.jpg new file mode 100644 index 0000000..2c5a1e7 Binary files /dev/null and b/internal/push/testdata/parity/cases/imgc-res-mismatch/images/a.jpg differ diff --git a/internal/push/testdata/parity/cases/imgc-res-mismatch/images/odd.jpg b/internal/push/testdata/parity/cases/imgc-res-mismatch/images/odd.jpg new file mode 100644 index 0000000..3a59e2e Binary files /dev/null and b/internal/push/testdata/parity/cases/imgc-res-mismatch/images/odd.jpg differ diff --git a/internal/push/testdata/parity/cases/imgc-res-mismatch/labels.csv b/internal/push/testdata/parity/cases/imgc-res-mismatch/labels.csv new file mode 100644 index 0000000..08c8284 --- /dev/null +++ b/internal/push/testdata/parity/cases/imgc-res-mismatch/labels.csv @@ -0,0 +1,3 @@ +image_id,label +a.jpg,cat +odd.jpg,dog diff --git a/internal/push/testdata/parity/cases/imgc-zero-byte/images/a.jpg b/internal/push/testdata/parity/cases/imgc-zero-byte/images/a.jpg new file mode 100644 index 0000000..2c5a1e7 Binary files /dev/null and b/internal/push/testdata/parity/cases/imgc-zero-byte/images/a.jpg differ diff --git a/internal/push/testdata/parity/cases/imgc-zero-byte/images/z.jpg b/internal/push/testdata/parity/cases/imgc-zero-byte/images/z.jpg new file mode 100644 index 0000000..e69de29 diff --git a/internal/push/testdata/parity/cases/imgc-zero-byte/labels.csv b/internal/push/testdata/parity/cases/imgc-zero-byte/labels.csv new file mode 100644 index 0000000..d37fca0 --- /dev/null +++ b/internal/push/testdata/parity/cases/imgc-zero-byte/labels.csv @@ -0,0 +1,3 @@ +image_id,label +a.jpg,cat +z.jpg,dog diff --git a/internal/push/testdata/parity/cases/tabular-bom/data.csv b/internal/push/testdata/parity/cases/tabular-bom/data.csv new file mode 100644 index 0000000..5c76b18 --- /dev/null +++ b/internal/push/testdata/parity/cases/tabular-bom/data.csv @@ -0,0 +1,3 @@ +age,income,label +30,100,1 +40,200,0 diff --git a/internal/push/testdata/parity/cases/tabular-dup-header/data.csv b/internal/push/testdata/parity/cases/tabular-dup-header/data.csv new file mode 100644 index 0000000..b92f2f9 --- /dev/null +++ b/internal/push/testdata/parity/cases/tabular-dup-header/data.csv @@ -0,0 +1,3 @@ +age,income,age,label +1,2,3,x +4,5,6,y diff --git a/internal/push/testdata/parity/cases/tabular-header-only/data.csv b/internal/push/testdata/parity/cases/tabular-header-only/data.csv new file mode 100644 index 0000000..70ec026 --- /dev/null +++ b/internal/push/testdata/parity/cases/tabular-header-only/data.csv @@ -0,0 +1 @@ +age,income,label diff --git a/internal/push/testdata/parity/cases/tabular-label-missing/data.csv b/internal/push/testdata/parity/cases/tabular-label-missing/data.csv new file mode 100644 index 0000000..4f59854 --- /dev/null +++ b/internal/push/testdata/parity/cases/tabular-label-missing/data.csv @@ -0,0 +1,2 @@ +age,income +30,100 diff --git a/internal/push/testdata/parity/cases/tabular-na-labels/data.csv b/internal/push/testdata/parity/cases/tabular-na-labels/data.csv new file mode 100644 index 0000000..01fe6f8 --- /dev/null +++ b/internal/push/testdata/parity/cases/tabular-na-labels/data.csv @@ -0,0 +1,3 @@ +age,label +30,NA +40,x diff --git a/internal/push/testdata/parity/cases/tabular-nonutf8/data.csv b/internal/push/testdata/parity/cases/tabular-nonutf8/data.csv new file mode 100644 index 0000000..31e4f05 --- /dev/null +++ b/internal/push/testdata/parity/cases/tabular-nonutf8/data.csv @@ -0,0 +1,3 @@ +age,city,label +30,Mnchen,1 +40,Berlin,0 diff --git a/internal/push/testdata/parity/cases/tabular-ok/data.csv b/internal/push/testdata/parity/cases/tabular-ok/data.csv new file mode 100644 index 0000000..e7096ec --- /dev/null +++ b/internal/push/testdata/parity/cases/tabular-ok/data.csv @@ -0,0 +1,3 @@ +age,income,label +30,100,1 +40,200,0 diff --git a/internal/push/testdata/parity/cases/text-clf-ok/labels.csv b/internal/push/testdata/parity/cases/text-clf-ok/labels.csv new file mode 100644 index 0000000..ea6020a --- /dev/null +++ b/internal/push/testdata/parity/cases/text-clf-ok/labels.csv @@ -0,0 +1,3 @@ +filename,label +a.txt,pos +b.txt,neg diff --git a/internal/push/testdata/parity/cases/text-clf-ok/texts/a.txt b/internal/push/testdata/parity/cases/text-clf-ok/texts/a.txt new file mode 100644 index 0000000..0cf5ead --- /dev/null +++ b/internal/push/testdata/parity/cases/text-clf-ok/texts/a.txt @@ -0,0 +1 @@ +great product, would buy again \ No newline at end of file diff --git a/internal/push/testdata/parity/cases/text-clf-ok/texts/b.txt b/internal/push/testdata/parity/cases/text-clf-ok/texts/b.txt new file mode 100644 index 0000000..7244ad7 --- /dev/null +++ b/internal/push/testdata/parity/cases/text-clf-ok/texts/b.txt @@ -0,0 +1 @@ +terrible, do not want \ No newline at end of file diff --git a/internal/push/testdata/parity/goldens.json b/internal/push/testdata/parity/goldens.json new file mode 100644 index 0000000..033a73b --- /dev/null +++ b/internal/push/testdata/parity/goldens.json @@ -0,0 +1,127 @@ +{ + "_comment": "GENERATED by scripts/gen-validator-goldens.py from the REAL data-ingestors validators \u2014 do not hand-edit. Regenerate when the ingestor's rules change; parity_golden_test.go pins these.", + "verdicts": { + "imgc-bom-labels": { + "errors": [], + "verdict": "accept" + }, + "imgc-corrupt": { + "errors": [ + "ImageResolutionValidator: Files that could not be processed: ['images/c.jpg: not a valid image (corrupt or unsupported format)']" + ], + "verdict": "reject" + }, + "imgc-dotted-stem": { + "errors": [], + "verdict": "accept" + }, + "imgc-dup-header": { + "errors": [], + "verdict": "accept" + }, + "imgc-empty-label": { + "errors": [], + "verdict": "accept" + }, + "imgc-header-only": { + "errors": [ + "IngestableRecordsValidator: No data rows found in CSV 'labels.csv': the file has a header but no data rows (0 ingestable records). Add at least one data row and re-ingest." + ], + "verdict": "reject" + }, + "imgc-label-case": { + "errors": [], + "verdict": "accept" + }, + "imgc-label-missing": { + "errors": [ + "LabelColumnValidator: Configured label column 'label' not found in CSV header (columns: image_id, target). Set 'label:' in the ingest config to an existing column, or add the 'label' column to the CSV." + ], + "verdict": "reject" + }, + "imgc-label-uniform": { + "errors": [ + "LabelDiversityValidator: Classification category requires at least 2 distinct label values in column 'label' (after whitespace stripping); this dataset has 1 distinct value(s): ['cat']. Raw value counts: {'cat': 2}. If this is intentional (e.g. you have a continuous target), pick a regression-family category like tabular_regression or time_series_forecasting instead." + ], + "verdict": "reject" + }, + "imgc-missing-file": { + "errors": [], + "verdict": "accept" + }, + "imgc-nonsquare": { + "errors": [], + "verdict": "accept" + }, + "imgc-nonsquare-swapped": { + "errors": [ + "ImageResolutionValidator: Images with incorrect resolution found: ['images/b.jpg: (8, 4) (expected: [4, 8])', 'images/a.jpg: (8, 4) (expected: [4, 8])']" + ], + "verdict": "reject" + }, + "imgc-ok": { + "errors": [], + "verdict": "accept" + }, + "imgc-res-mismatch": { + "errors": [ + "ImageResolutionValidator: Multiple image resolutions found: [(4, 4), (8, 8)]. All images must have the same resolution.", + "ImageResolutionValidator: Expected resolution: [8, 8]", + "ImageResolutionValidator: Invalid files: []", + "ImageResolutionValidator: Resolution errors: ['images/odd.jpg: (4, 4) (expected: [8, 8])']" + ], + "verdict": "reject" + }, + "imgc-zero-byte": { + "errors": [ + "ImageResolutionValidator: Files that could not be processed: ['images/z.jpg: empty file (0 bytes)']" + ], + "verdict": "reject" + }, + "tabular-bom": { + "errors": [ + "DataValidator: Schema columns not present in CSV: age." + ], + "verdict": "reject" + }, + "tabular-dup-header": { + "errors": [ + "DataValidator: Duplicate column name(s) in the CSV header: ['age']. Each column must be unique \u2014 otherwise the second is silently renamed '.1' by the parser and the schema maps onto the wrong column." + ], + "verdict": "reject" + }, + "tabular-header-only": { + "errors": [ + "IngestableRecordsValidator: No data rows found in CSV 'data.csv': the file has a header but no data rows (0 ingestable records). Add at least one data row and re-ingest.", + "DataValidator: No data found to validate" + ], + "verdict": "reject" + }, + "tabular-label-missing": { + "errors": [], + "verdict": "accept" + }, + "tabular-na-labels": { + "errors": [ + "LabelDiversityValidator: Classification category requires at least 2 distinct label values in column 'label' (after whitespace stripping); this dataset has 1 distinct value(s): ['x']. Raw value counts: {'x': 1}. If this is intentional (e.g. you have a continuous target), pick a regression-family category like tabular_regression or time_series_forecasting instead." + ], + "verdict": "reject" + }, + "tabular-nonutf8": { + "errors": [ + "csv-encoding: \u001b[91m'data.csv' is not valid UTF-8 \u2014 a non-UTF-8 byte was found at byte 19. Re-save the file as UTF-8 (in Excel: Save As \u2192 'CSV UTF-8 (Comma delimited)'), then re-ingest.\u001b[0m", + "IngestableRecordsValidator: Ingestable records validation error: 'utf-8' codec can't decode byte 0xfc in position 1: invalid start byte", + "DataValidator: Data type validation error: 'utf-8' codec can't decode byte 0xfc in position 1: invalid start byte" + ], + "verdict": "reject" + }, + "tabular-ok": { + "errors": [], + "verdict": "accept" + }, + "text-clf-ok": { + "errors": [], + "verdict": "accept" + } + } +} diff --git a/internal/push/walk.go b/internal/push/walk.go index a6bb163..9801761 100644 --- a/internal/push/walk.go +++ b/internal/push/walk.go @@ -5,6 +5,7 @@ import ( "fmt" "os" "path/filepath" + "sort" "strings" ) @@ -86,12 +87,15 @@ func (l *LocalLayout) FileCount() int { return n } -// imageExtensions accepts the file types the chart's -// image_classification ingestor processes by default. From -// data-ingestors' FileTypeValidator(images) defaults: .jpg, .jpeg, -// .png. The chart's defaults file (see chartversion 1.3.5+) also -// accepts .webp; we mirror that here so customers on a recent -// chart can stage webp files without hitting "no images found." +// imageExtensions accepts the file types the in-cluster ingestor can +// actually validate: data-ingestors' FileExtension enum + the +// ingest.v1 schema's file_options.extension enum both allow exactly +// .jpg, .jpeg, and .png for images. (.webp was listed here previously +// on the strength of a chart-defaults comment, but the ingestor's +// FileTypeValidator REJECTS it at construction — accepting it locally +// meant a full upload followed by a guaranteed in-cluster failure. +// cli#68 / backend#828 P5: this list and the emitted +// file_options.extension keep the two sides in lock-step.) // // Comparison is case-insensitive — filesystems vary (case-sensitive // on Linux, case-preserving-but-insensitive on macOS default APFS). @@ -99,14 +103,13 @@ var imageExtensions = map[string]struct{}{ ".jpg": {}, ".jpeg": {}, ".png": {}, - ".webp": {}, } // Discover walks rootDir and validates it matches the layout Phase 3 // expects for image_classification: // // - /labels.csv (required) -// - /images/*.{jpg,jpeg,png,webp} (at least one file) +// - /images/*.{jpg,jpeg,png} (at least one file, one type) // // Returns specific errors keyed to the layout mistakes a customer // is most likely to hit — these surface as the CLI's diagnostic @@ -152,8 +155,7 @@ func Discover(rootDir string) (*LocalLayout, error) { return nil, fmt.Errorf( "missing labels.csv in %q. The CLI expects "+ "/labels.csv + /images/ for image_classification; "+ - "see https://github.com/tracebloc/client/issues/147 for the "+ - "v0.1 layout contract.", + "see https://docs.tracebloc.io for the dataset layout.", abs) } return nil, fmt.Errorf("stat labels.csv: %w", err) @@ -191,7 +193,7 @@ func Discover(rootDir string) (*LocalLayout, error) { if errors.Is(err, os.ErrNotExist) { return nil, fmt.Errorf( "missing images/ subdirectory in %q. The CLI expects "+ - "/labels.csv + /images/*.{jpg,jpeg,png,webp}.", + "/labels.csv + /images/*.{jpg,jpeg,png}.", abs) } return nil, fmt.Errorf("stat images/: %w", err) @@ -211,6 +213,7 @@ func Discover(rootDir string) (*LocalLayout, error) { if err != nil { return nil, fmt.Errorf("reading images/: %w", err) } + skippedExts := map[string]int{} for _, entry := range entries { if entry.IsDir() { // Silently skip subdirectories so a stray .DS_Store or @@ -222,6 +225,9 @@ func Discover(rootDir string) (*LocalLayout, error) { } ext := strings.ToLower(filepath.Ext(entry.Name())) if _, ok := imageExtensions[ext]; !ok { + if ext != "" { + skippedExts[ext]++ + } continue } // entry.Info() returns Lstat-like metadata for the @@ -246,8 +252,15 @@ func Discover(rootDir string) (*LocalLayout, error) { } if len(layout.Images) == 0 { + if len(skippedExts) > 0 { + return nil, fmt.Errorf( + "no usable image files in %q — found %s, but the ingestor "+ + "accepts only .jpg, .jpeg, or .png. Convert the images "+ + "and re-run.", + imagesDir, extCounts(skippedExts)) + } return nil, fmt.Errorf( - "no image files found in %q. Expected .jpg, .jpeg, .png, or .webp; "+ + "no image files found in %q. Expected .jpg, .jpeg, or .png; "+ "got %d non-image entries.", imagesDir, len(entries)) } @@ -348,3 +361,45 @@ func HumanBytes(n int64) string { return fmt.Sprintf("%d B", n) } } + +// DetectExtension returns the single file extension (lowercased, with +// dot) shared by every image in the set. The in-cluster ingestor's +// FileTypeValidator enforces ONE extension per dataset, so a mixed set +// would pass local preflight and then fail in-cluster after the full +// upload — the exact cli#68 failure class. Detecting and emitting it +// here (spec.file_options.extension) keeps the preflight's promise: +// if this passes, the ingestor's file-type check passes too. +func DetectExtension(images []string) (string, error) { + if len(images) == 0 { + return "", fmt.Errorf("no image files to detect a type from") + } + counts := map[string]int{} + for _, img := range images { + counts[strings.ToLower(filepath.Ext(img))]++ + } + if len(counts) == 1 { + for ext := range counts { + return ext, nil + } + } + return "", fmt.Errorf( + "your images mix file types (%s) — the ingestor requires one type "+ + "per dataset. Convert them to a single type, or split them into "+ + "separate tables.", + extCounts(counts)) +} + +// extCounts renders {".png": 3, ".jpg": 120} as ".jpg ×120, .png ×3" +// (sorted for stable output). +func extCounts(counts map[string]int) string { + exts := make([]string, 0, len(counts)) + for ext := range counts { + exts = append(exts, ext) + } + sort.Strings(exts) + parts := make([]string, 0, len(exts)) + for _, ext := range exts { + parts = append(parts, fmt.Sprintf("%s ×%d", ext, counts[ext])) + } + return strings.Join(parts, ", ") +} diff --git a/internal/push/walk_test.go b/internal/push/walk_test.go index 441400c..114cbe8 100644 --- a/internal/push/walk_test.go +++ b/internal/push/walk_test.go @@ -72,18 +72,26 @@ func TestDiscover_TotalBytesSum(t *testing.T) { } } -func TestDiscover_AcceptsAllImageExtensions(t *testing.T) { - // Mirror the chart's FileTypeValidator(images) defaults — if a - // customer's image-set has .png + .webp, both should stage. +func TestDiscover_AcceptsIngestorExtensions(t *testing.T) { + // The accept-set mirrors what the in-cluster ingestor can actually + // validate: .jpg/.jpeg/.png (case-insensitive). .webp was accepted + // here historically on a chart-comment claim the ingestor never + // honored — staging it guaranteed an in-cluster failure after the + // full upload (cli#68), so it is now deliberately skipped. root := imgcDir(t, "a.jpg", "b.jpeg", "c.png", "d.webp", "e.JPG") got, err := Discover(root) if err != nil { t.Fatalf("Discover: %v", err) } - if len(got.Images) != 5 { - t.Errorf("len(Images) = %d, want 5 (case-insensitive); names=%v", + if len(got.Images) != 4 { + t.Errorf("len(Images) = %d, want 4 (webp skipped, case-insensitive); names=%v", len(got.Images), got.Images) } + for _, img := range got.Images { + if strings.HasSuffix(img, ".webp") { + t.Errorf("webp staged despite the ingestor rejecting it: %s", img) + } + } } func TestDiscover_SkipsNonImageFiles(t *testing.T) { @@ -161,8 +169,12 @@ func TestDiscover_NoAcceptedImageExtensions(t *testing.T) { if err == nil { t.Fatal("Discover returned nil error; expected no-images error") } - if !strings.Contains(err.Error(), "no image files") { - t.Errorf("error = %q, want it to mention no image files", err) + // The error must name what WAS found and what's accepted — the + // customer's fix is one conversion away. + for _, want := range []string{"no usable image files", ".gif", ".bmp", ".jpg, .jpeg, or .png"} { + if !strings.Contains(err.Error(), want) { + t.Errorf("error = %q, want it to mention %q", err, want) + } } } @@ -364,3 +376,23 @@ func TestHumanBytes(t *testing.T) { } } } + +// DetectExtension backs the cli#68 fix: the spec must tell the cluster the +// ONE extension the dataset actually uses, and mixed sets must fail locally +// (before upload), not in-cluster (after). +func TestDetectExtension(t *testing.T) { + ext, err := DetectExtension([]string{"a/x.jpg", "a/y.JPG", "a/z.jpg"}) + if err != nil || ext != ".jpg" { + t.Errorf("uniform: ext=%q err=%v, want .jpg/nil", ext, err) + } + + _, err = DetectExtension([]string{"a/x.jpg", "a/y.png", "a/z.jpg"}) + if err == nil { + t.Fatal("mixed extensions must error before any upload") + } + for _, want := range []string{".jpg ×2", ".png ×1", "one type"} { + if !strings.Contains(err.Error(), want) { + t.Errorf("mixed error = %q, want it to mention %q", err, want) + } + } +} diff --git a/internal/ui/ui.go b/internal/ui/ui.go index 7821fe8..e55b35b 100644 --- a/internal/ui/ui.go +++ b/internal/ui/ui.go @@ -14,7 +14,9 @@ import ( "fmt" "io" "os" + "runtime" "strings" + "time" "github.com/fatih/color" "golang.org/x/term" @@ -210,3 +212,97 @@ func (p *Printer) Section(title string) { func (p *Printer) Field(label, value string) { p.out(" %s %s\n", p.paint(fmt.Sprintf("%-14s", label+":"), color.Faint), value) } + +// Action prints an imperative instruction row — a bold verb label and its value, +// with no trailing colon: " Open https://…". Used for the device-flow +// sign-in steps (Open the URL / Enter the code), where the label is a thing to +// DO, not a field to read (contrast Field, which is dim + colon-terminated). +func (p *Printer) Action(label, value string) { + p.out(" %s %s\n", p.paint(fmt.Sprintf("%-5s", label), color.Bold), value) +} + +// spinnerFrames are braille cells; cycled, they read as a smooth rotation and +// match the ⠴ vocabulary the tracebloc/client installer already uses. +var spinnerFrames = []string{"⠋", "⠙", "⠹", "⠸", "⠼", "⠴", "⠦", "⠧", "⠇", "⠏"} + +// Spinner is a single self-redrawing status line for a blocking wait: a braille +// frame, a message, and mm:ss elapsed. Start it with Printer.Spinner and end it +// with Stop, which clears the animated line so the caller can print a final ✔/✖ +// in its place. +// +// It animates ONLY when the Printer colorizes (a real terminal, no --plain, no +// NO_COLOR). Otherwise — piped output, --plain, CI — it prints one static line +// and never redraws, so logs stay free of `\r`/ANSI. Concurrency: the spinner +// owns the writer between Spinner() and Stop(); the caller must not print in +// between, and Stop must be called from the same goroutine. +type Spinner struct { + p *Printer + msg string + hint string // optional trailing "(hint)", e.g. "Ctrl-C to cancel" + start time.Time + stop chan struct{} + done chan struct{} +} + +// Spinner starts a live wait line reading " M:SS (hint)". Pass +// hint == "" to omit the parenthetical. See the Spinner type for the +// animate-vs-static contract; the returned handle is always safe to Stop. +func (p *Printer) Spinner(msg, hint string) *Spinner { + s := &Spinner{p: p, msg: msg, hint: hint, start: time.Now()} + // Static line (no redraw) when we can't animate cleanly: non-color/non-TTY, or + // Windows — the redraw writes raw \r\033[K + SGR straight to the writer, which + // fatih/color's Windows VT-enable path never sees, so legacy consoles would show + // escape garbage every tick. Windows gets the clean one-liner instead. + if !p.color || runtime.GOOS == "windows" { + p.out(" %s %s\n", "·", msg) + return s + } + s.stop = make(chan struct{}) + s.done = make(chan struct{}) + go s.run() + return s +} + +func (s *Spinner) run() { + defer close(s.done) + t := time.NewTicker(120 * time.Millisecond) + defer t.Stop() + frame := 0 + s.draw(spinnerFrames[0]) + for { + select { + case <-s.stop: + return + case <-t.C: + frame = (frame + 1) % len(spinnerFrames) + s.draw(spinnerFrames[frame]) + } + } +} + +func (s *Spinner) draw(frame string) { + elapsed := time.Since(s.start) + mm := int(elapsed / time.Minute) + ss := int(elapsed/time.Second) % 60 + hint := "" + if s.hint != "" { + hint = " (" + s.hint + ")" + } + // \r returns to column 0; \033[K clears to end-of-line so a line that shrank + // (e.g. 10:00 → 9:59 never does, but the hint/msg could) leaves no residue. + s.p.out("\r\033[K %s %s %d:%02d%s", + s.p.paint(frame, color.FgCyan), s.msg, mm, ss, s.p.paint(hint, color.Faint)) +} + +// Stop ends the animation and clears the spinner line (animated mode); on a +// static spinner it's a no-op. Idempotent. After Stop the cursor sits at column +// 0 of a cleared line, so the caller's next Successf/Errorf prints in its place. +func (s *Spinner) Stop() { + if s.stop == nil { + return // static, or already stopped + } + close(s.stop) + <-s.done + s.stop = nil + s.p.out("\r\033[K") // clear the line; the caller prints the outcome next +} diff --git a/internal/ui/ui_test.go b/internal/ui/ui_test.go index e605a45..2325647 100644 --- a/internal/ui/ui_test.go +++ b/internal/ui/ui_test.go @@ -4,6 +4,7 @@ import ( "bytes" "strings" "testing" + "time" ) // esc is the ANSI SGR prefix every color escape starts with. Its @@ -98,3 +99,64 @@ func TestNoColorEnv_DefaultsPlain(t *testing.T) { t.Errorf("NO_COLOR set but ANSI emitted: %q", buf.String()) } } + +// TestAction_Plain: the imperative Open/Enter rows surface label + value with no +// ANSI when color is off. +func TestAction_Plain(t *testing.T) { + var buf bytes.Buffer + p := New(&buf, WithColor(false)) + p.Action("Open", "https://x/activate") + p.Action("Enter", "WDJB-MJHT") + + out := buf.String() + if strings.Contains(out, esc) { + t.Errorf("Action emitted ANSI with color off: %q", out) + } + for _, want := range []string{"Open", "https://x/activate", "Enter", "WDJB-MJHT"} { + if !strings.Contains(out, want) { + t.Errorf("Action output missing %q: %q", want, out) + } + } +} + +// TestSpinner_StaticWhenPlain: on a non-animating Printer the spinner prints ONE +// static line — no carriage return, no ANSI — and Stop is a no-op. +func TestSpinner_StaticWhenPlain(t *testing.T) { + var buf bytes.Buffer + p := New(&buf, WithColor(false)) + s := p.Spinner("Waiting for your browser…", "Ctrl-C to cancel") + s.Stop() + + out := buf.String() + if strings.Contains(out, "\r") || strings.Contains(out, esc) { + t.Errorf("static spinner emitted \\r or ANSI: %q", out) + } + if !strings.Contains(out, "Waiting for your browser…") { + t.Errorf("static spinner missing message: %q", out) + } + if n := strings.Count(out, "\n"); n != 1 { + t.Errorf("static spinner should print exactly one line, got %d: %q", n, out) + } +} + +// TestSpinner_DrawFormatsElapsed exercises the animated redraw directly (without +// the goroutine, to stay race-free): it renders mm:ss elapsed, the message, and +// the \r + clear-to-EOL control so the line self-overwrites. +func TestSpinner_DrawFormatsElapsed(t *testing.T) { + var buf bytes.Buffer + p := New(&buf, WithColor(true)) + s := &Spinner{p: p, msg: "Waiting for your browser…", hint: "Ctrl-C to cancel", + start: time.Now().Add(-65 * time.Second)} + s.draw(spinnerFrames[0]) + + out := buf.String() + if !strings.HasPrefix(out, "\r\x1b[K") { + t.Errorf("draw should start with a carriage return + clear-to-EOL, got: %q", out) + } + if !strings.Contains(out, "1:05") { + t.Errorf("draw should render 65s as 1:05, got: %q", out) + } + if !strings.Contains(out, "Waiting for your browser…") { + t.Errorf("draw missing message: %q", out) + } +} diff --git a/scripts/gen-validator-goldens.py b/scripts/gen-validator-goldens.py new file mode 100644 index 0000000..5634e51 --- /dev/null +++ b/scripts/gen-validator-goldens.py @@ -0,0 +1,134 @@ +#!/usr/bin/env python3 +"""Regenerate internal/push/testdata/parity/goldens.json by running the REAL +data-ingestors validators over the parity fixture cases. + +This is the Python half of the validator-parity harness (backend#828 P3): +the CLI's local preflight (internal/push/preflight.go) previews the +in-cluster validators, and parity_golden_test.go pins that the two sides +agree case-by-case. When the ingestor's rules change, re-run this and +commit the diff — a changed verdict then fails the Go test until the +preview is consciously updated. + +Usage: + DATA_INGESTORS_DIR=~/path/to/data-ingestors python3 scripts/gen-validator-goldens.py + +DATA_INGESTORS_DIR must point at a checkout of tracebloc/data-ingestors +with its dependencies importable (its own .venv works: + ~/repos/data-ingestors/.venv/bin/python scripts/gen-validator-goldens.py +also does the trick — the script only needs pandas + Pillow, no MySQL). + +Verdicts are recorded at the accept/reject level, not message-text level — +error copy may drift harmlessly; verdicts may not. TableNameValidator and +DuplicateValidator are skipped (they check cluster-side state — the table +name is validated separately by both sides, and destination-duplicate +handling is the cli#70 guard's territory, not a data-hygiene rule). +""" + +import json +import os +import re +import sys + +HERE = os.path.dirname(os.path.abspath(__file__)) +PARITY = os.path.join(HERE, "..", "internal", "push", "testdata", "parity") + +di = os.environ.get("DATA_INGESTORS_DIR", "") +if di: + sys.path.insert(0, os.path.abspath(di)) + +try: + from tracebloc_ingestor.utils.validators_mapping import map_validators + from tracebloc_ingestor.config import Config + from tracebloc_ingestor.ingestors import preflight +except ImportError as exc: # pragma: no cover + sys.exit( + f"can't import the data-ingestors package ({exc}).\n" + "Set DATA_INGESTORS_DIR to a checkout, or run with its venv python." + ) + +SKIP = {"TableNameValidator", "DuplicateValidator"} +ANSI = re.compile(r"\x1b\[[0-9;]*m") + + +def infer_schema(csv_path): + """Mirror the CLI's inference shape: every trimmed header column typed — + the schema content doesn't matter for these validators, membership does. + Read with pandas (BOM-stripping) exactly like the CLI's header reader.""" + import pandas as pd + + cols = list(pd.read_csv(csv_path, nrows=0, encoding="utf-8").columns) + return {str(c).strip(): "VARCHAR(255)" for c in cols} + + +def run_case(case): + case_dir = os.path.join(PARITY, "cases", case["name"]) + csv_path = os.path.join(case_dir, case["csv"]) + + if True: # (kept indented for a small diff; no tempdir needed — DuplicateValidator is skipped) + cfg = Config(SRC_PATH=case_dir, TABLE_NAME="parity_t") + options = {"label_column": case.get("label_column", "label")} + if case.get("extension"): + options["extension"] = case["extension"] + if case.get("target_size"): + options["target_size"] = case["target_size"] + if case["category"].startswith(("tabular", "time_")): + try: + schema = infer_schema(csv_path) + except Exception: + schema = {} + options["schema"] = schema + options["full_schema"] = schema + + errors = [] + try: + preflight.check_csv_encoding(csv_path) + except Exception as exc: + errors.append(f"csv-encoding: {exc}") + + for v in map_validators(case["category"], options, cfg): + if type(v).__name__ in SKIP: + continue + try: + res = v.validate(csv_path) + if not res.is_valid: + errors.extend( + f"{type(v).__name__}: {ANSI.sub('', str(e)).replace(case_dir + os.sep, '')}" + for e in res.errors + ) + except Exception as exc: # a raising validator is a rejection too + errors.append(f"{type(v).__name__}: raised {exc}") + + return { + "verdict": "reject" if errors else "accept", + "errors": errors[:6], + } + + +def main(): + with open(os.path.join(PARITY, "cases.json")) as f: + manifest = json.load(f) + + goldens = {} + for case in manifest["cases"]: + goldens[case["name"]] = run_case(case) + print(f" {case['name']:24s} → {goldens[case['name']]['verdict']}") + + out = os.path.join(PARITY, "goldens.json") + with open(out, "w") as f: + json.dump( + { + "_comment": "GENERATED by scripts/gen-validator-goldens.py from the REAL " + "data-ingestors validators — do not hand-edit. Regenerate when the " + "ingestor's rules change; parity_golden_test.go pins these.", + "verdicts": goldens, + }, + f, + indent=2, + sort_keys=True, + ) + f.write("\n") + print(f"wrote {out}") + + +if __name__ == "__main__": + main() diff --git a/scripts/install.ps1 b/scripts/install.ps1 index 85d0f6f..ab8f84b 100644 --- a/scripts/install.ps1 +++ b/scripts/install.ps1 @@ -185,8 +185,27 @@ try { $target = Join-Path $InstallPrefix $BinaryName Move-Item -Path (Join-Path $tmpDir $binaryFile) -Destination $target -Force + # Short alias: `tb` — a cmd shim next to the exe (symlinks need admin or + # dev-mode on Windows). Best-effort: an alias must never fail the install + # (mirrors install.sh), and we never clobber an unrelated tb.cmd — "ours" + # means it invokes exactly our binary path, not merely mentions the name. + $tbShim = Join-Path $InstallPrefix 'tb.cmd' + $shimBody = "@echo off`r`n`"$target`" %*`r`n" + $tbNote = '' + $tbExisting = if (Test-Path $tbShim) { Get-Content $tbShim -Raw -ErrorAction SilentlyContinue } else { $null } + if (-not (Test-Path $tbShim) -or ($tbExisting -like ('*"' + $target + '"*'))) { + try { + Set-Content -Path $tbShim -Value $shimBody -Encoding ascii -ErrorAction Stop + $tbNote = ' (short alias: tb)' + } catch { + Write-Host "Note: couldn't create the tb alias ($($_.Exception.Message)) — skipping." + } + } else { + Write-Host "Note: $tbShim already exists and isn't ours — skipping the tb alias." + } + Write-Host "" - Write-Host "✓ tracebloc CLI installed: $target" + Write-Host "✓ tracebloc CLI installed: $target$tbNote" Write-Host "" Write-Host "Verify with:" Write-Host " $target version" @@ -214,8 +233,10 @@ try { Write-Host "" Write-Host "First steps:" - Write-Host " tracebloc cluster info # confirm CLI can reach your cluster" - Write-Host " tracebloc dataset push --help # see the dominant flow" + Write-Host " tracebloc cluster info # confirm the CLI can reach your cluster" + Write-Host " tracebloc data ingest --help # stage a dataset onto the cluster" + Write-Host "" + Write-Host "Short alias: tb works everywhere tracebloc does (tb data ingest .\data)" } finally { # Always clean up the temp dir, even on early exit / Ctrl-C. diff --git a/scripts/install.sh b/scripts/install.sh index b76b8ba..ab6cdbe 100755 --- a/scripts/install.sh +++ b/scripts/install.sh @@ -399,8 +399,23 @@ fi chmod +x "$TMP/$BINARY_FILE" mv "$TMP/$BINARY_FILE" "$PREFIX/$BINARY_NAME" +# Short alias: `tb` — the kubectl→k pattern for the most-typed word in the +# product. A symlink next to the binary, created only when the name is free +# (or already ours): never clobber an unrelated tool the user has as `tb`. +TB_ALIAS_NOTE="" +if command -v ln >/dev/null 2>&1; then + if [ ! -e "$PREFIX/tb" ] || [ "$(readlink "$PREFIX/tb" 2>/dev/null)" = "$PREFIX/$BINARY_NAME" ]; then + # Best-effort: an alias must never be able to fail the install. + if ln -sf "$PREFIX/$BINARY_NAME" "$PREFIX/tb" 2>/dev/null; then + TB_ALIAS_NOTE=" (short alias: tb)" + fi + else + echo "Note: $PREFIX/tb already exists and isn't ours — skipping the tb alias." + fi +fi + echo "" -echo "✓ tracebloc CLI installed: $PREFIX/$BINARY_NAME" +echo "✓ tracebloc CLI installed: $PREFIX/$BINARY_NAME$TB_ALIAS_NOTE" echo "" echo "Verify with:" echo " $PREFIX/$BINARY_NAME version" @@ -492,5 +507,7 @@ if [ "$persist" = "yes" ]; then fi echo "First steps:" -echo " $BINARY_NAME cluster info # confirm CLI can reach your cluster" -echo " $BINARY_NAME dataset push --help # see the dominant flow" +echo " $BINARY_NAME cluster info # confirm the CLI can reach your cluster" +echo " $BINARY_NAME data ingest --help # stage a dataset onto the cluster" +echo "" +echo "Short alias: tb works everywhere tracebloc does (tb data ingest ./data)" diff --git a/scripts/sync-validator-goldens.sh b/scripts/sync-validator-goldens.sh new file mode 100755 index 0000000..8cfc02a --- /dev/null +++ b/scripts/sync-validator-goldens.sh @@ -0,0 +1,37 @@ +#!/usr/bin/env bash +# Regenerate (or --check) the validator-parity goldens against the REAL +# data-ingestors validators. Mirrors sync-schema.sh's shape. +# +# DATA_INGESTORS_DIR=~/repos/data-ingestors scripts/sync-validator-goldens.sh [--check] +# +# Needs a data-ingestors checkout with pandas + Pillow importable — its own +# .venv works: point PYTHON at it, e.g. +# PYTHON="$DATA_INGESTORS_DIR/.venv/bin/python" scripts/sync-validator-goldens.sh --check +set -euo pipefail +cd "$(dirname "$0")/.." +GOLDENS="internal/push/testdata/parity/goldens.json" +PYTHON="${PYTHON:-python3}" +if [[ "${1:-}" == "--check" ]]; then + tmp="$(mktemp -d)" + trap 'rm -rf "$tmp"' EXIT + cp "$GOLDENS" "$tmp/committed.json" + "$PYTHON" scripts/gen-validator-goldens.py >/dev/null + # Compare VERDICTS only — error text may drift harmlessly (and embeds + # fixture paths); verdicts may not. + if ! "$PYTHON" -c " +import json,sys +a=json.load(open('$tmp/committed.json'))['verdicts'] +b=json.load(open('$GOLDENS'))['verdicts'] +va={k:v['verdict'] for k,v in a.items()}; vb={k:v['verdict'] for k,v in b.items()} +sys.exit(0 if va==vb else 1) +"; then + cp "$tmp/committed.json" "$GOLDENS" # restore — check must not mutate + echo "DRIFT: the ingestor's validator verdicts changed. Re-run the generator," >&2 + echo "commit the new goldens, and update cases.json (+ the Go preview) consciously." >&2 + exit 1 + fi + cp "$tmp/committed.json" "$GOLDENS" # keep the committed copy (paths etc. unchanged) + echo "validator goldens in sync" +else + "$PYTHON" scripts/gen-validator-goldens.py +fi diff --git a/scripts/tests/install-verify.sh b/scripts/tests/install-verify.sh index f7ca660..ed388ad 100755 --- a/scripts/tests/install-verify.sh +++ b/scripts/tests/install-verify.sh @@ -58,7 +58,7 @@ EOF chmod +x "$BIN/curl" # coreutils the installer needs, so PATH=$BIN alone works (host cosign hidden). - for t in bash sh env mkdir mktemp cp cat awk grep sed head tr uname chmod mv rm sleep printf install dirname basename sha256sum shasum; do + for t in bash sh env mkdir mktemp cp cat awk grep sed head tr uname chmod mv rm sleep printf install dirname basename sha256sum shasum ln readlink; do p="$(command -v "$t" 2>/dev/null)" && ln -sf "$p" "$BIN/$t" done @@ -85,6 +85,13 @@ if [ "$rc" = 0 ] && grep -q "cosign signature valid" "$SBX/out" && [ -x "$DEST/t else bad "cosign present + valid sig installs (rc=$rc)"; sed 's/^/ /' "$SBX/out" fi +# the tb short alias rides along on the happy path (best-effort, but with ln +# available — as here — it must exist and point at the binary) +if [ "$(readlink "$DEST/tb" 2>/dev/null)" = "$DEST/tracebloc" ]; then + ok "tb short alias created next to the binary" +else + bad "tb short alias missing or wrong target ($(readlink "$DEST/tb" 2>/dev/null || echo none))" +fi drop_sandbox # ── 2. cosign present + FAILED signature → aborts ───────────────────────────