Skip to content

Auto-detect agent contexts for JSON output; gh-style exit codes#222

Merged
sdairs merged 13 commits into
mainfrom
agent-context-and-exit-codes
Jun 4, 2026
Merged

Auto-detect agent contexts for JSON output; gh-style exit codes#222
sdairs merged 13 commits into
mainfrom
agent-context-and-exit-codes

Conversation

@kcmannem
Copy link
Copy Markdown
Collaborator

@kcmannem kcmannem commented May 26, 2026

Summary

Mirrors the workos CLI behavior so agents and shell pipelines get machine-readable output and meaningful exit codes without callers having to opt in.

  • Auto-detect JSON output. clickhousectl now emits JSON to stdout when any of these hold: --json is passed, stdout is not a terminal, or one of CLAUDECODE / CURSOR_AGENT / CODEX_SANDBOX is set. Resolution lives in src/output_mode.rs and is computed once in main / run_cloud, so all downstream json: bool plumbing stays unchanged.
  • gh-style exit codes. Error::exit_code() now returns 0 success, 1 error, 2 cancelled, 4 auth required. CloudError carries a kind (Generic / Auth); 401/403 responses, missing creds, and the OAuth-only-can't-write block all route to Error::AuthRequired and exit with 4.
  • Docs (README.md, CLAUDE.md, cli.rs after_help) document the new behavior.

Test plan

  • cargo build -p clickhousectl
  • cargo test -p clickhousectl — 283 + 37 tests pass, including new tests for output_mode::resolve and Error::exit_code
  • Smoke: cargo run -- local list (non-TTY) → JSON without --json
  • Smoke: CLAUDECODE=1 cargo run -- local list → JSON
  • Smoke: cargo run -- cloud service list with no creds → exit 4
  • Smoke: cargo run -- local use 99.99 → generic error, exit 1

Note

Low Risk
Behavior changes are limited to CLI output mode, exit codes, and error classification; no auth protocol or cloud API contract changes beyond clearer exit 4 for credential failures.

Overview
Adds agent-aware JSON output and gh-style exit codes so scripts and coding agents can rely on machine-readable stdout and meaningful process exits without always passing --json.

JSON: A new json_output() helper turns on JSON when --json is set or is_ai_agent::detect() finds a known agent (same signal as the outbound User-Agent). Local and cloud dispatch use it instead of the raw flag; cloud auth status follows the same rule. local server start --foreground with --json no longer errors—the foreground path ignores JSON because it never prints a summary.

Exit codes: Error::exit_code() maps success/cancelled/auth to 0 / 2 / 4 (everything else 1); main exits with that code. CloudError gains CloudErrorKind (Auth vs Generic): missing/partial API keys, no credentials, and API 401/403 become Error::AuthRequired (exit 4); OAuth blocked on write commands and half-set login flags route there too. Cloud handlers that return Box<dyn Error> preserve auth kind via downcast to CloudError.

Docs (README, AGENTS.md, cloud after_help) describe auto-JSON for agents and the exit table.

Reviewed by Cursor Bugbot for commit dc032b4. Bugbot is set up for automated code reviews on this repo. Configure here.

Emit JSON automatically when stdout is not a terminal or one of
CLAUDECODE / CURSOR_AGENT / CODEX_SANDBOX is set, so agents and pipelines
get structured output without passing --json.

Exit codes now follow gh conventions: 0 success, 1 error, 2 cancelled,
4 auth required. CloudError carries a kind (Generic / Auth) so 401/403
responses and missing-credential paths route to Error::AuthRequired.
@kcmannem kcmannem temporarily deployed to cloud-integration May 26, 2026 16:53 — with GitHub Actions Inactive
Comment thread crates/clickhousectl/src/main.rs Outdated
Comment thread crates/clickhousectl/src/main.rs Outdated
…assify token-save IO as generic error

- Thread explicit --json through to start_server so JsonForegroundConflict
  only fires when the user actually passed --json. Auto-detected JSON in a
  non-TTY / agent context no longer breaks `local server start --foreground`.
- Map ensure_fresh_tokens() failures to Error::Cloud (exit 1), not
  AuthRequired (exit 4). Its only error path is a filesystem write failure
  while persisting refreshed tokens — refresh-rpc failures are swallowed and
  tokens cleared — so re-auth wouldn't help.
- Trim added comments / doc-comments to match the project's existing voice.
@kcmannem kcmannem temporarily deployed to cloud-integration May 26, 2026 17:23 — with GitHub Actions Inactive
@sdairs
Copy link
Copy Markdown
Collaborator

sdairs commented May 26, 2026

@kcmannem love the idea; dynamically adapting the experience for agents I've something I've been playing around with (even so far as completely changing the command layout for agents - some fun results in evals) - but I haven't played around with output formats yet. I assume most run --json anyway so that they can jq, so in theory I'm supportive.

Before merging, I'm going to run some comparative evals to see how models react to this change.

Comment thread crates/clickhousectl/src/error.rs
Bug fix added the json_explicit param, pushing the count to 8. Matches
the existing project pattern (see cloud/commands.rs, local/postgres.rs).
@kcmannem kcmannem temporarily deployed to cloud-integration May 26, 2026 18:01 — with GitHub Actions Inactive
sdairs and others added 4 commits June 3, 2026 16:25
Replace the hand-maintained AGENT_ENV_VARS list (CLAUDECODE / CURSOR_AGENT
/ CODEX_SANDBOX) with is_ai_agent::detect(), the same detection already
used for the outbound User-Agent in user_agent.rs.

This keeps the two consistent and broadens coverage to every agent the
crate knows about (the standard AGENT var, Claude Code, Cursor IDE + CLI,
all Codex vars, Gemini CLI, Goose, Devin, …) instead of a narrow subset.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drop the "stdout is not a terminal" trigger from output-mode resolution.
JSON is now emitted automatically only when --json is passed or a coding
agent is detected; pipes and redirects stay human-readable unless --json
is passed, matching gh/kubectl conventions.

The non-TTY default silently changed output format for any
`clickhousectl ... | grep` pipeline (it broke the project's own postgres
integration tests, which grep markdown rows from piped output). Agents
still get JSON via is_ai_agent::detect(); everyone else opts in.

Updates README, CLAUDE.md, and the cloud after_help text to match.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The output-mode decision is now trivial (--json or a detected agent), so
fold it into a small json_output() helper in main and drop the
output_mode module.

`server start --foreground` already ignores json — it streams the
server's stdout/stderr and never emits a JSON summary — so make that
explicit instead of erroring. This removes:

- the JsonForegroundConflict error and its runtime check
- the json_explicit plumbing that existed only to scope that check
- the #[allow(clippy::too_many_arguments)] on start_server (now 7 args)

Agents still get JSON automatically; everyone else opts in with --json.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cover the previously-untested logic behind gh-style exit codes:

- convert_error: 401/403 -> Auth, other statuses and non-API errors ->
  Generic.
- cloud_error_to_top_level / boxed_cloud_error_to_top_level: a CloudError
  routes by kind (Auth -> AuthRequired, Generic -> Cloud), survives the
  Box<dyn Error> downcast, and a non-CloudError falls back to Cloud.

The downcast fallback test pins the contract that anything which isn't a
concrete CloudError (e.g. a handler that stringifies before boxing)
silently loses the exit-4 signal.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 8440191. Configure here.

Comment thread crates/clickhousectl/src/main.rs
sdairs and others added 3 commits June 3, 2026 18:07
output_mode.rs was folded into a json_output() helper in main.rs, and
server start --foreground now ignores json instead of erroring. Point
the developer note at the current code.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The json_output() helper is small and self-explanatory; the detailed
note didn't earn its place in the dev guide.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Non-interactive `cloud auth login` with only one of --api-key / --api-secret
returned Error::Cloud (exit 1), while the same partial-flag condition in
resolve_auth exits 4 (auth required). Align login to Error::AuthRequired so
agents keyed on exit 4 for credential problems handle it consistently.

Reported by Cursor Bugbot.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@sdairs
Copy link
Copy Markdown
Collaborator

sdairs commented Jun 3, 2026

Pushed a round of review changes (commits 7b9931c..d4f23de):

JSON output mode

  • Use the existing is_ai_agent::detect() (same detection as the User-Agent) instead of a hardcoded CLAUDECODE/CURSOR_AGENT/CODEX_SANDBOX list — broader, consistent coverage (standard AGENT var, Gemini CLI, Goose, Devin, …).
  • Dropped the non-TTY → JSON trigger. Auto-JSON now fires only for detected agents or explicit --json; pipes/redirects stay human-readable (matches gh/kubectl). This also fixes the red local postgres edge cases job, which pipes output and greps human tables.
  • server start --foreground now simply ignores --json (it streams stdout/stderr, no JSON summary) instead of erroring. Removed JsonForegroundConflict, the json_explicit plumbing, and the #[allow(too_many_arguments)].
  • Collapsed output_mode.rs into a one-line json_output() helper.

Exit codes

  • Fixed an inconsistency (thanks Bugbot): partial --api-key/--api-secret on cloud auth login now exits 4 like the same condition in resolve_auth, not 1.
  • Added unit tests for the previously-untested convert_error classification (401/403 → Auth, else Generic) and the Box<dyn Error> downcast routing.

Docs: README, cli.rs after_help, and CLAUDE.md updated to match.

Follow-ups filed: #233 (return a concrete Error from cloud handlers to retire the downcast) and #234 (review the exit-code taxonomy — 403 granularity, 5xx, 429/rate-limit backoff, etc.).

sdairs and others added 3 commits June 3, 2026 18:14
--json is already documented as a flag; drop the prose restatement and
keep just the auto-JSON-for-agents line plus the exit codes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
# Conflicts:
#	CLAUDE.md
#	CLAUDE.md~HEAD
#	crates/clickhousectl/src/cloud/client.rs
#	crates/clickhousectl/src/cloud/mod.rs
@sdairs sdairs temporarily deployed to cloud-integration June 3, 2026 17:24 — with GitHub Actions Inactive
Comment thread crates/clickhousectl/src/output_mode.rs Outdated
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

take a look at how we do agent detection when building the user agent string for http calls; we should do it the same way (I maintain a library for it that detects more agents already)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Collaborator

@iskakaushik iskakaushik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@sdairs sdairs merged commit 4110d42 into main Jun 4, 2026
13 checks passed
@sdairs sdairs deleted the agent-context-and-exit-codes branch June 4, 2026 17:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants