Skip to content

Add CLI commands for write operations (automations, behaviors, projects, traces, judges)#2

Merged
adivate2021 merged 6 commits into
stagingfrom
devin/1776990383-write-tools
Apr 28, 2026
Merged

Add CLI commands for write operations (automations, behaviors, projects, traces, judges)#2
adivate2021 merged 6 commits into
stagingfrom
devin/1776990383-write-tools

Conversation

@devin-ai-integration
Copy link
Copy Markdown
Contributor

@devin-ai-integration devin-ai-integration Bot commented Apr 24, 2026

Summary

Regenerates src/judgment_cli/generated_commands.py against the updated CLI server OpenAPI spec (https://github.com/JudgmentLabs/judgment-mono/pull/1206) and adds human-focused help text for every new write command.

New write commands wired through the CLI:

  • judgment automations create|update|delete|get
  • judgment behaviors create-binary|create-classifier|update|delete
  • judgment judges update-settings
  • judgment projects create|add-favorite|remove-favorite
  • judgment traces add-tags|evaluate

Help overrides (scripts/help_overrides.py): every new command gets a multi-paragraph description with a worked judgment … example. Non-obvious JSON-shaped flags (--conditions, --actions, --cooldown-period, --trigger-frequency, --options, --advanced-settings, --span-triggers) have full shape references so a human running judgment automations create --help can assemble a valid payload without reading the server code.

Codegen was run via uv run python scripts/generate_cli.py --spec <local-cli-server-spec>; the spec isn't publicly served yet, so the generated file in this PR reflects the future staging spec once the CLI server changes land.

Review & Testing Checklist for Human

  • Run judgment <group> <command> --help for each new write command and confirm the rendered help text reads correctly (examples, flag descriptions, JSON shapes).
  • Point the CLI at a staging deployment of the CLI server from the monorepo PR and smoke-test at least one write per group, e.g. judgment projects create …, judgment behaviors create-binary …, judgment traces add-tags …, judgment automations create ….
  • Re-run uv run python scripts/generate_cli.py once the updated CLI server is reachable at the default --spec URL and diff the result against this PR to confirm the generated file matches.

Notes

  • Depends on https://github.com/JudgmentLabs/judgment-mono/pull/1206, which adds the matching CLI server routes. Merging this PR before the CLI server ships will produce 404s on any write command.
  • A few CLI server paths in the monorepo PR were renamed for CLI ergonomics so command names aren't ambiguous: /projects/{add,remove}-favorite (not /projects/favorite/{add,remove}) and /traces/add-tags (not /traces/tags/add).
  • uv.lock is intentionally not touched.

Link to Devin session: https://app.devin.ai/sessions/a33c164803e245e09aad04e3f8434432
Requested by: @adivate2021


Open in Devin Review

Co-Authored-By: Aaryan Divate <divateaaryan@gmail.com>
@devin-ai-integration
Copy link
Copy Markdown
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

devin-ai-integration[bot]

This comment was marked as resolved.

…ngs, span triggers, evaluation_mode)

Co-Authored-By: Aaryan Divate <divateaaryan@gmail.com>
devin-ai-integration[bot]

This comment was marked as resolved.

Comment thread src/judgment_cli/judges.py Outdated
"""Attach the hand-written judges commands to the auto-generated group."""
judges_group.add_command(judges_upload)
judges_group.add_command(judges_init)
MANUAL_GROUP_COMMANDS: dict[str, list[click.Command]] = {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this necessary? cant we just use the existing @click decorator and write new modules with it that extends the same object as the generated code?

@adivate2021 adivate2021 merged commit 0e7adca into staging Apr 28, 2026
@adivate2021 adivate2021 deleted the devin/1776990383-write-tools branch April 28, 2026 03:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants