Add CLI commands for write operations (automations, behaviors, projects, traces, judges)#2
Merged
Merged
Conversation
Co-Authored-By: Aaryan Divate <divateaaryan@gmail.com>
Contributor
Author
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
…ngs, span triggers, evaluation_mode) Co-Authored-By: Aaryan Divate <divateaaryan@gmail.com>
Co-Authored-By: Aaryan Divate <divateaaryan@gmail.com>
abhishekg999
approved these changes
Apr 24, 2026
| """Attach the hand-written judges commands to the auto-generated group.""" | ||
| judges_group.add_command(judges_upload) | ||
| judges_group.add_command(judges_init) | ||
| MANUAL_GROUP_COMMANDS: dict[str, list[click.Command]] = { |
Contributor
There was a problem hiding this comment.
is this necessary? cant we just use the existing @click decorator and write new modules with it that extends the same object as the generated code?
abhishekg999
approved these changes
Apr 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Regenerates
src/judgment_cli/generated_commands.pyagainst the updated CLI server OpenAPI spec (https://github.com/JudgmentLabs/judgment-mono/pull/1206) and adds human-focused help text for every new write command.New write commands wired through the CLI:
judgment automations create|update|delete|getjudgment behaviors create-binary|create-classifier|update|deletejudgment judges update-settingsjudgment projects create|add-favorite|remove-favoritejudgment traces add-tags|evaluateHelp overrides (
scripts/help_overrides.py): every new command gets a multi-paragraph description with a workedjudgment …example. Non-obvious JSON-shaped flags (--conditions,--actions,--cooldown-period,--trigger-frequency,--options,--advanced-settings,--span-triggers) have full shape references so a human runningjudgment automations create --helpcan assemble a valid payload without reading the server code.Codegen was run via
uv run python scripts/generate_cli.py --spec <local-cli-server-spec>; the spec isn't publicly served yet, so the generated file in this PR reflects the future staging spec once the CLI server changes land.Review & Testing Checklist for Human
judgment <group> <command> --helpfor each new write command and confirm the rendered help text reads correctly (examples, flag descriptions, JSON shapes).judgment projects create …,judgment behaviors create-binary …,judgment traces add-tags …,judgment automations create ….uv run python scripts/generate_cli.pyonce the updated CLI server is reachable at the default--specURL and diff the result against this PR to confirm the generated file matches.Notes
404s on any write command./projects/{add,remove}-favorite(not/projects/favorite/{add,remove}) and/traces/add-tags(not/traces/tags/add).uv.lockis intentionally not touched.Link to Devin session: https://app.devin.ai/sessions/a33c164803e245e09aad04e3f8434432
Requested by: @adivate2021