Add Judgment CLI (JUD-5155) by adivate2021 · Pull Request #1 · JudgmentLabs/cli

adivate2021 · 2026-03-31T03:27:47Z

linear · 2026-04-21T04:40:37Z

JUD-5155 Make Judgment CLI

Background

Make Judgment CLI

Technical

CLI that basically wraps our API, new CLI server

abhishekg999 · 2026-04-21T04:37:30Z

+
+
+def _apply_request_defaults(body: object, schema: dict) -> None:
+    """Fill in schema-driven defaults for generated POST bodies."""


why is this needed. the openapi schema defafults are not meant to be followed by consumers

abhishekg999 · 2026-04-21T04:38:03Z

+import click
+
+
+def _output(data: object) -> None:


can we have a more clean UI file to centralize UI because im sure CLIs will require more stuff specifi here

abhishekg999 · 2026-04-21T04:38:30Z

+_DEFAULT_BASE_URL = "https://api3.judgmentlabs.ai"
+
+
+def _config_dir() -> Path:


Will this work on windows? also why specifically XDG_CONFIG_HOME?

abhishekg999 · 2026-04-21T04:39:15Z

+        or cfg.get("base_url")
+        or _DEFAULT_BASE_URL
+    )
+    api_key = (


just a nit but do we want to follow similar patterns to judgeval with requireEnvVar and optionalEnvVar instaed of having this kinda thing

abhishekg999 · 2026-04-21T04:42:33Z

+# List traces with custom filters
+judgment traces list <PROJECT_ID> -d '{"pagination": {"limit": 10, "cursorSortValue": null, "cursorItemId": null}}'
+
+# Get trace details


these formats dont seem right for a CLI.
maybe try to follow the strategy something like: https://www.npmjs.com/package/@openapitools/openapi-generator-cli?

I think its a fine V1 but its a bit clunky of a CLI since you need to guess the arg orrder.

Can you also see if click comes with terminal auto completions? Something like this:
https://click.palletsprojects.com/en/stable/shell-completion/

you can look at https://cli.ahh.bet/ as an example CLI installation thats more common practice

abhishekg999 · 2026-04-21T04:43:21Z

@@ -0,0 +1,66 @@
+class JudgmentCli < Formula


If this works its cool. Do we want to support only homebrew or also a:

curl -fsSL https://judgmentlabs.ai/install.sh | bash

Kinda thing

devin-ai-integration

Devin Review found 2 new potential issues.

View 8 additional findings in Devin Review.

devin-ai-integration · 2026-04-22T18:23:44Z

+        if qp["required"]:
+            type_arg = click_param_args(qp["schema"])
+            lines.append(f'@click.argument("{qp["name"]}"{type_arg})')


🟡 Generator emits mismatched Click argument name vs function parameter for camelCase names

At scripts/generate_cli.py:299, required query params generate @click.argument("{qp["name"]}") using the raw API name (e.g. matchCount), but the function signature at line 335 uses py_var_name(q["name"]) (e.g. match_count). Click derives the Python parameter name from the argument declaration by only replacing - with _, so @click.argument("matchCount") maps to parameter matchCount, not match_count. The same mismatch exists for required scalar body props at line 311. When the OpenAPI spec contains a camelCase required parameter, this will produce a TypeError at runtime because Click passes the argument as a keyword argument that doesn't match the function signature.

Affected code paths

Line 299 for required query params:

lines.append(f'@click.argument("{qp["name"]}"...')

Line 311 for required scalar body props:

lines.append(f'@click.argument("{prop["name"]}"...')

But function signature at line 335 uses py_var_name() which converts camelCase → snake_case.

Was this helpful? React with 👍 or 👎 to provide feedback.

abhishekg999

I like that this is all codegen, I will list some issues with specific commands during testing. Also due to the fact that this is all codegen I think its fair we should have some tests in this repo to test these (non codegen).

abhishekg999 · 2026-04-23T00:36:51Z

@@ -0,0 +1,141 @@
+#!/usr/bin/env python3
+"""Render the Homebrew formula for judgment-cli.


how do we test this

abhishekg999 · 2026-04-23T00:39:34Z

+	python3 scripts/generate_cli.py $(SPEC_URL)
+
+# Install the CLI in the current Python environment
+install:


wait this will use my global pip by default right
we should be using uv here right

abhishekg999 · 2026-04-23T00:52:53Z

+
+
+@cli.command()
+def logout() -> None:


One suggestion:

login and logout is fine, I would suggest some judgment configure. right now if i login i dont really have a way to change my organizeation id right other than either manually editing the file, or regoing through the login flow.

abhishekg999 · 2026-04-23T00:56:41Z

❯ judgment status
❯ JUDGMENT_BASE_URL=http://localhost:10006 judgment status
Credential resolution (highest priority first):

  Env     JUDGMENT_API_KEY  (not set)
  Env     JUDGMENT_BASE_URL = http://localhost:10006
  Env     JUDGMENT_ORG_ID  (not set)
  Config  /Users/ahh/Library/Application Support/judgment/credentials.json
          api_key: 441f…2264
❯ JUDGMENT_BASE_URL=http://localhost:10006 judgment projects list
Error 401: {
  "error": "Authentication failed",
  "message": "X-Organization-Id header is required"
}

I think we should not propagate the error message of requiring organization id like this. For routes that require the organization that should be something we can do a more proper CLI error for.

abhishekg999 · 2026-04-23T00:57:45Z

❯ JUDGMENT_BASE_URL=http://localhost:10006 judgment docs search hi
Error 401: {
  "error": "Authentication failed",
  "message": "X-Organization-Id header is required"
}

It seeems even docs search required X-Organization-Id? is this intentional. It seems infact everything requires organization id and in that case should we simply make it mandatory.

abhishekg999 · 2026-04-23T01:00:08Z

Also in general we really need a completion script IMO. If we really just want agents to use it then its probably fine but:

❯ JUDGMENT_BASE_URL=http://localhost:10006 judgment sessions get
Usage: judgment sessions get [OPTIONS] PROJECT_ID SESSION_ID
Try 'judgment sessions get --help' for help.

Error: Missing argument 'PROJECT_ID'.

otherwise this is very inconvenient to work with ids here. I think having tab completion script here wuld be good.
https://click.palletsprojects.com/en/stable/shell-completion/

if this seems out of scope for this first Pr (which i would say so, lets atleast make a ticket for this)

Initial commit add judgment cli

7c31d76

This comment was marked as resolved.

Sign in to view

Remove handrolled logic + use CLI server

4c1b507

This comment was marked as resolved.

Sign in to view

adivate2021 changed the title ~~Initial commit add judgment cli~~ Add Judgment CLI Apr 21, 2026

adivate2021 changed the title ~~Add Judgment CLI~~ Add Judgment CLI (JUD-5155) Apr 21, 2026

abhishekg999 requested changes Apr 21, 2026

View reviewed changes

CLI ergonomics

00e9e67

adivate2021 force-pushed the aaryan/add-judgment-cli branch from e318e8b to 00e9e67 Compare April 22, 2026 18:15

devin-ai-integration Bot reviewed Apr 22, 2026

View reviewed changes

adivate2021 added 3 commits April 22, 2026 11:43

Port over judge upload from judgeval

cfc5c7c

Remove -d and -f autogenerated options

98037bc

homebrew fix

ae426cc

abhishekg999 requested changes Apr 23, 2026

View reviewed changes

abhishekg999 reviewed Apr 23, 2026

View reviewed changes

abhishekg999 self-requested a review April 23, 2026 01:12

abhishekg999 approved these changes Apr 23, 2026

View reviewed changes

adivate2021 force-pushed the aaryan/add-judgment-cli branch from dccf0d1 to 753823b Compare April 23, 2026 17:47

adivate2021 changed the base branch from main to staging April 23, 2026 18:20

Fixes based on PR review

87d20f6

adivate2021 force-pushed the aaryan/add-judgment-cli branch from 753823b to 87d20f6 Compare April 23, 2026 18:39

adivate2021 merged commit 5f6114f into staging Apr 23, 2026

adivate2021 deleted the aaryan/add-judgment-cli branch April 23, 2026 22:46



		def _apply_request_defaults(body: object, schema: dict) -> None:
		"""Fill in schema-driven defaults for generated POST bodies."""

		_DEFAULT_BASE_URL = "https://api3.judgmentlabs.ai"


		def _config_dir() -> Path:

		@@ -0,0 +1,141 @@
		#!/usr/bin/env python3
		"""Render the Homebrew formula for judgment-cli.

Conversation

adivate2021 commented Mar 31, 2026 • edited by devin-ai-integration Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

linear Bot commented Apr 21, 2026

Background

Technical

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

devin-ai-integration Bot Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abhishekg999 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abhishekg999 commented Apr 23, 2026

Uh oh!

abhishekg999 commented Apr 23, 2026

Uh oh!

abhishekg999 commented Apr 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

adivate2021 commented Mar 31, 2026 •

edited by devin-ai-integration Bot

Loading

devin-ai-integration Bot Apr 22, 2026 •

edited

Loading