ruby_llm-llm_judge

Use a RubyLLM chat model to answer RubyLLM Judge questions. Define probability, choice, or score questions and receive RubyLLM's native Judgment and typed answers. Choose between parallel per-answer ratings and one-call typed distributions.

gem 'ruby_llm-llm_judge'

The gem works with RubyLLM 1.13 and later, and is tested against 1.13.2, 1.16.0, 2.0.0, 2.1.0 and RubyLLM's main branch. On RubyLLM 2.1 it plugs into the Judge API; on earlier versions see RubyLLM 1.13 to 2.0.

Configure your chat provider's credentials through RubyLLM. The default scoring model is gpt-6-luna; you can choose another RubyLLM chat model for each judgment.

The gem also runs on RubyLLM 1.13 through 2.0, which predate the Judge API. There, RubyLLM::LLMJudge.judge runs the same scoring directly and accepts the same questions and provider_options. Answers have the same readers as RubyLLM's (probability, choice, probabilities, confidence, score, levels, and model, tokens, raw, [] and fetch on the judgment), so call sites keep working after you upgrade to 2.1.

What differs before 2.1:

• Results are RubyLLM::LLMJudge::Legacy objects, not RubyLLM::Judgment, RubyLLM::Choice, and so on. Avoid class checks until you upgrade.
• There is no RubyLLM.judge, RubyLLM::Judge class DSL, cost, instrumentation, or metadata:. Passing metadata: raises. context: is accepted and supplies its configuration.
• On 1.x, scoring_protocol accepts only :chat_completions, the API RubyLLM 1.x uses for OpenAI. RubyLLM 2.0 accepts any protocol its provider supports.
• On 1.x, the output limit and chat_provider_options are sent with with_params, using the field each provider expects (max_completion_tokens for OpenAI and Azure, generationConfig.maxOutputTokens for Gemini and Vertex AI, inferenceConfig.maxTokens for Bedrock, and max_tokens otherwise).

require 'ruby_llm/llm_judge'

result = RubyLLM::LLMJudge.judge(
  'Please refund the duplicate charge today.',
  questions: {
    urgent: {
      type: :probability,
      instructions: 'Does this need attention today?',
      criteria: { yes: 'An explicit deadline today', no: 'No deadline today' }
    },
    department: {
      type: :choice,
      instructions: 'Which team should handle this?',
      options: { billing: 'Payments and refunds', technical: 'Bugs and integrations' }
    },
    frustration: {
      type: :score,
      instructions: 'How frustrated is the customer?',
      levels: ['Calm', 'Frustrated', 'Angry']
    }
  }
)

result.urgent.probability
result.department.choice
result.department.probabilities
result.frustration.score
result.model  # => "gpt-6-luna"
result.tokens

You can also use a reusable Judge class:

class TicketTriage < RubyLLM::Judge
  model 'gpt-6-luna', provider: :llm_judge, assume_model_exists: true
  probability :urgent, 'Does this need attention today?'
end

TicketTriage.judge('Please refund today.').urgent.probability

Pass the chat model as model: and its provider as scoring_provider. For example, Claude Haiku 4.5 through OpenRouter:

result = RubyLLM::LLMJudge.judge(
  'Please refund the duplicate charge today.',
  model: 'anthropic/claude-haiku-4.5',
  questions: { urgent: { type: :probability, instructions: 'Does this need attention today?' } },
  provider_options: {
    scoring_provider: :openrouter,
    temperature: 0
  }
)

Both strategies return valid judgments with this model. Its one-call answers arrive in a Markdown code fence, which the gem removes before parsing. On the benchmark cases it scored 56–57/64 on AG News and 29–30/32 on SST-2 at about 1.1 s median, a little below Luna. Haiku results.

The default Luna rating request uses Chat Completions, temperature zero, reasoning disabled, store: false, and a four-token output limit. Tune requests with scoring_protocol, temperature, max_output_tokens, and chat_provider_options. chat_provider_options is merged over those defaults, so setting one key, such as service_tier: 'flex', keeps reasoning disabled; set a key explicitly to change a default. Set a key to nil to remove that field from the request, whether it is a default or, on RubyLLM 1.x, the output limit field. For example, an app that routes RubyLLM 1.x OpenAI chats through the Responses API can send chat_provider_options: { max_completion_tokens: nil, max_output_tokens: 256 }.

Luna defaults apply to each provider's own Luna model IDs: gpt-6-luna, gpt-5.6-luna, or gpt-luna-latest on :openai, which sends temperature zero, reasoning_effort: 'none', and store: false; and openai/gpt-6-luna or openai/gpt-5.6-luna on :openrouter, which sends temperature zero and reasoning: { enabled: false }. Pro variants, other providers such as :bedrock, and prefixed IDs such as us.openai.gpt-5.6-luna sent through :openai to an OpenAI-compatible gateway get no Luna defaults, because their request format may differ; set temperature: and disable reasoning in that endpoint's format yourself. With reasoning left on, the ratings strategy's four-token output limit is spent on reasoning: on eight AG News cases, gpt-6-luna through OpenRouter returned no digit for all eight, and gpt-5.6-luna for four. RubyLLM 1.x sends temperature 1.0 for any OpenAI model whose ID starts with gpt-5, so gpt-5.6-luna on :openai runs at 1.0 there unless you pass chat_provider_options: { temperature: 0 }.

Claude Haiku 5.5 (anthropic/claude-haiku-5.5 through OpenRouter) costs a tenth of Haiku 4.5 per token, and on the benchmarks it was as accurate or better. It thinks by default, which would spend the ratings strategy's four-token limit, so the gem sends reasoning: { enabled: false } for anthropic/claude-haiku-5.5, its :batch variant and ~anthropic/claude-haiku-latest on :openrouter. OpenRouter doesn't list temperature as a supported parameter for Haiku 5.5, so the gem leaves it unset.

RubyLLM::LLMJudge.judge(
  'Please refund the duplicate charge today.',
  model: 'anthropic/claude-haiku-5.5',
  questions: { team: { type: :choice, options: { billing: 'Payments', technical: 'Software' } } },
  provider_options: { scoring_provider: :openrouter, strategy: :single_request, structured_output: true }
)

With reasoning off, Haiku 5.5 got only 13–24% of StreamDecisionBench's rule-following states right. Turn reasoning on for those questions. Haiku 5.5 results.

Set strategy: :single_request to send the state and every Judge question to the model in one request:

result = RubyLLM::LLMJudge.judge(
  'Please refund the duplicate charge today.',
  questions: {
    urgent: { type: :probability, instructions: 'Does this need attention today?' },
    department: {
      type: :choice,
      instructions: 'Which team should handle this?',
      options: { billing: 'Payments and refunds', technical: 'Bugs and integrations' }
    }
  },
  provider_options: { strategy: :single_request, max_output_tokens: 1024 }
)

result.urgent.probability
result.department.probabilities

The model returns one JSON object containing a distribution for each question. The gem checks that every question and option is present, validates each probability, normalizes each distribution, and builds RubyLLM's typed answers. result.raw[:reported_probabilities] and result.raw[:reported_totals] preserve the model's original numbers for inspection. With an OpenAI key configured, this example calls Luna directly through Chat Completions with reasoning disabled, temperature zero, and store: false. The 1024-token limit suits small judgments; omit it for large question sets to use the 8192-token default. It makes one corrective retry for malformed JSON, missing fields, or a Choice whose top probabilities tie, counting both calls in result.tokens and result.raw[:attempts]. A tie that remains raises instead of selecting the first option; set tie_breaker: :first to accept the first option. Set malformed_retries: 0 to disable that retry. An invalid response after retries raises an error.

If you use OpenRouter for Luna, prioritize the provider with the lowest observed latency:

result = RubyLLM::LLMJudge.judge(
  'Please refund the duplicate charge today.',
  model: 'openai/gpt-6-luna',
  questions: {
    department: {
      type: :choice,
      instructions: 'Which team should handle this?',
      options: { billing: 'Payments and refunds', technical: 'Bugs and integrations' }
    }
  },
  provider_options: {
    strategy: :single_request,
    scoring_provider: :openrouter,
    temperature: 0,
    max_output_tokens: 1024,
    chat_provider_options: {
      reasoning: { enabled: false },
      provider: { sort: 'latency' }
    }
  }
)

OpenRouter's latency sorting chooses among its available providers using their observed response times. It sped up OpenRouter in our test, while direct OpenAI was faster still. The measured speed and answer-quality tradeoffs are below.

Set structured_output: true to send a strict JSON Schema with the one-call request, through RubyLLM's with_schema. The schema requires every question and every option, and nothing else, so a provider that supports structured outputs cannot return the wrong shape. Without it, the gem validates the reply and makes its corrective retry. In our runs it had little effect on accuracy, cut GPT-6 Luna's corrective retries from 3.3 to 0.7 per 96 judgments, and added 40–350 ms. Models and providers without structured-output support reject the request, so leave it off for them.

Set strategy: :committed to ask for one option per question instead of a distribution. The reply is shorter, and each answer comes back with probability 1 for the chosen option and 0 for the rest, so confidence is always 1.0 and the probabilities carry no calibration. Use it when you only need the decision; use :single_request or :ratings when you threshold on probabilities. It accepts structured_output: true, which limits each answer to that question's option IDs. On classification it was 100–300 ms faster than probabilities and as accurate or up to three cases in 64 less; on rule-following questions it was the most accurate mode with reasoning off.

The Luna defaults disable reasoning, which suits classification: on the benchmarks below, Luna is as accurate with reasoning off and much faster. Questions that apply explicit rules to a state are different. On StreamDecisionBench, whose questions carry decision policies, GPT-5.6 Luna through LLM Judge answered 47% of states correctly with reasoning off and 90% with reasoning effort low, at about twice the latency. The ratings strategy cannot apply a policy across a state and should not be used for these questions. Classification and rule-following results by mode.

To enable reasoning, override the default and leave room for it in the output limit:

provider_options: {
  strategy: :single_request,
  max_output_tokens: 4096,
  chat_provider_options: { reasoning_effort: 'low' }          # OpenAI
  # chat_provider_options: { reasoning: { effort: 'low' } }   # OpenRouter
}

Reasoning does not fit the :ratings strategy, whose four-token output limit is meant for a single digit.

The default :ratings strategy asks the model to rate every declared answer from 0 to 9. It makes one call per answer, with up to six calls running at once (max_workers: changes this), then applies softmax to the ratings. If a call fails after its retry, no new calls start, calls in flight finish, and the error is raised. It retries a malformed digit once. When the highest ratings tie on a Choice question, it makes a one-call JSON judgment for that question and uses its distribution. If that call also ties, it raises an error instead of choosing whichever option came first. result.raw[:tie_breaks] records each resolution, and token usage includes the extra call. Set tie_breaker: :first to use the original first-option rule, or tie_break_max_output_tokens: to change the tie-break response limit (default 8192).

The :single_request strategy asks for all distributions in one call. Both strategies build the same RubyLLM answer types: a choice returns the selected answer and distribution; a score returns the distribution and its weighted level; a probability returns the positive answer's share. The gem supports Judge's 1–255 choice options, 2–10 score levels, and multiple questions in one judgment.

These probabilities compare the answers you supplied. The :single_request values are reported by the chat model; the default ratings values come from softmax over its 0–9 ratings. Neither strategy establishes calibration by itself. Include an other or escalate choice when the named answers may not cover the input. confidence measures how concentrated the returned distribution is: 1.0 for a single option, 0.0 for a tie. Use labeled examples to set any automation thresholds.

Set optional max_arms and max_input_bytes in provider_options to cap work per judgment. For example, { max_arms: 12, max_input_bytes: 32_768 } rejects larger requests before scoring. Token usage is aggregated across calls, and raw contains the ratings and model metadata.

The one-digit-per-answer approach was inspired by @burkov's post on Jev. The one-call strategy follows the typed-question pattern in TypeSafe's System One adapter.

On September 23, 2026, we tested GPT-6 Luna (OpenRouter), DeepSeek V4.1 Flash (Fireworks), Celeris-1, and Jev 1.13.0 on 64 balanced AG News articles with four choices and 32 balanced SST-2 sentences. LLM runs used temperature zero with reasoning disabled. Correct counts all cases; Usable counts cases that returned a valid judgment. Brier scores and latency use usable cases only. Latency is the full client round trip, including retries.

The original 42/64 Celeris ratings result was not a one-off: the same behavior scored 47/64 in a fresh control run. In the original run, 10 cases returned no usable judgment and nine of the 12 incorrect usable choices had tied top ratings. With malformed-digit retry and Choice tie resolution, two runs scored 57/64 and 56/64. Those runs resolved nine of ten and seven of seven top ties to the reference label. Their observed median and p90 latencies were higher; tie-break calls add work. The control and revised run 2 covered only the 64 news cases; revised run 1 covered all 112 requests. Jev rows are saved earlier paired runs on the same cases, while Celeris was measured later.

In a paired API-level run using the gem's one-call prompt, direct OpenAI was faster than latency-sorted OpenRouter. Both used temperature zero, reasoning disabled, a 1024-token output limit, and one corrective retry for malformed JSON. Four direct cases were also verified through the gem. All final judgments were usable.

In that paired run, direct OpenAI cut median latency by about 40% on both datasets, but lost one correct label on each and had worse Brier scores. Its SST-2 calls needed five corrective retries versus one through OpenRouter. A later direct-only repeat scored 59/64 news at 868 ms median and 28/32 SST-2 at 1,022 ms median, all usable. Just one news label and two sentiment labels changed across direct runs. OpenRouter was not rerun simultaneously with the repeat. Methods and both runs' case-level results.

A separate paired gem run sent the same one-call Luna judgments through OpenRouter's default route and provider: { sort: 'latency' }, rotating request order across cases. All responses were usable. Each AG News request asked one four-choice question; each SST-2 request asked Choice, Probability, and Score together. Both routes used temperature zero, reasoning disabled, and a 1024-token output limit. These results are separate from the runs above.

Latency sorting reduced median time by 20% on AG News and 28% on SST-2. It was faster on 50/64 and 21/32 matched cases, respectively. AG News lost one correct label and its Brier score worsened; SST-2 gained one correct label. Check this route on your own labeled decisions before relying on its probabilities. Methods and case-level data.

The gem uses RubyLLM's Judge API when it is present and the Legacy path otherwise. test/llm_judge_test.rb covers the Judge API and test/legacy_test.rb covers RubyLLM 1.13 through 2.0; each skips on the other side. test/engine_test.rb and test/http_test.rb run on every version; the HTTP tests send real RubyLLM requests to WebMock stubs and check their bodies.

bundle exec rake test                              # the latest released RubyLLM
RUBY_LLM_VERSION=1.13.2 bundle exec rake test      # a specific release
RUBY_LLM_VERSION=main bundle exec rake test        # RubyLLM's main branch, with the Judge API
RUBY_LLM_PATH=../ruby_llm bundle exec rake test    # a local checkout

CI runs Ruby 3.2 through 4.0 against RubyLLM 1.13.2, 1.16.0, 2.0.0 and 2.1.0, and Ruby 3.4 against RubyLLM main.