v0.6.0 — Model recommendation with reasoning_effort support

[justi]

Summary

• Step.recommend("eval", candidates: [...]) — runs eval on all candidates and returns Recommendation with optimal model, retry chain, rationale, savings vs current config, and to_dsl code output
• Candidates as configurations — { model:, reasoning_effort: } hashes, not just model name strings
• Per-attempt reasoning_effort in retry policies — escalate accepts config hashes, each attempt gets its own reasoning_effort forwarded to the provider
• compare_models extended with candidates: parameter (backward compatible, models: still works)
• pass_rate_ratio (float 0.0–1.0) on Report/ReportStats
• History entries enriched with model, reasoning_effort, pass_rate_ratio

Test plan

• 1279 tests pass, 0 failures
• RubyCritic score 88.06
• Two code review rounds (panel review + deep audit), all findings addressed
• Two Copilot review rounds, all 9 comments addressed
• Weak test audit: 13 findings fixed — exact value assertions, full ReportStats coverage, normalize_candidate_config spec, no rand() in helpers
• Backward compatibility: existing models: %w[...] API unchanged
• retry_policy models: %w[a b] still works identically
• No mentions of o3 or non-GPT models in examples
• README claims verified against code — no unsubstantiated numbers, honest wording

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR bumps the gem to v0.6.0 and introduces a model/config recommendation workflow driven by eval results, including support for per-attempt reasoning_effort in retry policies and richer eval/report metadata.

Changes:

• Add Step.recommend(eval, candidates: [...]) plus new Eval::Recommender/Eval::Recommendation to return a best candidate, retry chain, rationale, and DSL output.
• Extend compare_models to accept candidates: (config hashes) and enrich ModelComparison/history entries with reasoning_effort + pass_rate_ratio.
• Upgrade retry policies/execution to use config-based escalation ({ model:, reasoning_effort: }) and forward per-attempt options to providers.

Reviewed changes

Copilot reviewed 24 out of 25 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
spec/ruby_llm/contract/step/retry_policy_spec.rb	Adds specs for config-based escalation normalization and attempt selection.
spec/ruby_llm/contract/step/retry_executor_spec.rb	Adds specs ensuring per-attempt reasoning_effort forwarding and trace behavior.
spec/ruby_llm/contract/step/base_spec.rb	Adds specs for .recommend and determining current model config.
spec/ruby_llm/contract/eval/report_stats_spec.rb	Adds tests for numeric pass_rate_ratio.
spec/ruby_llm/contract/eval/recommender_spec.rb	Adds tests for recommendation selection, retry chain, savings, warnings.
spec/ruby_llm/contract/eval/recommendation_spec.rb	Adds tests for immutability and DSL output generation.
spec/ruby_llm/contract/eval/model_comparison_spec.rb	Adds tests for candidate labeling/config mapping and pass_rate_ratio in to_h.
lib/ruby_llm/contract/version.rb	Bumps gem version to 0.6.0.
lib/ruby_llm/contract/step/retry_policy.rb	Switches from model lists to config lists (hash/string normalization).
lib/ruby_llm/contract/step/retry_executor.rb	Forwards per-attempt config extras (e.g., reasoning_effort) and adds config to trace.
lib/ruby_llm/contract/step/base.rb	Adds .recommend and a helper to determine current model config.
lib/ruby_llm/contract/eval/report.rb	Delegates pass_rate_ratio from stats.
lib/ruby_llm/contract/eval/report_storage.rb	Adds reasoning_effort to paths/entries and persists pass_rate_ratio.
lib/ruby_llm/contract/eval/report_stats.rb	Implements numeric pass_rate_ratio.
lib/ruby_llm/contract/eval/recommender.rb	New recommendation algorithm selecting best/first-try chain and computing savings/warnings.
lib/ruby_llm/contract/eval/recommendation.rb	New Recommendation value object + to_dsl output.
lib/ruby_llm/contract/eval/model_comparison.rb	Adds candidate labeling and config mapping support (vs model-only keys).
lib/ruby_llm/contract/eval.rb	Requires new recommender/recommendation files.
lib/ruby_llm/contract/concerns/eval_host.rb	Extends compare_models to accept candidates: and carry reasoning_effort into eval runs.
lib/ruby_llm/contract.rb	Adds normalize_candidate_config helper used across retry + eval flows.
README.md	Documents recommendation API and updates roadmap.
docs/guide/getting_started.md	Updates reasoning_effort docs language.
docs/architecture.md	Adds new recommender/recommendation components.
CHANGELOG.md	Adds v0.6.0 entry describing new features.
Gemfile.lock	Bumps locked gem version to 0.6.0.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

normalize_candidate_config returns the input Hash as-is, which means downstream callers that freeze configs (e.g., RetryPolicy/escalate and Recommendation) will end up freezing user-owned hashes. It also doesn’t validate that a Hash candidate actually contains a usable :model (or handle string keys), which can lead to model=nil being passed into adapters.

Consider returning a normalized copy (e.g., symbolize keys, require a non-empty :model String, and drop nil optional keys like reasoning_effort) so internal code can safely freeze without mutating caller objects and so invalid candidates fail fast with a clear ArgumentError.

Suggested change

	when String then { model: entry }
	when Hash then entry
	else raise ArgumentError, "Expected String or Hash, got #{entry.class}"
	when String
	model = entry.strip
	raise ArgumentError, "Candidate config model must be a non-empty String" if model.empty?
	{ model: model }
	when Hash
	candidate = entry.each_with_object({}) do |(key, value), normalized|
	normalized[key.respond_to?(:to_sym) ? key.to_sym : key] = value
	end
	model = candidate[:model]
	unless model.is_a?(String) && !model.strip.empty?
	raise ArgumentError, "Candidate config must include a non-empty String :model"
	end
	normalized = { model: model.strip }
	normalized[:reasoning_effort] = candidate[:reasoning_effort] unless candidate[:reasoning_effort].nil?
	normalized
	else
	raise ArgumentError, "Expected String or Hash, got #{entry.class}"

[Copilot]

escalate freezes each config hash. If the caller passes a Hash (not a String), this currently freezes the same object they provided (because normalize_candidate_config returns the Hash unchanged). That’s a surprising side effect and can break callers that reuse or mutate their config hashes.

Prefer duplicating/normalizing configs before freezing (e.g., dup + symbolize/validate in the normalizer, or deep-dup-freeze here) so RetryPolicy never mutates caller-owned objects.

[Copilot]

Recommendation is intended to be an immutable value object (it freezes itself), but retry_chain, savings, and warnings are only shallow-frozen; the nested Hashes/Arrays remain mutable, so callers can still mutate internal state through rec.retry_chain.first[:model] = ..., etc.

Consider using the existing Concerns::DeepFreeze helper (deep-dup-freeze) for these attributes so the object is deeply immutable without freezing caller-owned objects.

Suggested change

	attr_reader :best, :retry_chain, :score, :cost_per_call,
	:rationale, :current_config, :savings, :warnings
	def initialize(best:, retry_chain:, score:, cost_per_call:,
	rationale:, current_config:, savings:, warnings:)
	@best = best&.freeze
	@retry_chain = retry_chain.freeze
	@score = score
	@cost_per_call = cost_per_call
	@rationale = rationale.freeze
	@current_config = current_config&.freeze
	@savings = savings.freeze
	@warnings = warnings.freeze
	include Concerns::DeepFreeze
	attr_reader :best, :retry_chain, :score, :cost_per_call,
	:rationale, :current_config, :savings, :warnings
	def initialize(best:, retry_chain:, score:, cost_per_call:,
	rationale:, current_config:, savings:, warnings:)
	@best = best&.freeze
	@retry_chain = deep_dup_freeze(retry_chain)
	@score = score
	@cost_per_call = cost_per_call
	@rationale = rationale.freeze
	@current_config = current_config&.freeze
	@savings = deep_dup_freeze(savings)
	@warnings = deep_dup_freeze(warnings)

[Copilot]

build_rationale sorts candidates purely by cost_per_call, but candidates with unknown pricing are represented with cost_per_call = 0.0, so they will sort to the top and appear “cheapest” in the rationale despite being excluded from selection and flagged as unknown pricing.

Consider sorting unknown-pricing candidates last (e.g., sort key [cost_known? ? cost_per_call : Float::INFINITY, latency, label]) and/or formatting unknown costs as “n/a” to avoid misleading rationale ordering/output.

[Copilot]

The warning message still says “no models”, but the API is now config-based (@configs, config_list). Updating the wording (and the suggested DSL snippet) to reflect configs would make the warning clearer for users reading logs.

Suggested change

	warn "[ruby_llm-contract] retry_policy has max_attempts=1 with no models. " \
	"This means no actual retry will happen. Add `attempts 2` or `escalate %w[model1 model2]`."
	warn "[ruby_llm-contract] retry_policy has max_attempts=1 with no configs. " \
	"This means no actual retry will happen. Add `attempts 2` or " \
	"`escalate [{ model: \"model1\" }, { model: \"model2\" }]`."

[Copilot]

Pull request overview

Copilot reviewed 24 out of 25 changed files in this pull request and generated 4 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Hash#except is not part of Ruby core (Ruby >=3.2 has Hash#slice but not except), so config.except(:model) will raise NoMethodError in production. Replace this with a core-Ruby approach (e.g., config.reject { |k,_| k == :model } or dup+delete) before merging into extra_options.

Suggested change

	attempt_extra = extra_options.merge(config.except(:model))
	attempt_extra = extra_options.merge(config.reject { |k, _| k == :model })

[Copilot]

Treating missing latency as 0 (report.avg_latency_ms || 0) will bias tiebreaking toward candidates with unknown latency (they’ll look fastest). Consider using a sentinel like Float::INFINITY (or keeping nil and sorting with nil last) so “unknown” doesn’t win latency comparisons.

Suggested change

	latency: report.avg_latency_ms || 0,
	latency: report.avg_latency_ms || Float::INFINITY,

[Copilot]

When pricing is unknown, cost_per_call becomes 0.0, so rationale lines currently render as at $0.0000/call even though the same candidate is flagged as “unknown pricing”. Consider formatting cost as n/a (or similar) when cost_known? is false to avoid misleading output.

Suggested change

	header = "#{candidate[:label]}, score #{format("%.2f", candidate[:score])}, at $#{format("%.4f", candidate[:cost_per_call])}/call"
	notes = rationale_notes(candidate, best)
	notes.any? ? "#{header} — #{notes.join(", ")}" : header
	end
	header = "#{candidate[:label]}, score #{format("%.2f", candidate[:score])}, at #{format_cost_per_call(candidate)}/call"
	notes = rationale_notes(candidate, best)
	notes.any? ? "#{header} — #{notes.join(", ")}" : header
	end
	def format_cost_per_call(candidate)
	return "n/a" unless cost_known?(candidate)
	"$#{format("%.4f", candidate[:cost_per_call])}"
	end

[Copilot]

Recommendation includes Concerns::DeepFreeze, but @rationale = rationale.freeze and @warnings = warnings.freeze are only shallow freezes and keep the original object references. To ensure the object is truly immutable (and consistent with other DeepFreeze users), consider using deep_dup_freeze for rationale and warnings as well.

Suggested change

	@rationale = rationale.freeze
	@current_config = deep_dup_freeze(current_config)
	@savings = deep_dup_freeze(savings)
	@warnings = warnings.freeze
	@rationale = deep_dup_freeze(rationale)
	@current_config = deep_dup_freeze(current_config)
	@savings = deep_dup_freeze(savings)
	@warnings = deep_dup_freeze(warnings)

[Copilot]

Pull request overview

Copilot reviewed 25 out of 26 changed files in this pull request and generated no new comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.