Home

Compare three models with the same question and context

Three models can produce three convincing answers. Which response matches a reference, and what time and usage does it take?

This is a proposed protocol, not a report of completed experiments. This guide was drafted with AI assistance; no model runs were performed for it.

Interface example, not benchmark results. The screenshot alone does not establish matched complete contexts, controlled conditions or independently measured accuracy.

Fix the conditions

For each case, use a fresh conversation and identical system instructions, question and context:

Use only the supplied context. Return only the requested JSON object. Do not infer missing information. Do not use tools.

Disable tools and retrieval. Save the effective request where available, not just the visible question. Record provider, model identifier/revision, date, output limit, temperature and seed if supported. For local models, record runtime, hardware and quantization. Mark unknown settings explicitly.

Run each of three models five times per case: 45 requests in total. Rotate model order and keep concurrency fixed. Record cache state and warm/cold conditions where observable; preserve failures. Provider usage may incur charges.

Three synthetic cases with answer keys

1. Arithmetic

Context: A: 12 units at 2.50; B: 8 units at 4.00; C: 3 units at 10.00. No tax or discounts.

Question: Return the total as a two-decimal string in the key total.

Reference: {"total":"92.00"}

Check: 12 × 2.50 + 8 × 4.00 + 3 × 10.00 = 92.00.

2. Extraction

Context: Order AX-17. Dispatch date: 2030-04-08. Customer: Nara. Carrier: not stated.

Question: Return order_id, dispatch_date and carrier. Use null for missing information.

Reference: {"order_id":"AX-17","dispatch_date":"2030-04-08","carrier":null}

3. Missing evidence

Reuse case 2’s context. Question: Return the stated tracking number in tracking_number, or null if absent.

Reference: {"tracking_number":null}

Log performance and score separately

Preserve every response, error, provider-reported input/output token count, and cached/reasoning tokens when supplied. Missing usage is unknown, not zero. Token counts depend on tokenizer and formatting; they do not directly determine price.

Measure total elapsed time from request submission until the complete response arrives. Record time to first token separately only when instrumentation supports it. Confirm what a displayed timer measures before using it in a comparison.

Score outside Studio: parse JSON, compare the complete key/value object against the reference, and ignore key order. Record invalid JSON and incorrect or additional values separately. Do not assume automatic reference scoring in Studio.

Keep one row per case, model and repetition. The row below is an empty template, not a measurement.

Case	Model/revision	Run	Input/output tokens	Total ms	Reference match	Error
—	—	—	—	—	—	—

Summarize correct attempts out of five, failures and timing spread separately. Preserve raw records. These small cases test specific requirements; they do not establish a general intelligence ranking.

Share the run with your team

Use Studio Community to discuss a comparison alongside its files and result cards. In a group inside your organization's workspace, share:

• The CSV/JSON report and original outputs or logs.
• A run identifier, code commit, prompt/checker revisions, model/provider and hardware/runtime.
• Timings, provider-reported usage, pass/fail/not-run outcomes and the reference or test used for scoring.

Keep your benchmark runner and checker responsible for producing the measurements and scores. Each post can link to the versioned export so the team can trace a disputed result.

Community membership and attachments currently use the same organization's access. A group with external collaborators needs an agreed shared workspace and member access.

Existing Studio demo/QA examples below: audio inference, CPU training and a dataset snapshot.

Evaluate the workspace

Try the Windows x64 workspace with the 30-day Professional evaluation. Configure your provider API account or a supported local model server. Explore Sinaps Studio and evaluation options. Compute access and model-provider consumption are separate from the Studio licence.

Further reading: NVIDIA LLM benchmarking metric definitions · Hugging Face tokenizer documentation.