v0.0.3 - 'qt add' to customize your own benchmark from the remote benchmark registry

This is a small addition to v0.0.2, which added the benchmark registry and custom configuration benchmarks of type similarity, that adds a new qt add command. You can use this command to download the definition of a benchmark from the remote registry and add it to your local quantiles.toml configuration file (or create one if you didn't already have one). After you qt add a remote benchmark, you can customize it in your configuration file and treat it as your own locally-defined evaluation.

What's Changed

• Adding qt add command by @arschles in #73

Full Changelog: v0.0.2...v0.0.3

v0.0.2 - Introducing the Remote Benchmark Registry

This is the second release of the Quantiles open-source CLI. The main feature added in this release is the benchmark registry. Prior to this release, three benchmarks were hard-coded into the CLI. This release removes those hard-coded benchmarks and instead can download custom_nocode benchmarks from the Quantiles hosted benchmark registry.

Install the CLI with the following command:

curl -fsSL https://cli.quantiles.io/install.sh | bash

As of this release, the hosted registry contains pubmedqa, simpleqa-verified, and the benchmarks in the quantiles.toml custom no-code demo file. The full list of benchmark slugs (that you can run with qt run <eval_name>) is below:

• simpleqa-verified
• medqa
• medmcqa
• mmlu-pro
• gpqa
• pubmedqa

The result of this new feature is that the CLI appears to have more built-in benchmarks. It also allows us to publish more benchmarks transparently and without updates to the CLI itself. Existing published benchmarks will not be changed, but we may publish new versions of existing benchmarks from time to time.

What's Changed

• Fixing links to custom no-code evaluations by @arschles in #50
• Add badges and links by @phranzia in #51
• Add model configuration documentation link and description by @phranzia in #54
• Add model configuration guide to What's New by @phranzia in #55
• Validating prompt templates without actually rendering them by @arschles in #62
• Adding client for benchmark registry by @arschles in #57
• Adding similarity metric evals to custom no-code by @arschles in #65
• Making runs started from the remote benchmark registry resumable with strict version provenance by @arschles in #66
• Removing built-in (to code) custom benchmarks by @arschles in #67

Full Changelog: v0.0.1...v0.0.2

v0.0.1 - First GitHub release, and adding custom no-code benchmarks.

Summary

This is the first GitHub release in the Quantiles open-source CLI. It adds custom_nocode evaluations, which allow you to build your own evaluations without writing or maintaining any code. For details, please see custom no-code evaluations documentation. To install the Quantiles CLI, please run the below command, then see the quickstart documentation for next steps:

curl -fsSL https://cli.quantiles.io/install.sh | bash

What's Changed

• skill tweaks by @arschles in #1
• Adding some minor cleanups to the readme by @arschles in #2
• Root AGENTS.md by @phranzia in #4
• move typescript, python, cli by @phranzia in #5
• Lots of minor tweaks to README.md by @arschles in #3
• Minor GH actions cleanups etc by @arschles in #6
• quantiles-os => quantiles, and fixing Citation.cff by @arschles in #8
• agent.md and other docs tweaks by @arschles in #9
• Adding qt resume command by @arschles in #10
• Marking TypeScript SDK as pre-release by @arschles in #12
• Preparing to publish Python wheel, and moving python examples to python-examples/ top-level dir by @arschles in #13
• [codex] Mention llms.txt in agent docs by @phranzia in #15
• Add info on adding built-in benchmarks by @phranzia in #17
• Minor doc updates by @phranzia in #18
• Replace raw file paths in docs with clear text by @phranzia in #21
• Update README introduction by @phranzia in #23
• Adding a --version command by @arschles in #44
• Adding GitHub workflow for building/publishing the CLI by @arschles in #22
• Adding upload-beta-install-script logic by @arschles in #46
• Implementing custom_nocode benchmarks for deterministic benchmarks by @arschles in #14

New Contributors

• @phranzia made their first contribution in #4

Full Changelog: https://github.com/quantiles-evals/quantiles/commits/v0.0.1