v0.0.2 - Introducing the Remote Benchmark Registry

This is the second release of the Quantiles open-source CLI. The main feature added in this release is the benchmark registry. Prior to this release, three benchmarks were hard-coded into the CLI. This release removes those hard-coded benchmarks and instead can download custom_nocode benchmarks from the Quantiles hosted benchmark registry.

Install the CLI with the following command:

curl -fsSL https://cli.quantiles.io/install.sh | bash

As of this release, the hosted registry contains pubmedqa, simpleqa-verified, and the benchmarks in the quantiles.toml custom no-code demo file. The full list of benchmark slugs (that you can run with qt run <eval_name>) is below:

• simpleqa-verified
• medqa
• medmcqa
• mmlu-pro
• gpqa
• pubmedqa

The result of this new feature is that the CLI appears to have more built-in benchmarks. It also allows us to publish more benchmarks transparently and without updates to the CLI itself. Existing published benchmarks will not be changed, but we may publish new versions of existing benchmarks from time to time.

What's Changed

• Fixing links to custom no-code evaluations by @arschles in #50
• Add badges and links by @phranzia in #51
• Add model configuration documentation link and description by @phranzia in #54
• Add model configuration guide to What's New by @phranzia in #55
• Validating prompt templates without actually rendering them by @arschles in #62
• Adding client for benchmark registry by @arschles in #57
• Adding similarity metric evals to custom no-code by @arschles in #65
• Making runs started from the remote benchmark registry resumable with strict version provenance by @arschles in #66
• Removing built-in (to code) custom benchmarks by @arschles in #67

Full Changelog: v0.0.1...v0.0.2