Skip to content

v0.4.0 · Local/API evaluation workflow

Latest

Choose a tag to compare

@HowieHwong HowieHwong released this 04 Oct 23:18

TrustLLM 0.4.0 adds a shared local/API generation workflow, a lightweight Python installation, and reproducible run artifacts while retaining the original benchmark scorers.

Included

  • trustllm and python -m trustllm commands for dataset download, task discovery, generation and evaluation.
  • OpenAI-compatible text Chat Completions and local Hugging Face causal-model backends, with optional local and eval dependencies.
  • Per-sample checkpoints, guarded resume, bounded retries, explicit failure handling, input hashes and environment metadata.
  • JSON responses compatible with the existing scorers, plus separate HTML generation and score reports.
  • Refreshed documentation, seven README languages, and CLI/Python integration guidance for AI agents.
  • A tested wheel and source distribution, downloadable from this GitHub Release.

Install from GitHub

Install the wheel directly, without cloning the repository or installing Git:

python -m pip install "https://github.com/HowieHwong/TrustLLM/releases/download/v0.4.0/trustllm-0.4.0-py3-none-any.whl"
# Local models and scoring:
python -m pip install "trustllm[local,eval] @ https://github.com/HowieHwong/TrustLLM/releases/download/v0.4.0/trustllm-0.4.0-py3-none-any.whl"
python -m trustllm download --output data
python -m trustllm tasks

For the maintained source version, with Git installed:

python -m pip install "trustllm @ git+https://github.com/HowieHwong/TrustLLM.git@main#subdirectory=trustllm_pkg"

Replace main with a reviewed commit SHA to pin an experiment environment.

Migration and scope

Read the migration guide for changed output directories, model IDs and error handling. The archived provider-specific engine remains available through the legacy extra. Generation settings differ from historical paper runs; the release does not claim exact reproduction of published scores.

--limit is a smoke-test aid. Generation reports are not trustworthiness scores. Full scoring can require downloaded classifiers and paid judge or embedding APIs. Agent integration covers orchestration of model-response evaluation, not tool-use or agent-trajectory benchmarks.

Validation

Local validation passed all 52 tests, including CPU model and scorer integration fixtures. A fresh base-only wheel environment passed 46 tests, with six optional-dependency tests skipped. Wheel and sdist metadata passed validation; documentation passed strict build and checks for all 13 pages. See CI for Python 3.9–3.12 checks. No paid live-model evaluation was performed for this release.