Repository navigation
TrustLLM 0.4.0 adds a shared local/API generation workflow, a lightweight Python installation, and reproducible run artifacts while retaining the original benchmark scorers.
Included
trustllmandpython -m trustllmcommands for dataset download, task discovery, generation and evaluation.- OpenAI-compatible text Chat Completions and local Hugging Face causal-model backends, with optional
localandevaldependencies. - Per-sample checkpoints, guarded resume, bounded retries, explicit failure handling, input hashes and environment metadata.
- JSON responses compatible with the existing scorers, plus separate HTML generation and score reports.
- Refreshed documentation, seven README languages, and CLI/Python integration guidance for AI agents.
- A tested wheel and source distribution, downloadable from this GitHub Release.
Install from GitHub
Install the wheel directly, without cloning the repository or installing Git:
python -m pip install "https://github.com/HowieHwong/TrustLLM/releases/download/v0.4.0/trustllm-0.4.0-py3-none-any.whl"
# Local models and scoring:
python -m pip install "trustllm[local,eval] @ https://github.com/HowieHwong/TrustLLM/releases/download/v0.4.0/trustllm-0.4.0-py3-none-any.whl"
python -m trustllm download --output data
python -m trustllm tasksFor the maintained source version, with Git installed:
python -m pip install "trustllm @ git+https://github.com/HowieHwong/TrustLLM.git@main#subdirectory=trustllm_pkg"Replace main with a reviewed commit SHA to pin an experiment environment.
Migration and scope
Read the migration guide for changed output directories, model IDs and error handling. The archived provider-specific engine remains available through the legacy extra. Generation settings differ from historical paper runs; the release does not claim exact reproduction of published scores.
--limit is a smoke-test aid. Generation reports are not trustworthiness scores. Full scoring can require downloaded classifiers and paid judge or embedding APIs. Agent integration covers orchestration of model-response evaluation, not tool-use or agent-trajectory benchmarks.
Validation
Local validation passed all 52 tests, including CPU model and scorer integration fixtures. A fresh base-only wheel environment passed 46 tests, with six optional-dependency tests skipped. Wheel and sdist metadata passed validation; documentation passed strict build and checks for all 13 pages. See CI for Python 3.9–3.12 checks. No paid live-model evaluation was performed for this release.