SkillReason: Reasoning-Enhanced Agent Skill Retrieval for Implicit User Requests

SkillReason is a reasoning-enhanced dense retriever for selecting reusable agent skills from natural-language requests. It is designed for implicit requests that describe a task goal without naming the required skill or spelling out the execution procedure.

The training framework uses capability reasoning as privileged supervision, then optimizes reasoning trajectories with retrieval feedback. At deployment, this reasoning ability is internalized in the query representation: retrieval remains query-only and does not require autoregressive chain-of-thought generation.

Released Artifacts

All released checkpoints and SkillReason-Bench data are hosted on Hugging Face.

Models

Component	Size	Hugging Face repository
SkillReason retriever	0.6B	donghongjiang/SkillReason-embedding-0.6b
SkillReason retriever	4B	donghongjiang/SkillReason-embedding-4b
SkillReason reranker	0.6B	donghongjiang/SkillReason-reranker-0.6b
SkillReason reranker	4B	donghongjiang/SkillReason-reranker-4b

Evaluation Data

Benchmark	Repository	Public adapter
SkillReason-Bench	donghongjiang/skillreason-bench	skillreason
SRA-Bench	WeihangSu/SRA-Bench	sragents
SkillRet	ThakiCloud/SKILLRET	skillret
SkillBench Core	pipizhao/SkillRouter-Eval-Core	skillbench

The downloader uses pinned revisions by default and writes the resolved local paths to artifacts/download_manifest.json. Set HF_TOKEN or run hf auth login when authentication or higher Hugging Face rate limits are required.

Highlights

• Query-only deployment. Encode each request once and search against precomputed skill embeddings.
• Large-scale skill retrieval. SkillReason-Bench contains 3,729 requests and 61,228 candidate skills; exact cosine search, deterministic embedding caches, and data-parallel encoding are supported out of the box.
• Retriever and reranker releases. Both 0.6B and 4B checkpoints are available for retrieval-only or retrieve-then-rerank pipelines.
• Native benchmark support. Built-in adapters cover SkillReason-Bench, SRA-Bench, SkillRet, and SkillBench Core.
• Pinned artifacts and protocols. The downloader records exact model and dataset revisions, while evaluation outputs record the resolved protocol.

Installation

SkillReason requires Python 3.10 or newer. The reported experiments used Python 3.10, PyTorch 2.6.0, and Transformers 4.51.3.

git clone https://github.com/donghong1/SkillReason.git
cd SkillReason

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
pip install -e .

Install a PyTorch build compatible with the host CUDA runtime. FlashAttention is optional and is not required by the public evaluation path.

Quick Start

1. Download a retriever

skillreason-download \
  --artifact retriever-0.6b \
  --output-dir artifacts

Use --artifact retriever-4b for the 4B retriever, or inspect all available artifacts without downloading:

skillreason-download --artifact all --output-dir artifacts --dry-run

2. Retrieve skills for user requests

The repository includes a minimal skill corpus and query file under examples/:

skillreason-retrieve \
  --model artifacts/models/SkillReason-embedding-0.6b \
  --backend hf_last_token \
  --corpus examples/skills.jsonl \
  --queries examples/queries.jsonl \
  --output-dir outputs/quickstart/retrieval \
  --corpus-cache outputs/cache/example-skills.npy \
  --query-prefix official \
  --devices 0 \
  --max-length 4096 \
  --top-k 3

The command writes:

• predictions.jsonl: ranked skill IDs and cosine-similarity scores for each query;
• metadata.json: model, precision, prefix, length, batch, device, and cache settings used by the run.

Skill documents are cached independently of the queries. Reusing --corpus-cache avoids re-encoding an unchanged skill library.

3. Optionally rerank the retrieved candidates

skillreason-download \
  --artifact reranker-0.6b \
  --output-dir artifacts

skillreason-rerank \
  --model artifacts/models/SkillReason-reranker-0.6b \
  --candidates outputs/quickstart/retrieval/predictions.jsonl \
  --corpus examples/skills.jsonl \
  --output-dir outputs/quickstart/reranked \
  --devices 0 \
  --top-n 3

The reranker scores query-skill pairs with the released yes/no relevance prompt and writes a new ranked predictions.jsonl plus metadata.json.

Evaluate Released Benchmarks

The shared script downloads the selected retriever and benchmark when DOWNLOAD=1, applies the benchmark-specific protocol, and writes predictions and metrics. Change only BENCHMARK to select an adapter:

DOWNLOAD=1 \
MODEL_SIZE=0.6b \
BENCHMARK=skillreason \
DEVICES=0,1,2,3,4,5,6,7 \
bash scripts/evaluate_benchmark.sh

Valid values are skillreason, sragents, skillret, and skillbench. Equivalent single-benchmark scripts are also provided:

DOWNLOAD=1 bash scripts/evaluate_skillreason.sh
DOWNLOAD=1 bash scripts/evaluate_sra_bench.sh
DOWNLOAD=1 bash scripts/evaluate_skillret.sh
DOWNLOAD=1 bash scripts/evaluate_skillbench.sh

All benchmark scripts accept the same environment overrides:

Variable	Default	Purpose
MODEL_SIZE	0.6b	Released retriever scale: 0.6b or 4b
BENCHMARK	skillreason	Benchmark adapter used by the shared runner
DEVICES	0	Comma-separated CUDA devices, cpu, or auto
DOWNLOAD	0	Download the selected pinned model and dataset before evaluation
ARTIFACT_ROOT	artifacts	Local model and benchmark root
OUTPUT_ROOT	outputs/evaluation	Root for predictions, metrics, and caches
CORPUS_BATCH_SIZE	protocol default	Per-device skill encoding batch size
QUERY_BATCH_SIZE	protocol default	Per-device query encoding batch size

Each run produces predictions.jsonl and metrics.json. Corpus caches are content-addressed by model identity, rendered documents, precision, sequence length, batch size, and device geometry, so incompatible caches are not reused.

For direct CLI control, use skillreason-evaluate-benchmark --help. For a custom labeled JSONL dataset, use skillreason-evaluate instead of a native benchmark adapter.

Input Data

Custom inputs use UTF-8 JSONL with one object per line.

Skill corpus:

{"skill_id":"adaptive-mpc","name":"Adaptive MPC","description":"Robust predictive control","body":"Design and tune predictive controllers under changing system parameters."}

Unlabeled query:

{"query_id":"request-001","query":"Stabilize a robot arm while its payload changes during operation."}

For labeled evaluation, add gold_skill_ids to each query. See the User Guide for accepted aliases, graded relevance, reranker inputs, and native benchmark layouts.

Documentation

The SkillReason User Guide provides the complete reference for artifact downloads, JSONL schemas, custom inference, reranking, benchmark adapters, the released protocol, multi-GPU execution, and troubleshooting.

Repository Structure

SkillReason/
|-- src/skillreason/
|   |-- inference/          # Retrieval, reranking, encoding, and caches
|   |-- evaluation/         # Metrics and native benchmark adapters
|   |-- download.py         # Pinned Hugging Face artifact downloader
|   `-- prompts.py          # Public query and reranking prompts
|-- scripts/                # Ready-to-run shell entry points
|-- examples/               # Minimal custom inference inputs
|-- docs/                   # Consolidated public user guide
`-- tests/                  # CPU-only unit, CLI, packaging, and hygiene tests

Development

pip install -e ".[dev]"
pytest
ruff check .
ruff format --check src tests
python -m build

See CONTRIBUTING.md before submitting a change. GPU tests are optional; the default test suite is CPU-only.

Give Us a Star 👏

If you find SkillReason useful for your research or applications, please consider giving the repository a star ⭐. Thank you for your support!