Repository navigation
SkillRet Benchmark Catalog
Research Team edited this page Oct 10, 2026
·
3 revisions
This catalog covers 16 completed studies and bounded pilots in this repository. Rows describe recorded execution; a study can contain completed panels alongside failed, partial, superseded, or unrun arms. They are not 16 independent, fully completed evaluations of the entire dataset.
Reviewed 10 October 2026 against EXPERIMENTS.md and the matching study documentation/evidence at the reviewed source snapshot. No inference, training, paid request, or benchmark rerun was performed to prepare this catalog.
Environment labels: Local = documented local/on-machine execution; Cloud = documented remote compute or hosted inference; Hybrid = local preparation/retrieval/evaluation combined with cloud inference. Hardware and provider details are included only where recorded. The CPU archive documents its processor and runtime but not its physical hosting, so that row explicitly leaves Local/Cloud unresolved.
Unless stated otherwise, SkillRet refers to ThakiCloud/SKILLRET revision 6583d7d2ed07644d0fb8938ed8178f3a7dc42a12, with 4,392 test queries, 6,006 test skills, and 7,187 positive qrels. A full catalog, a full query count, or a separately frozen pilot does not make these previously exposed public evaluations blind. Different document views, context limits, candidate pools, and denominators are not directly comparable.
| # | Benchmark / study and sources | Execution environment | Dataset, query and catalog scope | Metrics actually recorded | Status / limitations |
|---|---|---|---|---|---|
| 1 | BM25 retrieval benchmark · metrics · executed implementation |
Hybrid for the recorded retrieval + selection package: in-process BM25, then hosted OpenAI gpt-4o-mini through Deep Agents. Retrieval alone uses no hosted model; local hardware is undocumented. |
50 generated single-target queries from the skill catalog, seed 42; name + description. Recorded retrieval sizes: 100 / 500 / 1,000 / 3,000 / 6,006. Selection: the same 50 queries, BM25 top 10, full catalog. |
Candidate Recall@1/3/5/10/20 (single-target hit fraction); selection candidate recall, exact-name and fuzzy-name match, mean selection latency. | Recorded historical run. The configuration lists size 50, but no size-50 retrieval row was produced: the implementation skips sizes no larger than the 50 gold skills. These generated queries are not the official 4,392-query panel; name-based single-target scoring is distinct from multi-positive qrel recall. Token usage and API cost are not recorded in this result package. |
| 2 | Full-listing benchmark · metrics · implementation |
Hybrid: local library construction/agent orchestration + hosted OpenAI gpt-4o-mini; local hardware undocumented. |
10 generated single-target queries, seed 42; all skill names/descriptions supplied directly. Eight recorded sizes: 25 / 50 / 100 / 200 / 400 / 800 / 1,200 / 1,800; 80 cases. | Exact-name and fuzzy-name match; average latency; error counts. | Recorded historical pilot, including a failed largest arm. At 1,800 skills, all 10 cases errored and match scores are zero; the short error-return latency is not successful inference latency. No full-6,006 listing result, recorded token total, or recorded API cost. |
| 3 | Initial SkillRet pilot · README · OpenRouter results | Hybrid: macOS/Python 3.12 CPU BM25 and BGE-small; local Ollama Qwen 3.5 4B/9B MLX; five hosted OpenRouter model/provider configurations documented in the README. | Full-document BM25: 200 fixed queries × 6,006 skills. Separate metadata/vector and five-query N=50 selector tests. Main hosted scaling: 3 queries × 3 sizes (50/250/1,000) × 4 methods × 5 models = 180 planned cases. Native activation checks: 25 separate cases. | BM25 Recall@5; selection precision, recall, F1, exact-set counts, validity/failures and coverage; activation/workflow completion and actual SKILL.md reads; provider-returned USD costs and budget accounting. |
Executed feasibility study with incomplete hosted coverage: 138/180 attempted, 121 valid selections, 24 exact matches; 53 cases reused. Activation: 15 completed workflows and 16 actual file reads. Reported valid-output means have conditional denominators. Retrieval, selection and activation panels are separate; downloaded skills were not executed. |
| 4 | Full-catalog RunPod benchmark · README · aggregate evidence | Cloud: RunPod NVIDIA A40, Python 3.12.3, PyTorch 2.8.0+cu128, vLLM 0.10.2. | All 4,392 test queries × all 6,006 test skills, same catalog for BM25, BGE-M3, frozen Qwen3-8B pooled embeddings, pretrained CLM heads and SkillRet-adapted CLM heads. | Recall@1/5/10, MRR (the saved reciprocal-rank metric, not MRR@5), Hit@1, NDCG@5, Completeness@5; paired bootstrap intervals. Runtime/report evidence also records timings and an observed RunPod credit difference. | Completed five-method study, one adaptation seed. Qwen3-8B stayed frozen; projection heads were trained, not a LoRA adapter. No reranker or generative selector followed retrieval. Synthetic public queries and incomplete judgments limit interpretation; observed credit change is not a settled invoice. |
| 5 | JevK5 LoRA benchmark · README · summary evidence | Cloud: RunPod NVIDIA A40, unquantized BF16 JevK5 backbone; BGE-M3 retrieval. |
Different revision: a050ad233a504a43135bafe8cdf45574052b5729, official training split. 10,123-skill metadata catalog; 2,000 selected training query groups and 256 validation queries. Natural BGE-M3 top 15 + NONE. Selected adapter completes 100 updates via resume. |
Recall, Hit, NDCG and Completeness@1/5/10; MRR@15; candidate recall/hit@15; oracle Recall@5; decision/NONE correctness, abstention and false abstention; paired bootstrap intervals. Resumed-stage time, peak allocated GPU memory, token-shortening audits and estimated compute/storage cost. | Completed bounded validation pilot. The first process stopped at update 67, with step 50 the last complete checkpoint; the completed resume selected step 100. Validation also selected the checkpoint. No official test evaluation or full-dataset training; shared training skill catalog, metadata-only view and natural shortlist ceiling. Cost is a historical estimate, not settled billing. |
| 6 | CPU-only benchmark · README · matched table · runtime receipt | Local/Cloud hosting undocumented. Direct CPU inference on Linux x86-64, Intel Xeon Platinum 8573C, 9 visible CPUs, affinity 0–3 and 4 numerical threads. No GPU or hosted API; provider/location not recorded. | Primary matched panel: 32 queries × 14 pipelines × 4 sizes (10/100/1,000/6,006) = 1,792 quality rows. Headers: name + first 1,024 description characters. Full-catalog timing: same 32 queries × 2 repeats. Retrieval64 and Julia/GLiNER48 are separate supplements. | MRR@5; Recall@1/5; Hit@1/5; NDCG@5; Complete@5; candidate recall/hit; warm latency mean/p50/p95; serial QPS; whole-process peak RSS; preparation/load timing and paired bootstrap intervals. | Completed matched-32 comparison. Smaller libraries preserve gold and vary by query; only the full-catalog arm has measured warm latency. Verdict48 stopped and Laya48 was not completed. Historical 4,392-query aggregates lack original raw records and are not paired with this panel. CPU quota/RAM enforcement and physical hosting are not established. |
| 7 | BM25 → Jev versus BM25 → CLM reranking · README · comparison evidence | Hybrid: Mac preparation, BM25 and evaluation; hosted TypeSafe SystemOne/Jev 1.13.0; CLM/Qwen3-8B BF16 inference on RunPod A40. | 200 matched public development queries, same 6,006-skill metadata catalog and identical natural BM25 top-20 pools; return top 5. Saved five-query/N=50 local and OpenRouter selectors are separate context. | Recall@5, Complete@5, candidate Recall@20; paired wins/losses/ties, service-failure counts and usage-based/compute cost estimates. Separate five-query selectors record precision, recall, exact-set matches and failures. | All 200 Jev and 200 CLM calls completed. Metadata-only exposed development result; missing top-20 gold cannot be recovered. Jev service and CLM GPU cost estimates use different accounting and are not settled invoices. Five-query, gold-preserving selector percentages must not be merged with the 200-query comparison. |
| 8 | SkillRet-Embedding-0.6B official reproduction · README · baseline evidence · environment | Local: Windows host, Ubuntu 26.04 WSL2; RTX 2060 SUPER, 8 GB VRAM; 2 guest CPUs / about 6 GB RAM. FP16, batch 1. |
All 4,392 test queries × all 6,006 full-text skills (name / description / skill_md); official instructed queries and 32,768-token limit; dense top 50. |
NDCG, Recall, Completeness, MAP and Precision@5/10/15; MRR@5/10; macro candidate recall and micro positive-pair coverage@20/50; deviation from official NDCG targets; active query latency, startup and resource/token-truncation evidence. | Completed; reproduction gate passed. NDCG@10 = 0.810990; maximum target deviation 0.000323 within 0.005 tolerance. Failed/interrupted attempts are retained; final verified resume completed all rankings. Public-data reproduction is exposed. Active query latency excludes the 50 ms safety pause and cache writes. |
| 9 | SkillRet + hosted Jev + RRF60 fusion · README · final findings · pilot evaluation | Hybrid: local full-text dense baseline and WSL preparation/evaluation; hosted TypeSafe Jev 1.13.0 scoring. | 10-query smoke subset and 200-query pilot, drawn from the 4,392-query test split; same 6,006-skill upstream retrieval catalog and fixed dense top-20 pools. Smoke is part of pilot, not an additional independent panel. 143/4,000 pilot candidate bodies right-truncated under the admission policy. | NDCG, Recall, Completeness, MAP and Precision@5/10/15; MRR@5/10; upstream candidate coverage@20/50; empty/failure rates and paired bootstrap intervals; input/output token usage, metered USD usage, held unknown cost, API timing and bounded wall-time evidence. | Smoke and pilot completed; full hosted expansion not run. Pilot NDCG@10: dense 0.830145, Jev 0.693412, RRF60 0.814578; no demonstrated NDCG improvement. 779 usable responses from 780 reservations, with 40 smoke responses reused. One lost original request remains execution/billing unknown after recovery. Do not sum overlapping scope usage or equate reservations/planning CAD values with invoices. |
| 10 | Local metadata/header retrieval benchmark · README · report · execution status | Local: RTX 2060 SUPER 8 GB, i5-9500T / 16 GB Windows host RAM; Ubuntu 26.04 WSL2 capped at 6 GB RAM / 2 CPUs / 2 GB swap. | Primary 4,392 corrected public queries × 6,006 headers, neural context capped at 1,024 tokens. BM25, Qwen and SkillRet plus RRF; separate 128-query screens include SkillRouter, SkillReason and TF-IDF. Candidate coverage at 20/50/100. | NDCG@10, Recall@10, Complete@10, MRR@5, failed/empty rate; candidate recall/completeness, rescued/displaced positives; warm encoding/search p50/p95, indexing/startup and resource evidence; exploratory cluster bootstrap intervals. | Primary full-query header comparison completed. SkillRet NDCG@10 0.730655 versus equal fusion 0.730792 is essentially flat. Metadata/1,024-token results are separate from row 8 full text. Full-body-1,024 and six-field runs were partial/resource-blocked; these are not completed full-query arms. |
| 11 | Local reranking and adaptation experiments · README · report · execution status | Local: same documented RTX 2060 SUPER / i5-9500T / WSL2 host as row 10; local native FP16 and quantized model deployments. | 32-query identical dense top-20 pools for SkillRet/SkillRouter/SkillReason/R3/Jina rerankers; separate 8-query JevK5/Plumb pointwise screen. Hard-negative rank-4 LoRA: 32-query development and separate 32-query transfer. Lexical adaptation: 512 training queries / 128 training-derived development queries. Additional bounded selection, 8K prompting and escalation diagnostics. | NDCG@10, Recall@10, Complete@10, MRR@5 and failures; selection NDCG@5, Complete@5, SetF1 and exact set; measured warm reranker p50/p95, context/invalid-output counts, escalation rate and exploratory paired cluster intervals. Escalation latency is a component-sum estimate. | Completed bounded panels, with explicit unfinished arms. Hard-negative LoRA development/transfer intervals include zero; corrected replay was interrupted and has no supported completed gain. Original replay/lexical defect results are superseded. Qwen8, local4B ranking, 21-option multipass, 32K prompting, external SRA/SkillReason full-catalog transfer and actual end-to-end escalation are blocked/incomplete or unrun. |
| 12 | Local rubric and hosted exploratory benchmarks · README · findings · rubric screen · local screen · candidate budget | Hybrid: local WSL2/CPU cached analysis and JevK5 Q5 on RTX 2060 SUPER; hosted TypeSafe Jev and partial OpenRouter calls. Provider GPU hardware is undocumented. | Distinct 12-query typed probe, separate 8-query RRF confirmation/selective/clause panels, 6-query varied-rubric screen (24 calls), 2-query local JevK5 screen (80 pointwise decisions), 2-case request-order check, and 4,392-query cached candidate-budget diagnostics over the 6,006-skill corpus. | Recall@5/10/20/30; NDCG@10; Completeness@5/30; macro/micro candidate coverage and pool-oracle NDCG@10; gains/losses/order changes; input tokens, known metered usage and reservations; API batch/local-ranking latency and failures; descriptive paired intervals. | Completed small exploratory panels and cached coverage analyses; some arms incomplete. Eight-query RRF NDCG gain does not establish full-panel improvement. OpenRouter: 4 successes, 1 HTTP error, 7 unattempted. No rubric qualified, so six conditional confirmation calls were not run. Local Plumb stopped before predictions; Qwen was not loaded. Pool-oracle scores are coverage ceilings, not measured reranker quality. Billing remains partially unreconciled. Distinct from row 9. |
| 13 | Seed-5150 scaling benchmark · README · audited report | Local: Linux/WSL2, 6 GiB RAM / 2 processors / 2 GiB swap, guarded sequential GPU execution. Hardware/provider detail beyond the report's runtime is not inferred. | 20 exposed queries × 17 retrieval methods × 6 sizes (10/50/100/250/1,000/6,006) = 2,040 retrieval rows. Nested gold-conditioned libraries. Two-query selector diagnostics across six sizes; separate top-20 reranking at 50/1,000/6,006. | Recall, Precision, NDCG and Completeness@5/10; MRR@5; selector precision/recall/F1/exact set, validity and failures; recorded component timings, logical tokens, cache provenance and resources; paired bootstrap and exact sign-flip test. | Audited preliminary, completed retrieval package with failed diagnostic arms. Full-catalog Recall@5: BM25 0.625, SkillRet 0.700, hybrid 0.800; paired sign-flip p=0.125. EG2/Jina historical producer provenance remains incomplete; vectors are excluded. Qwen8 failed before a full ranking and five cases were unattempted. Failures retain planned denominators; no broad selector ranking or complete fresh end-to-end latency claim. |
| 14 | EmbeddingGemma 2 benchmark · README · report | Local: RTX 2060 SUPER 8 GB, i5-9500T / 16 GB Windows host; Ubuntu 26.04 WSL2. Text-only FP32, eager attention, batch 1; 512-token cap. | 431 embedding calls including repeats/smoke. Bilingual fixture: 20 queries / 28 documents. SkillRet: 32 selected queries / 270 distinct candidate documents, from the 6,006-skill corpus; 34/60 gold pairs in the pool. Cached Qwen/SkillRet controls and fixed RRF60. | Fixture Recall@1/5 and MRR; SkillRet macro Recall@1/5/10, Completeness@5, pool ceiling and complementary gained/lost pairs; per-call wall/process-CPU timing, p50/p95, startup/load, input token/truncation counts and GPU/RSS/resource telemetry. | Successful compatibility and restricted-pool retrieval pilot. SkillRet Recall@5 52.08%, pool ceiling 56.77%. One query truncated at 512 tokens; Qwen control used 1,024. Compatibility failures retained separately. No full-catalog EG2 encoding or training; fixture and candidate-pool scores are not full-corpus quality. |
| 15 | Jina v5 text-nano benchmark · README · report | Local: RTX 2060 SUPER, Windows / Ubuntu 26.04 WSL2; retrieval adapter in FP32, eager attention, batch 1; 1,024-token cap. | 431 embedding calls; same bilingual 20-query/28-document fixture and SkillRet 32-query/270-document restricted pool as row 14. Historical Qwen, EG2 and SkillRet controls. Separate one-query EG2 1,024-token sensitivity check. | Fixture Recall@1/5 and MRR; SkillRet macro Recall@1/5/10, Completeness@5, complementary-pair/equal-budget analysis; warm wall/process-CPU p50/p95, startup/load, input-token/truncation and GPU/RSS/resource evidence. | Successful bounded pilot. Restricted-pool Recall@5 48.44% versus EG2 52.08% and SkillRet 56.77%; no truncated Jina inputs. Runtime comparisons include precision/configuration differences. The EG2 sensitivity check is separate and preserved original results. No full-catalog Jina quality claim or training. Recorded model license: CC-BY-NC-4.0. |
| 16 | SearchJev fine-tuning benchmark · README · report · paired evidence | Local: RTX 2060 SUPER (SM75) / WSL2; frozen Qwen3.5-0.8B text backbone with continued released rank-16 SearchJev adapter; FP16 autocast with FP32 adapters/loss. | Selected run pilot20-dynamic: 20 verified optimizer updates. Training: 32 queries / 66 balanced positive-negative pairs. Exposed development: 16 queries / 60 balanced pairs, identical before/after; header-only binary relevance, 256-token limit. |
Combined loss (CE + 0.5 sum-squared Brier), cross-entropy, Brier, threshold accuracy, ROC AUC; paired query-cluster intervals, before/after correctness changes; update/evaluation/wall time, token-truncation diagnostics and peak CUDA allocation/reservation. | Completed 20-update feasibility pilot, not retrieval evaluation. Accuracy 0.7667→0.7833 and AUC 0.8122→0.8228, while CE/Brier/combined loss worsened; no established generalization gain. Independent 2/2/5-update attempts must not be added to the selected trajectory. No 200-update/full training, blind test, Recall@5 or full-catalog retrieval result. |
- Recall@k usually averages the fraction of every query's positive qrels retrieved. Rows 1–2 use generated single-target, name-based scoring; their match/hit rates have a different meaning. Hit@1 and Recall@1 are not interchangeable when a query has several positives.
- Complete/Completeness@k means all labeled positives were retrieved. It differs from exact-set match, which also rejects extra selected IDs. MRR retains the cutoff used by the source: row 4's saved reciprocal rank must not be relabeled MRR@5, row 5 uses MRR@15, and other rows explicitly use MRR@5/10.
- Candidate recall, positive-pair coverage and pool-oracle scores describe the available pool or its ceiling. They do not establish downstream reranker accuracy or skill-execution success.
- Latency keeps its measured boundary: warm inference/search, API batch time, startup, or a component-sum estimate. Cache reuse is not zero-cost fresh serving. Unmeasured deployment end-to-end latency is not inferred from these components.
- Costs keep their evidence type: provider-returned costs, metered usage, estimates, observed credit differences, and held reservations are distinct. Overlapping/reused smoke and pilot calls are counted once. Missing usage and uncertain charges remain unknown.
- Completed applies only to the stated panel. Invalid outputs, context errors, resource stops, interrupted checkpoints and unattempted cases remain visible. Code support, offline fixture tests, or a planned expansion do not count as a completed experiment.
The authoritative implemented/incomplete/unrun section and metadata execution table preserve the full hosted expansion, corrected replay, full-body/six-field partial runs, 32K prompt diagnostic, external SRA/SkillReason transfer, multipass selection and actual escalation-latency study as unrun or incomplete. Later small Qwen8/scaling diagnostics do not complete the earlier blocked experiments. The user-PDF inventory remains pending input bytes and is not a benchmark. Consolidation/publication/offline validation is not an additional inference run.
Folder links below open the current source tree. Documentation/evidence links use the reviewed immutable commit so the basis of this catalog remains inspectable if main changes. EXPERIMENTS.md controls the recent-study boundaries; the preserved older studies and BM25 historical evidence supply rows not individually enumerated there.