Skip to content

Add RAG benchmark track plans - #9

Merged
muuuuuuushroom merged 1 commit into
mainfrom
agent/add-rag-track
Jul 30, 2026
Merged

Add RAG benchmark track plans#9
muuuuuuushroom merged 1 commit into
mainfrom
agent/add-rag-track

Conversation

@muuuuuuushroom

Copy link
Copy Markdown
Collaborator

What changed

  • Add the RAG benchmark planning documents for versions 0.2 through 0.4.
  • Add research covering scope, primary evidence, data volume, and metrics.
  • Keep corpus PDFs, references, temporary files, and generated artifacts out of the repository.

Why

RAG is a first-class benchmark track alongside the existing tracks in moi-benchmark. Keeping its reproducible plans and research under rag/ provides one source of truth without creating a permanently divergent branch or committing large/private corpus files.

Impact

Contributors can review and evolve the RAG benchmark from the shared benchmark repository while managing raw datasets and run artifacts separately.

Validation

  • Confirmed the branch is based on origin/main.
  • Confirmed only rag/plans/ and rag/research/ additions are included.
  • Confirmed corpus and temporary binary files are excluded.

@muuuuuuushroom
muuuuuuushroom marked this pull request as ready for review July 30, 2026 03:32
Copilot AI review requested due to automatic review settings July 30, 2026 03:32
@muuuuuuushroom
muuuuuuushroom merged commit 97156f3 into main Jul 30, 2026
1 check passed

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds the RAG track’s planning and research documents (v0.2–v0.4) under rag/, including a detailed v0.3 research recommendation on sample size/metrics, and a v0.4 one-week single-operator pilot plan with executable TODO templates.

Changes:

  • Added research notes covering scope/identity, primary evidence review, and v0.3 data volume + metrics recommendations.
  • Added v0.2–v0.4 draft benchmark plans, including a scoped v0.4 pilot plan and supporting TODO/acceptance templates.
  • Added v0.3 execution TODO breakdowns (research/data/system/acceptance/master plan) to operationalize the v0.3 plan.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
rag/research/v0.3-data-volume-and-metrics-recommendation.md v0.3 sample size + metric layering recommendations and rationale.
rag/research/v0.2-primary-evidence-review.md Primary-source review and reuse boundaries for public RAG benchmarks.
rag/research/moi-rag-platform-benchmark-scope.md Scope/identity note for “MOI” and benchmark boundary evidence.
rag/plans/drafts/v0.4.md v0.4 one-week MOI-only feasibility pilot plan and contracts.
rag/plans/drafts/v0.4-todos/README.md v0.4 TODO navigation and status conventions.
rag/plans/drafts/v0.4-todos/01-five-day-execution-plan.md v0.4 day-by-day execution plan with DoD and gates.
rag/plans/drafts/v0.4-todos/02-acceptance-and-result-template.md v0.4 acceptance checklist + result reporting templates.
rag/plans/drafts/v0.3.md v0.3 product-grade benchmark plan draft (TDAS, gates, artifacts, CI).
rag/plans/drafts/v0.3-todos/README.md v0.3 TODO navigation, roles, and crosswalk for effort accounting.
rag/plans/drafts/v0.3-todos/01-master-execution-plan.md v0.3 master execution plan with gates and schedule.
rag/plans/drafts/v0.3-todos/02-research-plan.md v0.3 decision-oriented research plan and evidence standards.
rag/plans/drafts/v0.3-todos/03-dataset-and-evaluation-plan.md v0.3 dataset/Gold/Judge/audit/bootstrap execution plan.
rag/plans/drafts/v0.3-todos/04-system-and-run-plan.md v0.3 system/run operational checklist and run-state rules.
rag/plans/drafts/v0.3-todos/05-acceptance-and-result-template.md v0.3 final acceptance and release packaging template.
rag/plans/drafts/v0.2.md v0.2 plan draft and earlier contract framing.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread rag/plans/drafts/v0.4.md
Comment on lines +217 to +221
| 母计划和 TODO | `rag/plans/drafts/v0.4.md`、`rag/plans/drafts/v0.4-todos/` |
| corpus、manifest、questions、Gold、freeze record | `rag/datasets/v0.4-pilot/` |
| system/config/identity/capability manifest | `rag/systems/v0.4-pilot/` |
| run ledger、raw responses、citations、status、hash | `rag/runs/v0.4-pilot/` |
| calibration、自审、指标、案例、最终报告 | `rag/reports/v0.4-pilot/` |
@muuuuuuushroom
muuuuuuushroom deleted the agent/add-rag-track branch July 30, 2026 03:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants