Skip to content

[2.0] Add new Frontier-CS 2.0 problem vector_db_ann_disk - #146

Merged
joyemang33 merged 5 commits into
FrontierCS:mainfrom
hyj2003:vector-db-ann-disk
Jun 23, 2026
Merged

[2.0] Add new Frontier-CS 2.0 problem vector_db_ann_disk#146
joyemang33 merged 5 commits into
FrontierCS:mainfrom
hyj2003:vector-db-ann-disk

Conversation

@hyj2003

@hyj2003 hyj2003 commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Summary

Please read CONTRIBUTING.md before submitting.

Type of Change

  • New research problem
  • New algorithmic problem
  • New Frontier-CS 2.0 problem
  • Bug fix
  • Documentation update
  • Other:

Testing

Checklist

  • Code follows the project structure and conventions
  • Self-review completed
  • Documentation updated (if applicable)

CI Validation (for new problems)

When adding new problems, CI will automatically validate that your reference solution achieves score > 0.

  • Algorithmic problems: Include reference.cpp in your problem directory
  • Research problems: Include reference.py (or reference.cpp if language: cpp in config.yaml)
  • 2.0 problems: Include reference.py unless the problem config declares another language

@joyemang33 joyemang33 changed the title Add new Frontier-CS 2.0 problem vector_db_ann_disk [2.0] Add new Frontier-CS 2.0 problem vector_db_ann_disk Jun 11, 2026
hyj2003 and others added 4 commits June 15, 2026 22:27
…adme cleanup

  - docker/: judge image that builds the benchmark from scratch at build time
    (download real BIGANN base/query + official GT, compile FreshDiskANN, build
    the on-disk graph+PQ, measure baseline); zero manual download/hosting
  - config.yaml: point to custom judge_image; drop the unused harbor mount block
  - evaluator.py: load truth/baseline into memory then delete from disk before any
    candidate runs (candidate executes in the judge container as root)
  - readme: strictly agent-facing; remove host paths / truth/baseline names; fix
    stale query-count and query-variation wording
FrontierCS#1 Two-tier evaluation. The judge sets FRONTIER_SUBMISSION_ROLE per
submission: iterative `submit.sh` feedback now times a small
FRONTIER_VECTOR_DB_ITER_Q subset (default 2000) for fast turnaround, while
the final verifier (role=final) times the full FRONTIER_VECTOR_DB_Q=10000
set for the authoritative score. recall@10 and QPS are stable averages so
the iterative estimate tracks the final; this cuts the iterate loop from
~4-5 min to ~2 min/submission and removes cancel-thrashing.

FrontierCS#4 Docs. readme: eval wall-time note + iterative-vs-final behavior.
docker/judge/Dockerfile: bake FRONTIER_VECTOR_DB_ITER_Q. docker/README.md:
document constrained/rootless local runs (mount-data variant, --cpus
ignore, env-only credentials).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@joyemang33

Copy link
Copy Markdown
Contributor

Thanks @hyj2003 @momoway . Merged

@joyemang33
joyemang33 merged commit 1489c30 into FrontierCS:main Jun 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants