[2.0] Add new Frontier-CS 2.0 problem vector_db_ann_disk

2.0/problems/vector_db_ann_disk/config.yaml

@@ -0,0 +1,52 @@
+tag: systems
+runtime:
+  language: rust
+  timeout_seconds: 10800
+  environment: "Rust project; hidden disk ANN benchmark; Python/NumPy judge"
+  apt_packages:
+    - build-essential
+    - cargo
+    - git
+    - rustc
+  judge_apt_packages:
+    - build-essential
+    - cargo
+    - rustc
+    - python3-pip
+    - python3-numpy
+  judge_pip_packages:
+    - faiss-cpu
+  docker:
+    # Agent image is the default ubuntu:24.04 (the agent needs no hidden data).
+    # The judge image bakes the SIFT100M benchmark data in; build it before a
+    # local Harbor trial with 2.0/problems/vector_db_ann_disk/docker/build_images.sh.
+    image: ubuntu:24.04
+    judge_image: frontiercs/vector-db-ann-disk-judge:experimental-v1
+environment:
+  # If these resource limits change, also update the resource budget text in
+  # readme and harbor/app/README.md so agents can design parallel algorithms
+  # for the actual CPU and memory budget.
+  cpus: 8
+  memory_mb: 8192
+  storage_mb: 8192
+  build_timeout_seconds: 3600
+evaluation:
+  # The judge drives the search service with this many concurrent workers.
+  # Keep this aligned with the CPU budget unless the task is intentionally
+  # changed into a higher-concurrency service benchmark.
+  query_concurrency: 8
+  # Fallback only: the actual timed query count is pinned to the full official
+  # query set via FRONTIER_VECTOR_DB_Q=10000 in docker/judge/Dockerfile, which
+  # overrides this. (The local CI smoke in evaluate.sh sets its own small Q.)
+  queries_per_worker: 64
+# The hidden benchmark data is baked into the custom judge image
+# (runtime.docker.judge_image); the data paths, dtype, N=100,000,000, and
+# Q=10,000 are pinned as ENV in docker/judge/Dockerfile. truth.bin / baseline.json
+# live under /data/private_100M, which is never passed to /load.
+submission:
+  kind: directory
+  path: /app
+  exclude:
+    - target
+    - .git
+    - .frontier-cs

2.0/problems/vector_db_ann_disk/docker/README.md

@@ -0,0 +1,118 @@
+# Experimental vector_db_ann_disk Judge Image
+
+This task **constructs** the hidden SIFT100M benchmark data from scratch at
+judge-image build time and bakes it in (the duckdb-e2e / vllm bake pattern). No
+manual download, no data hosting, no host staging — the image build downloads the
+public BIGANN dataset, builds the DiskANN index, and measures the baseline.
+
+The **agent** image stays the default `ubuntu:24.04` — the agent never sees the
+hidden data; it only implements the `/load` + `/search` contract.
+
+## Build
+
+```bash
+bash 2.0/problems/vector_db_ann_disk/docker/build_images.sh
+```
+
+Default tag (kept in sync with `config.yaml` `runtime.docker.judge_image`):
+
+```text
+JUDGE_TAG=frontiercs/vector-db-ann-disk-judge:experimental-v1
+```
+
+Full scale (N=100M) is **network / RAM / time heavy** (see Resources). Quick
+smoke image at small scale:
+
+```bash
+N=100000 Q=1000 bash docker/build_images.sh
+```
+
+## Pipeline (docker/build_all.sh, run inside the image builder)
+
+1. **build_data.py** — downloads the real BIGANN data and slices it:
+   - `100M.u8bin` — first N rows of `base.1B.u8bin` via HTTP range, header
+     rewritten to N (`uint8`, 128-dim).
+   - `query.bin` — `query.public.10K.u8bin` (10,000 × 128 `uint8`).
+   - `truth.bin` — top-`TOP_K` ids per query, sliced from the **official** exact
+     ground truth `GT_100M/bigann-100M` (so no 51 GB exact-search is needed).
+2. **build_index.sh** — clones + compiles
+   [FreshDiskANN-baseline](https://github.com/g4197/FreshDiskANN-baseline) and runs
+   `build_disk_index` → `100M_disk.index`, `100M_pq_pivots.bin`,
+   `100M_pq_compressed.bin`.
+3. **build_baseline.py** — exact Faiss `IndexFlatL2` throughput → `baseline.json`.
+
+Source URLs (public, from big-ann-benchmarks):
+
+```text
+https://dl.fbaipublicfiles.com/billion-scale-ann-benchmarks/bigann/base.1B.u8bin
+https://dl.fbaipublicfiles.com/billion-scale-ann-benchmarks/bigann/query.public.10K.u8bin
+https://dl.fbaipublicfiles.com/billion-scale-ann-benchmarks/GT_100M/bigann-100M
+```
+
+## What the judge image contains
+
+```text
+/data/index_100M/     # handed to the candidate via /load
+  100M.u8bin  query.bin  100M_disk.index  100M_pq_pivots.bin  100M_pq_compressed.bin
+/data/private_100M/   # judge-only, NEVER referenced by /load
+  truth.bin  baseline.json
+```
+
+Data paths, dtype, `N=100,000,000`, `Q=10,000` are pinned as image `ENV`
+(`judge/Dockerfile`). The adapter builds the final judge image on top of this one,
+layering `cargo`/`rustc` + `numpy`/`faiss-cpu` from `config.yaml`.
+
+## Resources (full N=100M build)
+
+- **base download**: ~12.8 GB (first 100M rows of `base.1B.u8bin` via range).
+- **DiskANN index build**: tens of GB output, multiple cores, ~hours; tune
+  `R/L/B/M/T` via env in `build_index.sh` (`B` = PQ budget; keep PQ within the
+  8 GiB eval budget).
+- **baseline**: exact FlatL2 over 100M holds vectors as float32 (~**51 GB RAM**)
+  on the build host. The baseline therefore reflects the *build* host, not the
+  8 GiB eval container — a deliberate bake. Adjust if you want a different
+  reference.
+
+## Running locally on constrained Docker (rootless / vfs / limited disk)
+
+The baked judge image is **large (~58 GB)** because the SIFT100M data lives
+inside it. On a host with `overlay2` and ample disk this is fine. But on a
+**rootless daemon using the `vfs` storage driver** (no copy-on-write), every
+derived image layer and every container re-copies the full image, so a single
+trial can need **~3x** the image size in scratch space, and a near-full shared
+disk can be exhausted.
+
+For those hosts, use a **mount-data variant**: build a tiny base image (ubuntu +
+the pinned evaluator ENV + the small `private_100M` secrets) and bind-mount the
+54 GB `index_100M` read-only into the judge service at `/data/index_100M`
+instead of baking it. The data then lives once on the host. Other rootless
+gotchas: set `DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock`; pass
+`--cpus ignore` to `harbor trial start` if the `cpu` cgroup controller is not
+delegated (only `memory`/`pids` usually are); and inject agent credentials via
+**env vars** (e.g. `OPENAI_API_KEY` / `CLAUDE_CODE_OAUTH_TOKEN`), since
+`docker cp` of host-owned credential files fails under the user namespace.
+
+Note also that iterative `submit.sh` evaluations time only
+`FRONTIER_VECTOR_DB_ITER_Q` queries (default 2000) for fast feedback; the final
+verifier (`FRONTIER_SUBMISSION_ROLE=final`) always times the full
+`FRONTIER_VECTOR_DB_Q` set.
+
+## Security note (anti-cheat)
+
+The candidate service is built and run by the judge **in this same container**,
+so plain directory isolation is not enough on its own. Three layers protect the
+ground truth / baseline:
+
+1. **Out of the /load directory** — `truth.bin` / `baseline.json` live under
+   `/data/private_100M`, never passed to `/load`, so they cannot be reached via
+   `dirname(vector_path)`.
+2. **Restricted permissions** — `/data/private_100M` is `0700` root-only (the
+   `/load` files under `/data/index_100M` stay world-readable). A non-root
+   candidate cannot read them.
+3. **Removed after load** — `evaluator.py` loads the truth + baseline into memory
+   at judge startup and then **deletes the files from disk** before any candidate
+   runs, so even a root candidate finds nothing to read. (`_ensure_benchmark`
+   caches in memory; later submissions never need the files. Opt out for
+   debugging with `FRONTIER_VECTOR_DB_KEEP_TRUTH=1`.)
+
+They are also never named in the agent-facing `readme`.

2.0/problems/vector_db_ann_disk/docker/build_all.sh

@@ -0,0 +1,36 @@
+#!/usr/bin/env bash
+# Full data-build pipeline for vector_db_ann_disk. Operator tooling, NOT
+# agent-facing. Produces, with zero manual download, everything the judge needs:
+#
+#   <data_root>/index_100M/{100M.u8bin,query.bin,100M_disk.index,
+#                           100M_pq_pivots.bin,100M_pq_compressed.bin}
+#   <data_root>/private_100M/{truth.bin,baseline.json}
+#
+# Steps:
+#   1. build_data.py   download real BIGANN base/query + slice official GT
+#   2. build_index.sh  clone+compile FreshDiskANN, build the on-disk graph + PQ
+#   3. build_baseline.py  exact Faiss baseline throughput
+#
+# Parametrized by FRONTIER_VECTOR_DB_N / _Q for small-scale testing. At full
+# scale (N=100,000,000) this needs a large-RAM host, tens of GB of disk, and
+# hours of compute — run it offline on the build host.
+set -euo pipefail
+
+SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+DATA_ROOT="${1:?usage: build_all.sh <data_root>}"
+
+export FRONTIER_VECTOR_DB_N="${FRONTIER_VECTOR_DB_N:-100000000}"
+export FRONTIER_VECTOR_DB_Q="${FRONTIER_VECTOR_DB_Q:-10000}"
+export FRONTIER_VECTOR_DB_TOP_K="${FRONTIER_VECTOR_DB_TOP_K:-10}"
+
+echo "=== [1/3] data (vectors + queries + ground truth) ==="
+python3 "$SCRIPT_DIR/build_data.py" "$DATA_ROOT"
+
+echo "=== [2/3] DiskANN on-disk graph + PQ ==="
+bash "$SCRIPT_DIR/build_index.sh" "$DATA_ROOT"
+
+echo "=== [3/3] reference baseline throughput ==="
+python3 "$SCRIPT_DIR/build_baseline.py" "$DATA_ROOT"
+
+echo "=== done. contents: ==="
+ls -la "$DATA_ROOT/index_100M" "$DATA_ROOT/private_100M"

2.0/problems/vector_db_ann_disk/docker/build_baseline.py

@@ -0,0 +1,88 @@
+#!/usr/bin/env python3
+"""Measure the reference (exact, brute-force) baseline throughput -> baseline.json.
+
+Operator tooling — NOT agent-facing. Runs at judge-image build time. Mirrors the
+evaluator's reference: an exact Faiss IndexFlatL2 over the base vectors, queried
+with the same Q query set and CONCURRENCY, timed to produce baseline_qps. The
+candidate must beat this baseline_qps to score.
+
+NOTE (resource + fairness): an exact FlatL2 over N=100M holds the vectors as
+float32 in RAM (~51 GB), so this step needs a large-RAM BUILD host. The resulting
+baseline reflects the build host, not the 8 GiB eval container — bake it
+deliberately. Parametrized by N/Q so it is testable at small scale.
+
+    N=2000 Q=256 python3 build_baseline.py <data_root>
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import struct
+import sys
+import time
+from pathlib import Path
+
+import numpy as np
+
+DIM = 128
+N = int(os.environ.get("FRONTIER_VECTOR_DB_N", "100000000"))
+Q = int(os.environ.get("FRONTIER_VECTOR_DB_Q", "10000"))
+TOP_K = int(os.environ.get("FRONTIER_VECTOR_DB_TOP_K", "10"))
+CONCURRENCY = int(os.environ.get("FRONTIER_VECTOR_DB_CONCURRENCY", "8"))
+ADD_BATCH = int(os.environ.get("FRONTIER_VECTOR_DB_REFERENCE_BATCH_SIZE", "50000"))
+
+
+def _load_u8_matrix(path: Path, rows: int) -> np.memmap:
+    with path.open("rb") as f:
+        npts, dim = struct.unpack("<II", f.read(8))
+    if dim != DIM:
+        raise RuntimeError(f"{path}: dim {dim} != {DIM}")
+    return np.memmap(path, dtype=np.uint8, mode="r", offset=8, shape=(rows, DIM))
+
+
+def main() -> int:
+    if len(sys.argv) != 2:
+        print("usage: build_baseline.py <data_root>", file=sys.stderr)
+        return 2
+    root = Path(sys.argv[1])
+    base = _load_u8_matrix(root / "index_100M" / "100M.u8bin", N)
+    queries = _load_u8_matrix(root / "index_100M" / "query.bin", Q)
+
+    import faiss  # noqa: PLC0415 — only needed here, present in the judge image
+
+    faiss.omp_set_num_threads(CONCURRENCY)
+    index = faiss.IndexFlatL2(DIM)
+    print(f"[baseline] adding {N:,} vectors to FlatL2 (float32, ~{N * DIM * 4 / 1e9:.0f} GB RAM) ...")
+    for start in range(0, N, ADD_BATCH):
+        end = min(start + ADD_BATCH, N)
+        index.add(np.asarray(base[start:end], dtype=np.float32))
+
+    qf = np.asarray(queries[:Q], dtype=np.float32)
+    # Warm up, then time the exact search of the full query set.
+    index.search(qf[: min(32, Q)], TOP_K)
+    t0 = time.perf_counter()
+    index.search(qf, TOP_K)
+    baseline_seconds = max(time.perf_counter() - t0, 1e-9)
+    baseline_qps = Q / baseline_seconds
+
+    out = root / "private_100M" / "baseline.json"
+    out.parent.mkdir(parents=True, exist_ok=True)
+    out.write_text(
+        json.dumps(
+            {
+                "baseline_seconds": baseline_seconds,
+                "baseline_qps": baseline_qps,
+                "baseline_load_seconds": 0.0,
+            },
+            indent=2,
+        ),
+        encoding="utf-8",
+    )
+    print(f"[baseline] {Q:,} queries in {baseline_seconds:.3f}s -> baseline_qps={baseline_qps:.3f}")
+    print(f"[baseline] wrote {out}")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())