feat: cherry-pick scrub-secrets, session-diversify, bench from #21#29
Merged
Conversation
Adds a generic session-diversify helper and wires it into both the broker packet builder (hardcoded cap=2) and search_memories (session_cap arg, default 3, 0 disables). Prevents one chatty session from starving other relevant context from top-K results. Closes #13. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds internal/privacy.Scrub with patterns for AWS keys, GitHub tokens, Anthropic/OpenAI keys, Slack tokens, bearer tokens, RSA/EC private-key blocks, URLs with embedded credentials, and <private>/<secret> tag blocks. Matches are replaced with [REDACTED:<kind>] markers and the tally is merged into the memory's metadata as scrub_count/scrub_kinds. Wired into store_memory, bulk_store_memories, update_memory, and observe_work. Can be disabled with SOLR_MEM_PRIVACY_SCRUB=off for trusted corpora. Closes #15. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New binary at cmd/solr-mem-bench that seeds a namespaced bench-* corpus into any memories collection (safe to run against a live one — only touches bench-* IDs), runs a shipped query set with gold labels, and reports R@1/R@3/R@5/R@10 and MRR plus a per-query breakdown as Markdown. Ships 30-doc synthetic corpus + 25 queries covering easy keyword lookups and harder semantic paraphrases. The paraphrase queries are where BM25 is expected to struggle, giving us a baseline to measure hybrid retrieval (#16 + #17) against. New Makefile target: make bench. 11 unit tests cover R@K, MRR, and Aggregate; no live Solr required for the test suite. Closes #20. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-picks three self-contained, schema-free improvements from the stale stacked PR #21 (
feature/session-diversified-ranking, opened 2026-04-18). The rest of #21 (content-hash dedup, hybrid retrieval/embeddings) is deferred — those need schema migrations / embedding infra.Included (original commits, unchanged — cherry-picked cleanly onto current main)
4ebb8dacap results per session —diversifyBySessionhelper caps per-session hits in ranked lists; broker packet builder enforces cap=2;search_memoriesgainssession_cap(default 3, 0 disables). Stops one chatty session from dominating results.a5a9fb5scrub secrets before storage — newinternal/privacypackage redacts AWS/GitHub/Anthropic/OpenAI/Slack tokens, bearer tokens, RSA/EC keys, creds-in-URLs, and<private>/<secret>blocks →[REDACTED:<kind>]. Tally recorded in doc metadata (scrub_count/scrub_kinds). Wired into all write paths (store/bulk/update/broker). Default on, opt-out viaSOLR_MEM_PRIVACY_SCRUB=off. Defensive win for a shared memory store.d3f92b9retrieval benchmark harness —cmd/solr-mem-benchseeds a namespacedbench-*corpus (safe against live data) and reports R@K + MRR;make bench. Lets us measure ranking changes.No schema changes
Scrub writes to the existing
metadatafield; diversify usessession_id; bench is standalone. So this deploys with just a server rebuild + restart — no configset reload.Tests
go build/vet ./...clean; full suite passes incl. newinternal/privacyandcmd/solr-mem-benchtests.🤖 Generated with Claude Code