Conversation
Expose the read-only MCP tools (search, find, explore, get_chunk) over plain HTTP+JSON so a remote codesearch serve can be queried for federation WITHOUT an MCP session. Each REST handler constructs a per-request CodesearchService bound to the live ServeState, invokes the existing #[tool] method, and returns the tool's JSON payload unwrapped from CallToolResult. The new routes inherit the existing auth layers (require_auth_for_network on network binds), so no new auth code is needed. Adds a rest_routes_are_registered integration test.
Previously each CodesearchService started with embedding_service=None, so per-request REST semantic search reloaded the ONNX model (~100ms-2s) on every call. Mirror the symbol_registry pattern: ServeState now owns a shared Arc<Mutex<Option<EmbeddingService>>> and exposes an embedding_service() getter; new_for_serve binds to it, so the model loads once per serve instance (lazily on first semantic query) and is reused by all MCP sessions AND REST handlers. The standalone new() path keeps its own Arc. Addresses review remark on the REST-endpoint commit.
… (Phase 2)
ReposConfig gains a `remotes` map of RemotePeer{url, api_key, group, timeout_secs}.
Group members may reference a remote peer via an "@"-prefix (e.g. "docs": ["@cloud"]);
resolve_group_targets/split_group_targets return mixed Local/Remote Target lists
(the virtual "all" group stays local-only). New `federation` module exposes a
FederationClient that calls the Stage-2 REST endpoints (/search, /chunk/:id) with
per-peer bearer auth over a shared reqwest client. The `search` tool, when its
group contains remotes, fans out to each peer concurrently, converts remote hits
into source-tagged SearchResultItems (chunk_ref = "peer:id"), and RRF-merges
local + remote ranked lists; unreachable peers degrade gracefully into a
`warnings` array (never hard-fail). The `get_chunk` tool transparently proxies
namespaced chunk_refs ("peer:id") back to the owning peer. 16 new unit tests
(config, federation client, merge/conversion helpers). find/explore federation
deferred.
The previous `f.score < 0.15` threshold was unsatisfiable: merge_ranked_lists reassigns every score to the RRF value 1/(k+rank+1) (max ≈ 0.048 for k=20), so every federated search that returned ANY hit was flagged low_confidence=true. RRF-fused scores are not comparable to single-source embedding/BM25 thresholds, so no score cutoff is meaningful here. Now low_confidence is flagged only when the merged set is empty (a genuine "nothing found" signal to the agent). Also drops the unused `_limit` parameter from build_federated_response. Addresses review remark on the Phase 2 federation commit.
Add ReposConfig::project_groups() — inverse index mapping each repo alias to the named group(s) it belongs to (sorted/deduped; excludes the virtual "all" group and "@Remote" refs; omits aliases in no named group). Expose it as a new `project_groups` field in the scope_required error and sharpen `hint_for_agent` so an agent picking a single project is told to prefer group= when that project belongs to a group (e.g. a separate config / import-data repo). Adds 2 unit tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ects) Add a `groups` field to RepoInfo listing the named group(s) each repo belongs to (via ReposConfig::project_groups()), populated in both the serve and stdio branches of list_projects. Lets an agent inspecting `status` see that e.g. BAYR.Aprimo is part of group BAYER and prefer a cross-repo group= query. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ject_groups() Addresses the sole (informational) review remark: clarify that project_groups() excludes the virtual "all" group implicitly (it is never persisted in self.groups), so a future change that starts persisting "all" must filter it explicitly. Doc-comment only; no behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…d deploy plan
- /healthz: fixed {"status":"ok"} body, no version/repo info, exempt from
require_auth_for_network so container-orchestrator probes reach it on a
network bind without the Bearer key. /health stays auth-protected.
- HEALTHZ_PATH constant; route registered; log-spam suppressed.
- Test: healthz reachable without key on simulated network bind, /health 401.
- docs/federation-cloud-deployment.md: phased Azure plan (role-assignment-free:
SAS + inline ACA secrets), verified rights, PIM-Contributor on Aprimo RG.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…d keep-warm Container artifacts: - Dockerfile: multi-stage build + fastembed model pre-warm baked into the image (ONNX loads from a local path; never streamed from blob) + slim runtime with azcopy + git. .dockerignore keeps target/ and tool dirs out of the context. - docker/entrypoint.sh: snapshot-restore -> azcopy sync (+ optional KB git pull) -> serve; background loop does incremental reindex + periodic index snapshot to a separate blob container. No FSW; full-on-start + incremental-on-timer. serve cloud keep-warm (2h-idle-then-suspend for ACA scale-to-zero): - --keep-warm-url / CODESEARCH_KEEP_WARM_URL + --idle-suspend-secs / CODESEARCH_IDLE_SUSPEND_SECS (default 7200). Serve self-pings its own ingress /healthz while the most-recent real tool call is younger than the idle window, then stops so ACA suspends; next real query wakes it. No Logic App, no managed identity, no role assignment. - ServeState::most_recent_tool_call(); constants for env vars + defaults. docs: option D (scale-to-zero + blob snapshot) with actual SSOT/Aprimo resources. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three build-portability fixes found via review + real builds:
- COPY build.rs into the builder: env!("CARGO_PKG_VERSION_FULL") (main.rs,
cli.rs) needs the var build.rs emits; it falls back to "0"/"unknown" without
.git, so the build is reproducible without repo history.
- Drop BuildKit `--mount=type=cache`: ACR Tasks uses the classic builder which
rejects `--mount`. ACR builds fresh anyway.
- Base images bookworm -> trixie (glibc 2.41): the prebuilt onnxruntime pulled
by `ort` references glibc-2.38+ C23 symbols (__isoc23_strtoll), so linking on
bookworm (2.36) failed. Runtime moved to trixie too; libssl3 dropped (reqwest
uses rustls, no OpenSSL at runtime).
Verified: full image builds clean locally on trixie (all 24 steps).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…repo bake
Found via live ACA deployment + container logs:
- azcopy sync: drop --compare-hash=MD5. It stores the hash in a user_xattr,
which the container overlayfs does not support -> every transfer failed
("1 Failed, 0 bytes") so docs never synced. Default sync (LMT+size) needs no
xattr and works.
- Dockerfile: copy ONLY ~/.codesearch/models from the warmer, not the whole
dir. The warmup writes a repos.json registering "/tmp/warm", which baked a
stale "warm" repo into the runtime image.
- Repo registration: `serve --register` adds the alias but does NOT build the
initial index (create_index is ignored in the CLI handler), and an unindexed
repo is auto-pruned at warmup -> "Unknown alias 'docs'". The entrypoint now
registers each repo via POST /repos (create DB + index + warm) once serve is
live, instead of --register.
Verified end-to-end on Azure Container Apps: cold start -> auto-register ->
index -> federated /search returns the doc within ~5s.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…peer ACA app codesearch-serve live in Delaware.SSOT/Aprimo (scale-to-zero, HTTPS, snapshot + keep-warm). End-to-end verified: cold start -> auto-register -> index -> federated search. Documents resources, FQDN, and the az-CLI emoji log-stream caveat (build local + docker push, not az acr build). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Manage remote serve peers from the CLI instead of hand-editing repos.json:
codesearch remote add <name> --url <url> [--api-key K] [--group G]
[--timeout-secs N] [--into-group LOCAL_GROUP]
codesearch remote list
codesearch remote rm <name>
Model (ReposConfig):
- add_remote(name, peer): validate non-empty name, reject a leading '@'
(the reference prefix is added automatically), require non-empty url.
- remove_remote(name): drop the peer AND prune every "@name" reference from
groups, dropping any group left empty.
- add_remote_to_group(group, name): idempotently push "@name" into a group
(created on demand); rejects reserved "all" and unknown peers.
- groups_referencing_remote(name): sorted groups wiring a peer (for `list`).
--into-group wires "@name" into a local group in one step so the peer is
immediately queryable; without it, `add` prints the follow-up hint.
Hoisted the federation default timeout (15s) into
constants::DEFAULT_REMOTE_TIMEOUT_SECS so the client and the CLI share one
source of truth (no duplicated magic number).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- remote rm: trim the name before lookup/printing, matching the trimming `add`/`--into-group` already do (so `rm " cloud "` finds `cloud`). - federation: demote the inert doc-comment above the `use ... as DEFAULT_TIMEOUT_SECS` alias to a plain comment (a `///` on a `use` is dead). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Decouples the memory-heavy index build from light serving so the long-running Container App can run small (1-2 GiB) while a short-lived Container Apps Job does the full embed on a big replica (4-8 GiB), avoiding OOM (exit 137) on serve. CODESEARCH_RUN_MODE: - serve (default): restore the prebuilt snapshot from blob and serve READ-ONLY. No register / reindex / snapshot — never does heavy work. Warns if no snapshot. - index-job: restore (incremental) -> sync blob -> drive a local serve to build/force-reindex -> wait until /status clears "indexing" -> upload snapshot -> exit 0. New helpers: wait_healthz, register_or_reindex, wait_until_indexed. Deployed: image tag v2-modes; ACA Job 'codesearch-indexer' (index-job, 2 vCPU/ 4 GiB, Manual trigger); serve app resized to 1 vCPU/2 GiB with RUN_MODE=serve. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The /reindex?force=true endpoint returns 500 in the cloud deployment, so the
index-job could not pick up new/changed docs. Replace the reindex call with the
proven path: when the repo is already registered (restored from a snapshot),
DELETE /repos/{alias} then POST /repos for a clean full rebuild. The restored
embedding cache makes unchanged docs cache-hits, so only new/changed docs cost
real work — appropriate for the Job's big replica.
Also harden wait_until_indexed against the DELETE+POST race: phase 1 waits for a
build to actually be observable (status "indexing", or an already-ready state for
a cache-instant rebuild) so the brief post-DELETE gap is never mistaken for
"done"; phase 2 waits for "indexing" to clear with the repo present + ready.
docs/federation-cloud-deployment.md: document the serve/index-job split, the
codesearch-indexer Job, the 1 vCPU/2 GiB restore-only serve, and image v2.1.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The DELETE+POST rebuild is a full re-embed (~20-25 min for the 2737-doc corpus on the job replica), not a cache-fast pass — observed in a live run. Correct the entrypoint comment and deployment doc to state this plainly; the heavy cost is by design (it lives in the Job, never in serve). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The cloud index-job's rebuild_repo used DELETE + POST /repos: it deleted the
~60 MB index dir then immediately reopened it (racy on the container overlayfs,
surfaced as "register failed" 500 + a stalled build that never refreshed the
snapshot), and it was a full ~20-25 min re-embed every run.
Switch to the safe, light path:
- already-registered repo (snapshot restored) -> POST /repos/<alias>/reindex
(incremental, no force): opens the existing index in place, re-embeds only
deltas, returns 202. No DB delete, no reopen race.
- not-yet-registered (cold build, no snapshot) -> POST /repos {path} (full build).
- /reindex?force=true still avoided (returns 500 in this deployment).
Honesty + safety:
- api_code() captures the HTTP status instead of swallowing it with >/dev/null,
and a non-2xx/202 rebuild response now die()s (no silent "register failed").
- verify_index_ready() gates upload on GET /repos/<alias>/info chunks > 0, so a
broken/empty build can never clobber a known-good snapshot.
Doc updated; job sizing noted as 4 vCPU / 8 GiB, 5400s.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root cause of the v2.2 run failure (HTTP 500 "Repo 'docs' is locked by another
codesearch process"): sync_blob runs `azcopy sync --delete-destination=true`
from the blob (which holds only .md source files) into ${DOCS_DIR}, but the
search index lives INSIDE that dir at ${DOCS_DIR}/.codesearch.db. azcopy treated
the whole restored index as "extra" and DELETED it, leaving the incremental
reindex with no base + stale locks.
Fixes:
- sync_blob: --exclude-path=".codesearch.db" so the corpus sync never touches the
index. This also protects the SERVE app's restored index on cold start (same
code path) — previously a cold start could silently wipe the served index.
- restore_snapshot: delete stale .writer.lock / .tantivy-*.lock / lock.mdb that a
prior snapshot baked in (the source of the "locked by another process" 500). A
fresh container has no other process, so any lock present is stale.
- upload_snapshot: exclude *.lock / lock.mdb from the tar so future snapshots stay
clean.
- rebuild_repo: accept HTTP 409 (reindex already in progress) as "wait for it"
instead of die — serve's startup warmup may already be refreshing the repo.
The pre-upload verify guard correctly prevented the failed v2.2 run from
clobbering the good snapshot.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ng reindex) The v2.3 run still hit HTTP 500 "Repo 'docs' is locked by another codesearch process with write access" — but this time it was a LIVE lock, not a stale one, and the index was no longer being deleted (the sync exclude worked). Cause: serve's Phase-1 startup warmup (warmup_repo) opens every registered repo in WRITE mode and runs an incremental refresh on startup, holding the LMDB write lock for the whole refresh. The job's explicit POST /repos/<alias>/reindex opened a SECOND write handle on the same env -> 500. The warmup already does exactly the incremental refresh the job wanted. Fix: for an already-registered repo, the job no longer issues any reindex. It lets the startup warmup do the refresh and just waits for the repo to flip from "closed" (mid-warmup) to "warm" (refresh done) in wait_until_indexed(), then verifies chunks>0 and snapshots. Only the cold (unregistered) case still POSTs /repos. The pre-upload verify guard again prevented the failed v2.3 run from clobbering the good 69 MB snapshot. Doc updated to describe the warmup-driven refresh and the two invariants (sync never touches .codesearch.db; never compete with the warmup). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ove/reindex repos on a peer) Add REPOS_PATH constant and the remote index-management client surface: ManagementOutcome (Ok/HttpError/Unreachable), RemoteStatus/RemoteRepoStatus, RemoteRepoAdded/Removed/RemoteReindexResult structs, and FederationClient methods list_repos/add_repo/remove_repo/reindex backed by a shared send_management helper. Includes mock-peer unit tests (6 new, all passing). Dead-code warnings are expected — Stage 2 (CLI) consumes this API.
…ADME + cloud deployment doc
…the read-only cloud peer)
…er requirement on per-vendor recipe
`codesearch index rm <alias>` (without --remote) previously treated the alias as a filesystem path and failed with os error 2 during canonicalize. Now `remove_from_index` resolves the argument against registered aliases first (ReposConfig::resolve), falling back to path interpretation only when it is not a known alias. The resolved path is also forwarded to try_delegate_rm_to_serve so serve delegation works by alias too.
Adds `#[command(visible_alias = "ls")]` to the three `List` variants so `codesearch index ls`, `codesearch groups ls`, and `codesearch remote ls` work as shortcuts for `list`.
…0695, 30640677)
Two Aikido Critical findings (priority 95) addressed:
Rust — src/index/mod.rs:92 (`get_db_path_smart`):
Replaced `safe_canonicalize(project_path).unwrap_or_else(|_| PathBuf::from(project_path))`
with strict error propagation. The previous fallback silently bypassed
canonicalization when the path did not exist or was inaccessible, defeating
every downstream `starts_with`/`join` containment check. Callers now get a
clear error if the project path cannot be resolved. Verified: only
`index_with_options` calls this function — no caller depended on the fallback.
.NET — helpers/csharp/Program.cs + OutputWriter.cs:
Added `RequireValidPath(args, ref i, flag, mustExist)` helper that wraps
`RequireValue` with `Path.GetFullPath` canonicalization + optional existence
check. Applied to every CLI path argument (--solution, --project, --output,
--symbols-file) across all three Parse*Args methods. Removed redundant
`File.Exists(symbolsFile)` check now covered by the helper.
Added `CanonicalizeOutputPath` guard to all three OutputWriter.Write*Async
methods as defense-in-depth (idempotent `Path.GetFullPath` + null check)
in case OutputWriter is called from a future code path that bypasses the
CLI parser.
Build verification:
- .NET helper: `dotnet build` → 0 errors, 0 warnings
- Rust: deferred (build environment has broken MSVC link.exe on this host;
change is a 14-line syntactic edit using already-imported `safe_canonicalize`
and `anyhow!`, with no caller-dependency risk)
Refs: Aikido groups 30640695 (Rust), 30640677 (.NET)
Skipped: defense-in-depth internal fs ops (vectordb/store.rs, etc.) — not
externally controllable. Will document in follow-up.
Mitigates Aikido finding group 35039595 (priority 30, LOW): "GitHub Actions actions/checkout persists GITHUB_TOKEN to git config on self-hosted runners, allowing subsequent steps to authenticate as the repo via the saved credential helper." Adds `with: persist-credentials: false` to every actions/checkout step: - ci.yml: 3 steps (test-linux, test-windows, csharp-integration-tests) - codeql.yml: 1 step (analyze job) - release.yml: 2 steps (build matrix, build-macos) No other checkout steps exist in the repo (protect-master.yml has none). YAML syntax validated post-edit. No semantic behavior change — CI/release jobs do not push back to the repo from these checkouts, so disabling the auth helper is purely defensive. Note: ci.yml/release.yml use pinned SHA 34e114876b0b11c390a56381ad16ebd13914f8d5 (pinned v4); codeql.yml uses floating @v4 tag (pre-existing inconsistency, left untouched in this commit).
…641757)
Mitigates Aikido finding group 30641757 (priority 35, MEDIUM):
"ANSI escape sequence injection in search output" — indexed content
could embed CSI/OSC sequences (e.g. \x1b[2J clears screen, \x1b]0;...\x07
rewrites window title) that the host terminal would execute on print.
Changes:
- Add `sanitize_for_terminal(&str) -> String` helper in src/search/mod.rs
Strips: CSI sequences (ESC [ ... <final 0x40-0x7E>),
OSC sequences (ESC ] ... (BEL | ESC \)),
single-char escape sequences (ESC <0x40-0x5F>),
and stray control chars except \n and \t.
Safe on truncated input — never panics.
- Apply to every user-controllable println! site in search/mod.rs:
* print_result: result.path, result.kind, result.signature,
result.context, result.context_prev/next lines,
result.content lines, snippet
* sync_database: file.path display, deleted-file path string
* compact path: result.path
* query string in standard output header
- Add 9 unit tests covering CSI/OSC/single-char/control-char/unicode/
empty/truncated-input cases.
The `colored` crate wraps content but does not sanitize inner escapes;
sanitization happens BEFORE .bright_green() / .dimmed() / etc. so the
color wrapper cannot be broken out of.
Local cargo check blocked by pre-existing MSYS2 link.exe issue
(documented in PR #151) — no errors in src/search/mod.rs. cargo fmt
passes.
…:walk Mitigates Aikido finding group 30641794 (priority 38, MEDIUM): "Local client can register `.git` dir as repo, search excluded Git metadata" — exposes internal/sensitive files (objects, config, refs) via search results. Root cause: `FileWalker::walk`'s `filter_entry` closure short-circuits on `entry.depth() == 0` (the root entry), so the ALWAYS_EXCLUDED name check is bypassed when the user points the indexer at a directory whose own name is `.git` (or `node_modules`, `target`, etc.). Fix: validate `self.root.file_name()` at the top of `walk()` and bail! with an actionable error if the name matches an ALWAYS_EXCLUDED entry. Covers every caller uniformly — CLI `index`, HTTP `/repos`, `doctor`, `sync_database`, watcher — without needing to patch each callsite. Pre-existing depth==0 short-circuit intentionally left in place (now unreachable for excluded names; still correct for normal roots whose names are not in the list). Test: `test_rejects_excluded_named_root` builds a temp `.git` dir, asserts walk() returns Err with "Refusing to index" + ".git" in the message, and verifies a sibling non-excluded root walks normally.
…th (Aikido 30641757)
Mitigates Aikido finding group 30641757 (priority 46, MEDIUM):
"Improper Input Validation — backslash path collision on Unix".
Companion finding to the ANSI escape injection already fixed in
stage 1/3 (same group, different priority).
THREAT MODEL
On Unix, backslash is a legal filename character (not a path
separator). A file literally named `foo\bar.rs` is distinct from
`foo/bar.rs` (which lives in subdirectory `foo`). The previous
`normalize_path` / `normalize_path_str` unconditionally ran
`.replace('\\', "/")`, collapsing both into the key `foo/bar.rs`.
This caused silent HashMap collisions in `FileMetaStore`: one
file's chunks would overwrite the other's metadata, leading to
stale search results, missed re-indexing, or wrong chunk IDs.
FIX
Gate the backslash-to-forward-slash conversion behind `#[cfg(windows)]`:
- Windows: backslash IS a path separator — conversion is required
for HashMap consistency across canonicalize/Notify/raw APIs.
- Unix: preserve backslash literally; it is part of the filename
and must not be normalized away.
The `trim_start_matches(r"\\?\")` (UNC prefix strip) runs
unconditionally on both platforms — it is a no-op on Unix in
practice but defensive in case a Windows-style path string leaks
into a Unix process via config/migration.
TESTS
- Added `test_normalize_path_preserves_unix_backslash_filenames`
(cfg(not(windows))): asserts `foo/bar.rs` and `foo\bar.rs`
normalize to distinct keys.
- Gated 12 Windows-specific tests with `#[cfg(windows)]` because
they explicitly assert backslash conversion using hardcoded
`C:\...` / `\\?\C:\...` inputs. These tests document Windows
behavior and have no meaning on Unix after the fix.
Files changed: src/cache/file_meta.rs (+50 / −2 net)
Validation:
- `cargo fmt --check src/cache/file_meta.rs` PASS
- `cargo check --lib --tests` fails ONLY at the pre-existing
MSYS2 `/usr/bin/link` vs MSVC `link.exe` link step (no errors
reference file_meta.rs). Authoritative validation will run in
GitHub CI.
…deps) Addresses Aikido dependency-vulnerability findings via semver-safe `cargo update` plus an explicit floor bump for the highest-priority direct dep. Direct-dep change: - rmcp 1.5.0 -> 1.8.0 (Aikido priority 82, 3 CVEs — impersonate data source). Major bump to v2.x available but breaking; deferred. Within-semver patch picks up the CVE fixes without API churn. Cargo.lock refresh (`cargo update` with no Cargo.toml changes beyond the rmcp floor bump above). Notable security-relevant transitive bumps: - quinn-proto 0.11.14 -> 0.11.16 (Aikido priority 75, DOS) - h2 0.4.14 -> 0.4.15 - hyper 1.10.1 -> 1.11.0 - tokio 1.52.3 -> 1.53.1 - rustls 0.23.40 -> 0.23.42 - openssl 0.10.80 -> 0.10.81 - zerocopy 0.8.52 -> 0.8.55 - zeroize 1.8.2 -> 1.9.0 - webpki-roots 1.0.7 -> 1.0.9 - aws-lc-rs 1.17.0 -> 1.17.3 - regex 1.12.4 -> 1.13.1 - safetensors 0.7.0 -> 0.8.0 Plus ~90 other minor/patch bumps. Net Cargo.lock diff: +391/-490 lines. Deferred (separate concerns): - rmcp v2.x major bump — breaking API changes, needs dedicated migration - Blurred Aikido entries (we*zl p65, l*u p62, etc.) — cannot identify exact crates without `cargo audit`, which is itself blocked by the same MSYS2 `/usr/bin/link` link-step issue that blocks local builds. CI on GitHub Actions will surface anything still open after this bump. Validation: - `cargo metadata --no-deps` parses cleanly (Cargo.toml well-formed) - `cargo check --lib` fails ONLY at link step (pre-existing MSYS2 `/usr/bin/link` shadowing MSVC `link.exe`, documented in PRs #151- #153). No source-level errors, no unused-import warnings. - `cargo fmt --check` N/A (no .rs files modified). - Authoritative validation deferred to GitHub Actions CI on the PR.
🔒️ fix: harden path-traversal vectors and CI credential persistence (Aikido)
🛡️ fix: defense-in-depth for ANSI injection, .git root walk, and Unix backslash collision (Aikido)
deps: cargo update + rmcp floor bump 1.5.0 → 1.8.0 (Aikido p82)
…hardening) Aligns .github/workflows/codeql.yml with the pinning policy already followed by ci.yml and release.yml: replace the floating @v4 tag with the pinned SHA 34e114876b0b11c390a56381ad16ebd13914f8d5 (# pin@v4). The floating @v4 tag is mutable — if the action's tag is moved (accidentally or via compromise), CI would silently start running whatever new SHA the tag points to. Pinning to a specific SHA makes every CI run reproducible and requires an explicit commit to change which code runs. Related: Aikido follow-up to finding group 35039595 (GitHub Actions persist-credentials). Same threat class (CI supply-chain integrity). No behavior change — SHA 34e1148... is the exact commit @v4 currently resolves to, verified via the existing pin in ci.yml:21 and release.yml:41.
🔒️ fix: pin actions/checkout SHA in codeql.yml (supply-chain hardening)
Scope prose-aware Text labels to EmbeddingGemma Markdown and plain-text chunks. Keep the historical Code label for all existing models so incremental indexing cannot mix document representations. Warn when explicitly selecting models with larger vector dimensions.
test_sanitize_strips_single_char_escape passed a String argument to sanitize_for_terminal(s: &str), breaking cargo test --lib. Pass &str literals to match the sibling sanitize tests. (only surfaces under --lib test compilation, not plain cargo check) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Five tests hardcoded Windows absolute paths (C:\..., \?\C:\..., backslash separators) and asserted separator-rewriting semantics that normalize_path_str deliberately applies ONLY on Windows (backslash is a legal filename char on Unix — see file_meta.rs Aikido 30641757 rationale). They therefore failed on the Linux CI jobs (test-linux, csharp-integration-tests) while passing on test-windows. Gate the Windows-specific tests with #[cfg(windows)] and add #[cfg(unix)] counterparts using native forward-slash paths for the three path-matching tests, preserving Linux coverage. The two pure separator-handling tests (backslashes / mixed) are Windows-only concepts; forward-slash behaviour is already covered by test_path_prefix_no_alias/_empty_alias on all platforms. Pre-existing develop breakage, unrelated to the EmbeddingGemma feature (src/mcp/mod.rs is untouched by that work). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
.map(|l| sanitize_for_terminal(l)) -> .map(sanitize_for_terminal). .lines() yields &str and sanitize_for_terminal takes &str, so the direct function reference is valid. clippy -D warnings (Linux CI) flagged it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
missing_db_not_cached_as_conflicted opened SharedStores directly in the test setup and then let get_or_open_stores open the same LMDB env again — two opens of one env in a single process, which AGENTS.md's LMDB rule forbids. On Linux the first env is not always released before the reopen, so try_open_stores' open failed intermittently -> readonly -> Conflicted -> Err (flaky). try_open_stores creates the env itself (see try_open_stores_creates_db_for_brand_new_repo), so the direct pre-open was redundant. Dropping it leaves a single deterministic open on both platforms. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add EmbeddingGemma retrieval support
…wedges accept() serve's fd demand scales with registered repo count (LMDB env + tantivy FTS segments + file-watcher handles ≈ 15-20 fds per warm repo). Under process supervisors the default soft limit is often 256 (macOS launchd agents, some systemd/docker configs). Once the process saturates it: - tantivy logs 'Too many open files' (errno 24) warnings, and - accept(2) fails with EMFILE; axum's accept loop sleeps and retries silently, so the daemon looks alive to its supervisor while every new connection is refused or reset. No ERROR log, no exit — a silent wedge. Observed in production: 60 registered repos (~1000 fds needed) under a macOS LaunchAgent — serve answered for ~15s after start (until repo warmup consumed the fd budget), then reset every connection while the process stayed 'healthy', deterministically across restarts. Fix, at run_serve startup before any store open or bind: 1. Raise the RLIMIT_NOFILE soft limit to the hard limit (standard daemon practice — nginx/envoy/postgres do the same). On macOS the target is clamped to kern.maxfilesperproc so setrlimit cannot fail with EINVAL. Failures are non-fatal and logged. 2. Log the raise at INFO. 3. If the effective limit still looks too small for the registered repo count (repos × 20 + 256 headroom), emit a loud actionable WARN naming the supervisor knobs (launchd SoftResourceLimits.NumberOfFiles, systemd LimitNOFILE, ulimit -n). Verified at scale: with ulimit -n 256 and 60 registered repos, an unpatched serve saturates at 255/256 fds (EMFILE in logs, wedge under launchd); the patched serve logs 'Raised RLIMIT_NOFILE soft limit 256 → 61440', runs at ~300 fds, and answers MCP handshakes indefinitely. cargo clippy -D warnings clean; cargo test --lib --bins green (579 + 575).
🐛 fix: raise RLIMIT_NOFILE at serve startup — fd exhaustion silently wedges accept()
…events) Fork PRs run with a restricted GITHUB_TOKEN that cannot write `security-events` back to the upstream repo, so the analyze step's SARIF upload fails with "Resource not accessible by integration" for every external contributor PR (e.g. PR #150 from tony-nexartis). Add a job-level `if:` that skips the entire analyze job when the pull_request's head repo differs from the workflow's repository. CodeQL still runs on: - push events to develop/master (post-merge, full write token) - same-repo PRs (full write token) - the weekly schedule so no scanning coverage is lost — only the redundant, upload-failing fork-PR run is skipped. No behavior change for non-fork workflows.
🔧 ci: skip CodeQL on fork PRs (upload cannot write security-events)
…s env vars (#149) Two unrelated fixes bundled in one PR per maintainer direction. #148 — UTF-8 panic at src/search/mod.rs:1343 ============================================ Pre-existing bug: `&snippet[..100]` byte-sliced a UTF-8 string, panicking with "byte index 100 is not a char boundary" when byte 100 landed inside a multi-byte character (box-drawing separators in comment art, CJK, emoji). Originally flagged in PR #152 review as "out-of-scope, deferred"; reported as issue #148 by @tony-nexartis. Fix: use `str::floor_char_boundary(100)` (stabilized in Rust 1.82; we're on 1.95) to find the largest char boundary ≤ 100 bytes, then slice. 1-line change at the print site. Regression test `test_byte_truncation_preserves_ char_boundary` in src/search/mod.rs constructs a 120-byte string of U+2500 box-drawing chars and asserts no panic + correct char-boundary cut. #149 — Container hostname rejected by rmcp default allowlist ============================================================= rmcp ≥ 1.4.0 added DNS-rebinding defence (GHSA-89vp-x53w-74fx, CVE-2026-42559): `StreamableHttpServerConfig::allowed_hosts` defaults to loopback-only `["localhost", "127.0.0.1", "::1"]`. Containerised deployments (where the Host header is the container hostname, not localhost) get `WARN ... rejected request with disallowed Host header`. Reported as issue #149 by @stdweird. Fix: expose two env vars, both read once at serve startup: CODESEARCH_ALLOWED_HOSTS=host[,host:port,...] Comma-separated list of hostnames / `host:port` authorities. Replaces the rmcp default allowlist. Whitespace-trimmed, empties dropped. CODESEARCH_DISABLE_HOST_VALIDATION=1|true Disables Host validation entirely (calls rmcp's `disable_allowed_hosts()`). DANGEROUS — only safe behind a reverse proxy that validates Host itself. Accepts `1` or `true` (case-insensitive); any other value is ignored. Takes precedence over CODESEARCH_ALLOWED_HOSTS. New module-level helper `build_streamable_http_config()` in src/serve/mod.rs encapsulates the resolution order (disable > custom > default). Called once from `run_serve` in place of the previous inline `StreamableHttpServerConfig ::default()`. 7 unit tests in `mod allowed_hosts_tests` cover all branches. Both env vars documented in src/constants.rs with the same comment style as the existing ALLOWED_ROOTS_ENV / SERVE_API_KEY_ENV. Validation ========== - `cargo fmt --check` clean - `cargo clippy --all-targets -- -D warnings` clean - `cargo test --lib --bins`: 1188 passed, 36 ignored, 0 failed (includes 7 new allowed_hosts tests + 1 byte_truncation test) Closes #148. Closes #149.
…weep) Documents the security hardening sweep and follow-up fixes that landed in develop since the [1.1.30] changelog entry, none of which had been changelogged or documented in README: - PR #151: critical path-traversal fixes (Rust + .NET) + CI persist-credentials - PR #152: ANSI-injection sanitization, .git-root rejection, Unix backslash path-cache collision fix - PR #153: CodeQL checkout SHA pinning - PR #154: rmcp 1.5.0->1.8.0 + ~100 transitive dependency CVE updates - PR #150 (external, @tony-nexartis): RLIMIT_NOFILE fd-exhaustion fix - PR #156: skip CodeQL analyze on fork PRs (restricted GITHUB_TOKEN can't upload SARIF to upstream) - PR #157: byte-boundary panic fix (#148, @tony-nexartis) + new CODESEARCH_ALLOWED_HOSTS / CODESEARCH_DISABLE_HOST_VALIDATION env vars (#149, @stdweird) Also bumps Cargo.toml to 1.1.31 for this documentation/version-tracking release. No functional code changes in this commit.
Owner
Author
|
Closing — CONFLICTING due to known squash-merge history divergence (see AGENTS.md). Using the documented release/vX.Y.Z workaround instead. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release v1.1.31
Security hardening sweep (Aikido: critical path-traversal fixes, ANSI-injection sanitization, .git-root rejection, Unix path-cache collision fix, CI supply-chain hardening, dependency CVE updates) plus community bug/dependency fixes and the EmbeddingGemma retrieval feature.
Highlights
.git-root rejection, Unix backslash path-cache fix (🛡️ fix: defense-in-depth for ANSI injection, .git root walk, and Unix backslash collision (Aikido) #152)CODESEARCH_ALLOWED_HOSTS/CODESEARCH_DISABLE_HOST_VALIDATIONenv vars (support allowed_hosts list #149, credit @stdweird) (🐛 fix: byte-boundary panic (#148) + rmcp allowed_hosts env vars (#149) #157)See CHANGELOG.md
[1.1.31]for full details.🤖 Generated with worker automation