feat: research-fed detection⇄evasion loop, coordination/QUIC/FP-gate rungs, grounding harness#32
Merged
Merged
Conversation
… from the FP corpus browserforge cross-samples userAgent / navigator.platform / userAgentData semi-independently from its Bayesian network, so a fraction of its output is internally incoherent — a Windows UA with navigator.platform "Linux x86_64" (navplatform_vs_ua), a UA Chrome major that disagrees with the UA-CH major (ch_he_version_vs_ua), or a HeadlessChrome brand set (ch_he_headless). A real browser's UA/platform/CH all come from one binary and cannot disagree, so the convicting rules flag these correctly — counting them as false positives over-reported the FP rate (the residual ~1% bot FPs the docs attributed to "data artifacts") and risked pressure to weaken a correct convicting rule to chase a corpus artifact. generate_profiles now drops internally-incoherent fingerprints from the legit corpus via the new calibration.fingerprint_coherence (mirrors the mapper's own derivations, so Android's Linux-kernel platform under an Android UA stays coherent) and reports the excluded count by reason — never a silent drop. Measured: hard bot FP rate 1% -> ~0% over the coherent denominator, no rule touched. Fonts are deliberately excluded from the filter: cross-OS font installs are real, so a UA-vs-font mismatch is not a binary-bound incoherence — br.font_os_vs_ua is left to be evaluated on its merits. harness: ruff + mypy clean, 223 passed, coverage 96.36%.
…hin-session coherence QUIC is the last gap in the within-session coherence axis. The edge attributes QUIC fingerprints per source IP (the elicit-and-close capturer + FingerprintByIP), which is NAT-confounded, single-shot, and ungroundable — the reason net.quic_grease_vs_ua / net.quic_pq_keyshare_vs_ua were retired and a net.quic_unstable_within_session rotation rule cannot ship. Records the decision to serve H3 and key the captured Initial by QUIC connection ID (DCID) instead of source address: ks_sid-attributed, migration-safe, and reusing the already-multi-packet parser (ParseQUICInitials) and the TCP path's session-minting. Explicitly does NOT authorise shipping a QUIC rule — promotion stays gated on out-of-sandbox GROUNDED-LIVE evidence (a browser-trusted cert for a real-Chromium h3 negative + a uTLS/quic-go rotation evader positive). The plumbing is buildable/unit-testable in-sandbox (Docker golang:1.26); the rule is not groundable here. Status: Proposed.
…itive (ADR-0005) First in-sandbox increment of the ADR-0005 QUIC rework. The capturer attributed Initials by source address (FingerprintByIP), which is NAT-shared and breaks on path migration — the root of the QUIC cross-attribution FPs. This adds attribution by the client-chosen Destination Connection ID, which is unique to one handshake: - fingerprint.InitialDCID(pkt) extracts the DCID from the Initial's cleartext long header (no decryption); decryptInitialCrypto now reuses it (one header parse, not two). - teeEntry records the DCID (all Initials of a handshake share it); take()'s predicate now receives the entry, and QUICCapturer.FingerprintByDCID(dcid) retrieves a hello per-connection. The live per-IP bridge is untouched (FingerprintByIP retained) — this is additive, with no detector /registry/scoring change, so zero FP risk. It is the foundation the H3-serving step consumes to link a captured Initial to its connection's ks_sid (the next increment); the within-session QUIC rule stays unbuilt until that lands and can be GROUNDED-LIVE on a trusted-cert host (per ADR-0005). Tests: TestInitialDCID (pure extraction + rejections), TestFingerprintByDCID (real quic-go handshake → retrieve by recorded DCID, and an unknown DCID does not mis-attribute). Full edge vet + tests green via Docker golang:1.26.
…(ADR-0005) Second ADR-0005 increment: a QUICServer that serves H3 over the tee'd UDP socket instead of eliciting-and-closing, so the ks_sid arrives ON the QUIC connection (the H3 request cookie) and the captured Initial is attributed to it by DCID — removing the source-IP match across the TCP/QUIC boundary that made the QUIC tells NAT-confounded. - QUICServer runs http3.Server.Serve(tee): the tee is a net.PacketConn, so one socket both serves H3 and records each connection's Initial + DCID. ConnContext binds the connection's DCID (resolved via tee.dcidForAddr at connection start, where the remote addr still equals the Initial's source addr) into the request context, so the handler attributes by DCID and survives later path migration. - The handler reuses prepare() for ks_sid + header signals, appends quicTells looked up by fingerprintByDCID, forwards, and reverse-proxies to the backend. - tee gains fingerprintByDCID + dcidForAddr; QUICCapturer.FingerprintByDCID now delegates (DRY). - go.mod: quic-go/qpack pulled in (indirect) by the http3 import. NOT yet wired into ListenAndServe (the live edge still runs the elicit-and-close QUICCapturer) — the swap, and the net.quic_unstable_within_session rule, stay gated on out-of-sandbox GROUNDED-LIVE evidence (a browser-trusted cert + a QUIC rotation evader), per ADR-0005. Test: TestQUICServerAttributesFingerprintByDCID drives a real http3 client through the server to a stub backend and asserts quic_observed is emitted under the request's ks_sid (proving the Initial→DCID→ ks_sid path end-to-end over real QUIC). Full edge vet + tests green via Docker golang:1.26.
… clustering Red⇄blue rung on the cross-session coordination seam. RED: the fleet detector clustered ONLY by JA4 prefix (score_corpus / FleetTracker), and every collision check ran within a single JA4 cluster — so a fleet that rotates its JA4 per node (uTLS egress / mixed builds) lands in singleton clusters, is never graded, and its convicting tells are never computed. Grounded: a cloned-fp / replayed-trace / shared-origin fleet with a distinct JA4 per node scored 0 clusters, 0 convictions (EVADES) — including the unambiguous trace-replay and shared-origin tells. BLUE: cluster by ANY cross-instance binding, not just JA4. _collision_clusters groups sessions by a cloned fp_hash / replayed trace_hash / shared webrtc_public_ip spanning >=2 distinct observed IPs; _clusters unifies JA4-prefix + collision clusters (a collision subset of a JA4 cluster is deduped, so a fleet sharing both is reported once). score_corpus and FleetTracker (online) both use it; score_cluster gains a `basis` label and is reused unchanged, so the conviction gate — and its FP-safety — is identical. Grounded both directions: the three JA4-rotating fleets now convict (offline + a single online alert on the 2nd arrival); FP-safety locked — distinct-profile cohorts form no collision cluster, and a corporate one-image fp-collision still caps at candidate (ambiguous, uncorroborated) on the new path too. All 42 prior coordination tests pass, the scenario gate holds at 100%/100% precision/recall, +6 grounding tests. harness: ruff + mypy clean, 229 passed, coverage 96.45%.
…ration 1) Seeds the research-fed red⇄blue loop from a deep-research pass (5 angles, 23 sources, 25 claims 3-vote-verified, 22 confirmed). docs/research-radar.md is the durable queue: external detection/evasion techniques mapped to Kitsune seams, each tagged groundable-in-sandbox vs external-data-bound (which IS the real-world-testability evaluation), tracked lead→grounded. Iteration 1 leads: - Groundable: G1 cross-attribute (spatial) fingerprint inconsistency (FP-Inconsistent, ACM IMC 2025 — the strongest external validation of Kitsune's incoherence thesis: coherence rules cut DataDome evasion 48%); G2 GAN/diffusion mouse-synthesis evader to pressure-test the coalesced terminus; G3 keystroke dynamics; G4 JA4+ suite coverage audit. - External queue: X1 encapsulated-TLS proxy/tunnel fingerprinting (USENIX Sec 2024); X2 RESIP relayed/tunnel classifier; X3 CGNAT detection; X4 real-traffic prevalence prior. - Validations (do not rebuild): temporal inconsistency == the within-session axis; session-replay == trace_replay/trace_collision; engine-level spoof defeated by coherence not property probes. Methodology guardrails recorded: grounding discipline, lab-classifier≠production deltas, and Rosetta's packet-length-sequence non-transfer warning.
The edge pump needs local Docker + go-task, so the loop runs on demand in a local session rather than as a cloud routine. Drive a cycle by asking Claude to "run a research-loop cycle" or via /loop. A cloud routine can be added later via /schedule once GitHub is connected for datascry/kitsune.
…nts 0 (research-radar G1)
First research-fed pump rung (docs/research-radar.md G1, from FP-Inconsistent, ACM IMC 2025 — the
strongest external validation of Kitsune's incoherence thesis). The device-database-FREE form of its
spatial inconsistency check ("an iPhone with an impossible geometry").
RED (gap): a desktop bot wearing a mobile UA (CDP setUserAgentOverride to a phone UA without touch
emulation) reports navigator.maxTouchPoints === 0. The existing touch rules miss it — maxtouch_desktop
is the inverse, and pointer_touch_incoherent sees a coherent desktop (CSS coarse=false AND maxTouch=0).
So the mobile-spoof produced zero convicting tells: EVADES.
BLUE: a phone/tablet is a touchscreen device — every real one reports maxTouchPoints > 0 (iOS Safari = 5).
A Mobile/iPhone/iPad-token UA with maxTouchPoints === 0 is a desktop wearing a mobile UA. FP-safe with no
device DB: scoped to the Mobile/iPhone/iPad tokens, NOT bare "Android" (which also matches touch-less
Android TV/Auto). Emitted by demo.py (authoritative collector) + the calibration mapper.
GROUNDED both ways: convicts the iPhone/Android mobile-spoof as the SOLE convicting tell (bot); real
iOS/Android (maxTouchPoints 5/>0) and desktop UAs (out of scope) do NOT fire it. 0 FP on browserforge
N=600 (real mobile fps carry touch>0). detector 260 passed (99.32%), harness 230 passed (96.50%);
catalog/README/matrix regenerated; new regression test in test_calibration_methodology.
The screen-geometry-DB half of G1 split to X5 (external — needs a real device→resolution map). Follow-up:
mirror the emission into the production collector (collector/src/livepage). Loop RECORD step: G1 → done.
…roundable, G4 covered Iteration 3 of the research-fed loop ran the rest of the groundable column: - G2 (DMTG/diffusion mouse synthesis): VALIDATED — the convicting behavioral tell is structural (synthetic_no_coalesced / coalesced_untrusted key on getCoalescedEvents length + isTrusted), so trajectory-shape realism is orthogonal; a DMTG path via CDP still has coalescedMax<=1 → caught. Already grounded (KS_BEHAVE); escape is privilege-gated XTEST. No new rule. - G3 (keystroke dynamics / cGAN): NOT GROUNDABLE — keystroke timing is jitter-unsound across instances, no structural channel analogous to mouse coalesced sampling; entropy floor stays corroborating. - G4 (JA4+ suite): COVERED — JA4 + JA4H (h2_header_order_vs_ua); JA4T detection-value = TCP-OS coherence (tcp_kernel + tcp_os_vs_ua); JA4L marginal/latency-external; JA4S server-side (N/A for client detection). Groundable column is now DRY; remaining leads (X1-X5) are all external-data-bound. Net research-fed-loop result: 1 new grounded rule (br.mobile_no_touch, G1) + the rest validated/routed. Re-run SCAN periodically.
Completes the G1 rung across collector surfaces (the mirror-across-collectors discipline): the production livepage collector now emits browser.mobile_no_touch (phone/tablet UA + maxTouchPoints === 0) alongside the authoritative demo.py emission, so live-page traffic is scored on it too. Same scope (Mobile/iPhone/ iPad tokens, not bare Android) and FP-safety as demo.py + the calibration mapper. collector: typecheck + eslint + prettier + vitest all green (Docker node:22).
…ata grounding The recurring unlock, made turnkey. The in-sandbox seams are saturated and every remaining frontier is external-data-bound (real proxy egress / real-device diversity / real traffic / real QUIC); the ingest paths existed but were scattered. This adds the single consumer: - kitsune_harness.grounding (`task grounding -- <dir> [--expect legit|bot] [--build-prior OUT]`): over a directory of REAL session captures (collector/edge shape) it runs per-session FP/recall (against real traffic, not synthetic browserforge), coordination grading (a per-session mimic caught at the cluster layer is not counted as a miss), and an optional prevalence-prior rebuild. Pure evaluate()/render() + thin IO CLI; 4 tests (legit-clean, legit-FP-flagged, bot-per-session, bot-via-fleet). - docs/grounding.md: the runbook mapping each blocked frontier (radar X1-X5 + Tier-3 prevalence + QUIC) to the exact capture + command + what it unlocks, plus the existing capture tooling and the no-shortcut discipline. Validated over the committed captures: `task grounding -- corpus/sessions --expect bot` surfaces the 10 known EVADES mimics as residual per-session misses and catches the 64-member stealth fleet via coordination. harness: ruff + mypy clean, 234 passed, coverage 95.83%.
…le_no_touch
NAIVE=1 on the mobile-emulation evader does the header-only mobile spoof: a phone UA via newContext({
userAgent }) with NO device emulation (no hasTouch/isMobile/viewport) — the common scraper shortcut. This
leaves navigator.maxTouchPoints at the desktop default 0, so a phone UA on a touch-less client trips
br.mobile_no_touch (the rule shipped this session, previously synthetic-test-only).
GROUNDED LIVE against the running edge->detector stack: NAIVE=1 → label bot WITH br.mobile_no_touch fired;
the full Pixel-5 emulation (hasTouch:true → maxTouchPoints>0) → br.mobile_no_touch correctly does NOT fire.
Lights an active-but-unexercised rule via a faithful red-team mode (validation; no version bump).
…e mobile escalation ladder Two more faithful mobile-spoof modes that complete the desktop-faking-mobile ladder, each catching the spoofer who patched the previous tell (Tier-1 mobile uplevel): - IOS=1: an iPhone-Safari UA on a Chromium host (no touch). GROUNDED LIVE → br.apple_ua_nonwebkit + br.safari_ua_no_webkit_api + br.mobile_no_touch all fire (the iOS naive-spoof path, previously unexercised — we'd only grounded the Android naive spoof). - FIXED_TOUCH=1: the escalation past mobile_no_touch — addInitScript fakes navigator.maxTouchPoints=5 but leaves the CSS pointer surface desktop. GROUNDED LIVE → br.pointer_touch_incoherent fires and br.mobile_no_touch correctly stays quiet (maxTouchPoints>0). Lights pointer_touch_incoherent, an active rule no prior evader exercised (every desktop evader is touch-coherent; the Pixel-5 emulation sets both). Ladder now grounded end-to-end: naive-UA-spoof → mobile_no_touch; iOS-naive → apple/safari/mobile_no_touch; fixed-touch → pointer_touch_incoherent; full device-emulation → OS-leak (fingerprint_improbable/font_os). Validation (lighting unexercised active rules via faithful modes); no version bump.
…7 + killed sensor lead Tier-2 of the mobile uplevel: a mobile-targeted deep-research pass (6 angles, 26 sources, 25 claims 3-vote-verified, 18 confirmed). New radar rows: - G6 (top groundable): mobile GPU-family ↔ OS coherence — Adreno/Mali⟹Android, "Apple GPU"⟹Apple; an Android UA + "Apple GPU" (or iPhone + Adreno) is a clean incoherence, the mobile extension of webgl_os_vs_ua. FP-safe, locally groundable via a mismatched-mobile-renderer evader. - G5 (weak): WebView surface (wv token / X-Requested-With) — NOT convicting (WebView is the dominant LEGIT mobile surface + UA-spoofable); only a non-UA-vs-UA mismatch is a tell, and that's niche → X7. - X6 (external): mobile touch biometrics (Touchalytics/BeCAPTCHA) — needs real mobile traffic; adversarially synthesizable (random-vector attacks raise FAR 22-27%), so corroborating-only by nature. - X7 (external/open): iOS WKWebView client-side discriminator — none survived verification. KILLED (do not build): device-motion-sensor fingerprinting/coherence (refuted 1-2/0-3 — would be FP-prone; the research saved the build), iOS WebView Version-token/messageHandlers signals (refuted 0-3), WebView "259x uniqueness" (refuted). Next groundable rung: G6.
…GAP-1) Holistic-review finding (HIGH/security). ParseQUICInitials reassembled CRYPTO fragments with need=int(f.off)+len(f.data) then make([]byte, need-len(buf)) and NO upper bound on the offset. The CRYPTO offset is an attacker-controllable varint (up to 2^62), and the Initial AEAD keys derive from the PUBLIC DCID — so a crafted, validly-authenticated Initial with a huge CRYPTO offset could force a multi-GB allocation on the NET_RAW edge (remote OOM). Extracted the stitch into reassembleCrypto() and bounded it at maxCryptoReassembly (64 KiB): a real ClientHello (even with PQ key shares across several Initials) is a few KB, so an oversized offset is a forged/garbage Initial — skip the fragment (the rest still stitch; a truncated buffer just never parses as a hello). New unit test TestReassembleCryptoBoundsOffset drives adversarial offsets directly (the multi-packet reassembly path was previously untested + unreachable by the AEAD-gated fuzzer). edge vet + fingerprint tests green via Docker.
Holistic-review finding (HIGH/concurrency). countingConn.Read calls scanner.Feed() (writes the frame counters) on the http2 server's per-connection read goroutine, while request handlers call RapidReset()/ContinuationFlood()/ControlFrameFlood() (read those counters) from concurrent per-stream handler goroutines — with no synchronisation. Single-goroutine tests kept -race green and hid it. Converted the five counters (Headers/RSTStreams/Continuations/Settings/Pings) to atomic.Uint64 — Feed uses .Add(1), the detectors use .Load(). The parser state (prefaceSkipped/hdr/hdrFilled/payloadLeft) is single-writer (only Feed, one read loop per connection) so it needs no synchronisation. Atomics beat a mutex on this per-Read hot path. New TestH2FrameScannerConcurrentFeedAndReadIsRaceFree drives Feed concurrently with all three detectors; `CGO_ENABLED=1 go test -race` is clean on internal/fingerprint + internal/proxy. go vet + gofmt clean.
…alog (GAP-3) Holistic-review finding (HIGH/doc-gen). _MAIN_CANDIDATES matched no script in evaders/pow/ (entrypoint is under cmd/; the technique file is pow.go) or evaders/xtest-coalesce/ (probe.py), so _evader_main returned None and both vanished from docs/evasion-catalog.md — the doc claimed "20 evader tools" while 22 dirs exist, and --check passed because generation was self-consistent. Added "pow.go" and "probe.py" to _MAIN_CANDIDATES (both carry a descriptive 2-line header) and regenerated: the catalog now lists 22 evader tools including `pow` (multi-class PoW primitive) and `xtest-coalesce` (the XTEST-motion coalesced-tell terminus probe). evasion/readme drift checks green; ruff clean.
…av.language) (GAP-4) Holistic-review finding (HIGH). The mapper emits language_list_incoherent (calibration.py) but the kind was absent from DERIVABLE_KINDS, so br.language_vs_languages (active, coherence, convicting) was marked calibrated=False and silently EXCLUDED from the browserforge FP gate — an active convicting rule whose real-traffic FP rate was never measured. Two parts: (1) add "language_list_incoherent" to DERIVABLE_KINDS; (2) _fingerprint_to_dict now derives navigator.language (= languages[0], the HTML-standard invariant — browserforge carries only the list), so the coherence check actually runs on a coherent input instead of short-circuiting on an empty value. Result: calibrated browser rules 30→32 of 88; br.language_vs_languages is now exercised and shows 0% FP on the coherent corpus (FP-safe by construction — a real browser's language always equals languages[0]). test_calibration (read ⊇ DERIVABLE_KINDS) + test_mapper_coverage scope pin both green; ruff/mypy clean.
Holistic-review finding (HIGH/CI). The docs:check drift gate ran only in local `task ci`, not GitHub CI — so a PR that changed registry.yaml/corpus without `task docs` left the committed catalogs/README/matrix stale and GitHub CI passed green. And ci.yml's own header claimed a "contracts" job that did not exist (contract validation only rode transitively inside detector pytest); `task ci` also omitted the contracts task. Added two jobs mirroring the existing pinned-action pattern: `docs` (readme_stats/readme_redteam/ rule_catalog/evasion_catalog --check + the matrix git-diff gate, in harness) and `contracts` (load + validate the rule registry, in detector). Added `- task: contracts` to the local `task ci` pipeline (docs:check was already there). Verified: ci.yml parses (jobs now include contracts + docs), the contracts command prints "contracts ok", and the matrix git-diff gate is clean against the committed docs.
…P-6) Holistic-review finding (MED/provenance). Two rules had two `source:` lines each; PyYAML silently keeps the LAST, so the wrong provenance won and the correct note was dropped — and source/last_validated feed the signal-decay honesty matrix: - br.nav_property_spoofed kept a "native plugins" note (wrong subject); restored the pdfViewerEnabled/ mimeTypes own-property + FLOOR_SPOOF note. - net.ch_ua_no_grease_brand kept a Sec-Fetch note that belongs to net.sec_fetch_vs_ua; restored the Sec-CH-UA GREASE note, and gave net.sec_fetch_vs_ua (which had NO source) the Sec-Fetch provenance. Added test_registry_has_no_duplicate_yaml_keys: a SafeLoader subclass that rejects duplicate mapping keys (deep), self-checked against a synthetic dup, then run over the real registry — so a repeated key in any rule block fails CI instead of silently dropping a value. detector 261 passed (99.32%); catalog/readme fresh.
Holistic-review finding (MED). tcpfp.Store.Get treated an expired entry as a miss but never deleted it, and there was no sweep — so every distinct source IP the SYN sniffer saw was retained forever. On the NET_RAW edge (explicitly meant to face floods/scans with many distinct/spoofed source IPs) the map grew without bound. Added an amortised sweep in Put (at most once per TTL, drop expired entries — mirrors the QUIC tee's expiry sweep; O(n)-once-per-TTL, not per-Put, so a flood can't make it O(n^2)) plus delete-on-read in Get. Bounds the map to ~one TTL window of SYNs. New TestStoreSweepsExpiredOnPut: 1000 IPs + a TTL gap + one Put leaves exactly the fresh entry. edge vet + tcpfp tests green via Docker; existing TTL/overwrite tests unaffected.
…y (GAP-8) Holistic-review finding (MED/dead-code). fleet.py (fleet_signature/detect_fleets/render_fleets) was JA4-prefix clustering fully reimplemented and superseded by coordination.py (_clusters/score_corpus with the JS-divergence paradox + collision conviction gate). Nothing imported it — only its own test_fleet.py and an unwired CLI (no Taskfile target). It padded coverage with a non-current implementation. Deleted fleet.py + test_fleet.py; fixed the now-dangling [[fleet]] reference in coordination.py's docstring (it describes coordination's own JA4 grouping). harness ruff/mypy clean, 229 passed, coverage 95.68% (>95%).
Holistic-review finding (MED/doc-drift). The hand-maintained architecture.md had drifted from reality:
- ruleset_version 0.70.0 → made version-agnostic ("carries its own ruleset_version") so it can't restale.
- "~114" rules → "~130" + a pointer to the generated detection-catalog.md (the authoritative table).
- §1/§3 evader enumerations (named ~14 of 22 tools) → added the missing ones (mobile-emulation, pow,
azuretls, playwright-extra, apify-fp-inject, webkit-ua-spoof, firefox-os-spoof, xtest-coalesce) and a
pointer to the generated evasion-catalog.md so the fleet list can't restale.
- §12 CI claims → matched to reality: the `contracts` job now exists (added in GAP-5), the e2e job is the
in-process spine smoke (not docker-compose), there is no CI calibrate gate (it's local), and the new
`docs` drift gate is documented. The end-to-end testing-table row corrected likewise.
Doc-only (architecture.md is hand-written, not generated); headers clean.
…mo.py lag (GAP-10) Holistic-review finding (MED/drift). The livepage self-test page lagged the authoritative demo.py by ~15 signals, including ones backing convicting/within-session rules. demo.py stays authoritative (the detector serves it; rules validate against it), but the self-test page was under-reporting. Ported the cheap, high-value static probes into probes.ts: webgl_renderer + webgl_vendor (raw GPU identity — feed the prevalence model + the within-session br.fp_unstable convicting rule), maxtouch_desktop (the inverse of mobile_no_touch), and nav_language_primary (accept-language coherence). The remaining lag is now bounded to the heavy/async probes the lightweight page intentionally omits (WebRTC STUN, composite fp_hash, WebGPU family, the biomech stream, honeypot/adblock) — documented as an explicit, conscious scope in architecture §3 with a "add new convicting probes to livepage or list them here" guard, so the lag can't silently widen. collector typecheck + eslint + prettier + vitest green via Docker node:22; headers clean.
…rules_json guard, evader docs) Holistic-review LOW batch — the last items in the gap queue: - (a) prevalence.features_from_session dropped the vestigial "color" feature (the colour factor was removed v0.74.20; the key was computed but never read by _FACTORS/_SCORED). Test expectation updated. - (b) test_rule_firing: a parametrized test that every active convicting present-predicate single-read rule (65 of them) FIRES when its read signal is present — engine-level, so a typo'd `reads` kind (an unfireable dead rule) is caught instead of silently passing. - (c) rules_json._client_evaluable now guards the vacuous-true (empty reads => not client-evaluable, was an empty all()==True); new test_rules_json covers _client_evaluable + build() (was 0% covered). - (d) architecture §12 documents that the bare-run.mjs Playwright evaders (mobile-emulation/firefox-os-spoof/ webkit-ua-spoof) run via the shared kitsune-stealth image rather than duplicating a Dockerfile. detector 326 passed (99.32%), harness 231 passed (96.30%); mypy/ruff/headers clean.
… both exhausted Probed G6 (mobile GPU↔OS coherence) confirm-EVADES-first: the Apple-GPU half is ALREADY caught (_webgl_os maps Apple→macOS, so Android UA + "Apple GPU" already fires webgl_os_vs_ua); the Adreno/Mali→Android half is FP-UNSAFE (real Windows-on-ARM ships Adreno, ChromeOS ships Mali — both verified to correctly not fire today, and would FP if mapped to Android) → external (device-class disambiguator, X5-class); the FP-safe sliver (Adreno/Mali under an Apple UA) is redundant with the iOS-spoof tells. No new rule — avoids a real-device FP, mirroring the font_os_vs_ua / vendor_vs_ua discipline. With G6 resolved, every groundable lead (G1–G6) is done/covered/validated/external and the entire gap queue (GAP-1–11) is cleared. Both queues exhausted; the loop stops. Remaining radar items (X1–X7) are all external-data-bound and route to the grounding harness. Re-arm the loop after an external-sourcing pass.
The proxy_exit seed was Tor-only (a thin slice of real proxy egress) and the datacenter seed was AWS/GCP-only. Add the MIT-licensed X4BNet/lists_vpn feeds (~11k VPN CIDRs → proxy_exit, ~42k hosting CIDRs → datacenter) to the deploy-time refresh, with a generic one-CIDR-per-line parser and per-source floor guards (x4b_vpn/x4b_datacenter >= 1000) so a drifted source fails loud. Output stays uncommitted per the existing deploy-not-commit rule; only the parser, offline tests, and source URLs are committed. Licence verified at source: GitHub's detector reports X4BNet null, but its README carries full MIT text covering "the list itself (source files and generated output)" — confirmed before wiring (don't-trust-single-source, applied to a licence). Also adds the "Real-data sources -> grounding input" shopping list to the research radar (each vetted dataset mapped to its X-item + access + licence + grounding command) and flips X4's IP-rep half to partially-unblocked. FireHOL stays a licence-gated candidate (GPLv2 mixed-upstream aggregate); the Berke prevalence corpus is the next unlock (request research-use terms).
Resolves the 3 high Dependabot alerts (one advisory, GHSA-7mvr-c777-76hp, vulnerable < 1.55.1) across the playwright-extra, apify-fp-inject, and stealth evader manifests. Applied locally as datascry per the single-author rule (not a GitHub bot merge). Manifest-only; rebrowser-playwright/patchright are unaffected (separate packages, not flagged). Evaders run in Docker images, no committed lockfile.
A real refresh (in-memory, output uncommitted) with the deploy-time floor guard active passes and grows the feeds from a thin seed to proxy_exit 12,231 CIDRs (~10x, was Tor-only ~1.2k) and datacenter 53,297 CIDRs (~4.6x, was AWS/GCP-only ~11.6k); real AWS/Google IPs classify as datacenter, private LAN stays clean.
The grounding harness is the documented unlock for every external-data-bound frontier (X1-X7), but its CLI entry point was pragma:no-cover and had an order-dependent bug: the index-based scan only excluded --expect's value, so `grounding --build-prior out.json --expect bot caps` mistook out.json for the corpus directory. Extract a pure, tested parse_args() that consumes BOTH option values (and skips unknown flags), and slim main() to IO. Add tests for arg parsing and for the operator-facing NEEDS-ATTENTION render path (false-positive + fleet output). Verified end-to-end: the turnkey command grounds the local 96-session corpus clean (86 bot + 10 caught-via-fleet, 2 fleets). grounding.py 83% -> 99%; harness suite 234 green at 97.09%.
The X4 prevalence-prior half: a real-traffic prior is the unlock for promoting br.fingerprint_improbable (the shipped browserforge prior is same-source-blind). Berke et al. published 8,400 consented real fingerprints covering Kitsune's exact prevalence attributes — but the browser-attributes file is on Harvard Dataverse under research-use terms (no re-identification, no resharing) and isn't in the repo, so it can't be fetched in-sandbox. Build the turnkey consumer instead: berke_corpus.py maps each CSV row into the detector's own features_from_fingerprint (no bucketing duplication → the prior stays in sync with the runtime scorer) and emits ONLY the aggregate prior (frequency tables, never rows → honours the no-resharing term). Column names pinned from the published data dictionary + the repo's preprocessing notebook; offline-tested against synthetic rows in that schema (7 tests, 100% module cov, harness 240 green). When an operator accepts the terms and downloads the CSV: `uv run python -m kitsune_harness.berke_corpus <csv>` builds the real prior. Both halves of X4 now have turnkey consumers (IP-rep via X4BNet last iteration, prevalence via this); only the operator-gated Dataverse download remains. Docs: radar real-data table + iteration log + grounding runbook updated.
Reverts ef056f5. The 1.52.0 pin is a documented decision, not a stale version: the evader Dockerfiles explicitly `npm install playwright@1.52.0` because rebrowser-playwright caps at 1.52.0, so all three drivers (playwright/patchright/ rebrowser-playwright) must share ONE Chromium revision. GHSA-7mvr-c777-76hp is an ACCEPTED build-time-only risk on an evader that runs only against Kitsune's own detector + the allow-list, so the dependabot alerts are intentionally left open. My package.json bump was also ineffective (the Dockerfiles override it) and repeated exactly the mistake commit 94fa3b9 already reverted. Pins back to 1.52.0.
gitleaks-action v2 made GITHUB_TOKEN required to scan PRs; without it the secrets
job errors out ("GITHUB_TOKEN is now required") and the secret scan silently does
not run on any PR. Wire the read-only default token into the step's env so the
scan actually executes. No real leak was present — the job was failing on this
config gap, not a finding.
datascry
added a commit
that referenced
this pull request
Jul 8, 2026
…ounded) Dependabot #32/#33: golang.org/x/crypto <0.45.0 in os-spoof (indirect, v0.36.0) — the ssh/agent panic and ssh unbounded-memory advisories. Bumped to v0.53.0 (matching the other Go modules); go mod tidy, build green. os-spoof does not use x/crypto/ssh, but the bump clears the transitive alert.
datascry
added a commit
that referenced
this pull request
Jul 8, 2026
…ounded) Dependabot #32/#33: golang.org/x/crypto <0.45.0 in os-spoof (indirect, v0.36.0) — the ssh/agent panic and ssh unbounded-memory advisories. Bumped to v0.53.0 (matching the other Go modules); go mod tidy, build green. os-spoof does not use x/crypto/ssh, but the bump clears the transitive alert.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A full session of the red⇄blue arms-race loop, now research-fed and capped by a turnkey grounding harness. 12 commits, all CI-green (detector 260 · harness 234 · edge vet+test · collector tsc/eslint/vitest · headers).
Detection / evasion rungs (grounded)
fix(harness)honest FP gate — exclude internally-incoherent browserforge fingerprints (Win-UA+Linux-platform, UA/CH version skew, HeadlessChrome brands) from the legit corpus; hard bot-FP rate 1%→~0%, no rule weakened.feat(harness)coordination — catch JA4-rotating fleets via collision clustering (cluster by cloned fp_hash / replayed trace / shared WebRTC origin across distinct IPs, not just JA4). Closes a real evasion; scenario gate stays 100/100.feat(edge)ADR-0005 QUIC per-connection (DCID) attribution —InitialDCID+ DCID-keyed tee + anhttp3.Serverthat links a captured Initial to its connection'sks_sid. Plumbing built + unit-tested; the rule stays unshipped pending out-of-sandbox grounding (trusted-cert host + QUIC rotation evader).feat(detector)br.mobile_no_touch— phone/tablet UA withmaxTouchPoints == 0(device-DB-free spatial coherence from FP-Inconsistent, ACM IMC 2025). Grounded both ways, 0 FP, mirrored across demo.py + calibration mapper + livepage collector.The loop itself
docsADR-0005 (QUIC attribution design) + research radar (docs/research-radar.md) — the loop's cited intake queue, groundable-vs-external split.feat(harness)grounding harness —docs/grounding.mdrunbook +task grounding -- <dir>(kitsune_harness.grounding): one command runs per-session FP/recall + coordination + prevalence-prior rebuild over real captures. The turnkey path for when operator real-data (proxy egress / real-device / real traffic / trusted-cert QUIC) arrives.State of play
In-sandbox detection is saturated; every remaining frontier is external-data-bound and routed to the grounding harness. Net new grounded detection this session:
br.mobile_no_touch+ the coordination JA4-rotation fix.