Scan memory benchmarks: RSS sampling, mid-scan census, parallel-chains workload

[liquidsec]

Summary

Adds three measurement layers on top of the existing peak-tracemalloc subprocess harness — what tracemalloc alone misses — plus two new workloads exercising chain shapes the wide-and-shallow web_crawl doesn't reach.

Measurement

• RSS time-series sampling via psutil (~200 ms cadence). Reports peak, end, and retention (median of last 25% of samples — the metric most sensitive to "stuck for the rest of the scan" pathologies).
• Live-event census with LiveEventTracker — a WeakSet hooked into BaseEvent.__init__ so mid-scan census is O(live events) instead of O(every Python object). Counts events by type, totals HTTP_RESPONSE body bytes.
• Lineage retention: walks every live event's parent chain back to the seed, buckets pinned events by seed. Surfaces the chain-retention pathology directly.
• Queue residence breakdown: per-checkpoint, splits live events into "in-pipeline" (in some module queue or being handled) vs "chain-only" (held alive only by parent chain). Per-module queue depths exposed for bottleneck visibility.

New workloads

• _scan_memory_deep_chain.py: strict 200-page linear chain (page N links only to page N+1). Produces a 600-event-deep parent chain, exposing structural retention that the wide workload masks.
• _scan_memory_parallel_chains.py: N independent seeds running concurrently, mirroring real-scale "many domains in flight" patterns where bodies pile up across chains even when each chain is naturally serial.

Why

Tracemalloc undercounts dramatically — it sees only Python-allocated memory, missing Rust extensions (blasthttp, blastdns, asndb, radixtarget), regex/lxml C buffers, and yara. RSS measurements show 2-18x gaps between tracemalloc and actual process memory, depending on workload. The existing peak_tracemalloc_mb metric was off by an order of magnitude in some cases.

The new metrics also distinguish peak (transient burst) from retention (stuck baseline) — different pathologies need different fixes, and the existing peak-only output didn't separate them.

[github-actions]

📊 Performance Benchmark Report

Comparing blasthttp-integration-clean (baseline) vs scan-memory-benchmarks (current)

📈 Detailed Results (All Benchmarks)

📋 Complete results for all benchmarks - includes both significant and insignificant changes

🧪 Test Name	📏 Base	📏 Current	📈 Change	🎯 Status
Bloom Filter Dns Mutation Tracking Performance	4.33ms	4.20ms	-3.1% ⚪	✅
Bloom Filter Large Scale Dns Brute Force	18.28ms	17.50ms	-4.3% ⚪	✅
Large Closest Match Lookup	358.27ms	361.20ms	+0.8% ⚪	✅
Realistic Closest Match Workload	191.57ms	192.90ms	+0.7% ⚪	✅
Event Memory Medium Scan	1784 B/event	1784 B/event	+0.0% ⚪	✅
Event Memory Large Scan	1768 B/event	1768 B/event	+0.0% ⚪	✅
Event Validation Full Scan Startup Small Batch	419.48ms	411.21ms	-2.0% ⚪	✅
Event Validation Full Scan Startup Large Batch	587.08ms	585.86ms	-0.2% ⚪	✅
Make Event Autodetection Small	31.65ms	31.66ms	+0.1% ⚪	✅
Make Event Autodetection Large	318.15ms	321.80ms	+1.1% ⚪	✅
Make Event Explicit Types	14.13ms	14.00ms	-0.9% ⚪	✅
Excavate Single Thread Small	4.043s	3.976s	-1.7% ⚪	✅
Excavate Single Thread Large	9.776s	9.772s	-0.0% ⚪	✅
Excavate Parallel Tasks Small	4.164s	4.197s	+0.8% ⚪	✅
Excavate Parallel Tasks Large	6.578s	6.603s	+0.4% ⚪	✅
Is Ip Performance	3.24ms	3.25ms	+0.5% ⚪	✅
Make Ip Type Performance	11.57ms	11.75ms	+1.6% ⚪	✅
Mixed Ip Operations	4.58ms	4.65ms	+1.6% ⚪	✅
Memory Use Web Crawl	151.6 MB	660.9 MB	+335.9% 🔴🔴🔴	⚠️
Memory Use Subdomain Enum	19.4 MB	33.3 MB	+71.7% 🔴🔴🔴	⚠️
Memory Use Deep Chain	-	722ns	New 🆕	🆕
Memory Use Parallel Chains	-	511ns	New 🆕	🆕
Scan Throughput 100	4.126s	4.107s	-0.5% ⚪	✅
Scan Throughput 1000	31.432s	31.951s	+1.7% ⚪	✅
Typical Queue Shuffle	66.69µs	66.07µs	-0.9% ⚪	✅
Priority Queue Shuffle	728.96µs	747.66µs	+2.6% ⚪	✅

🎯 Performance Summary

! 2 regressions ⚠️
  22 unchanged ✅

🔍 Significant Changes (>10%)

• Memory Use Web Crawl: 335.9% 🐌 more memory
• Memory Use Subdomain Enum: 71.7% 🐌 more memory

🆕 New Tests

• Memory Use Deep Chain: 722ns, 1385.0K ops/sec
• Memory Use Parallel Chains: 511ns, 1956.9K ops/sec

---

🐍 Python Version 3.11.15

[codecov]

Codecov Report

❌ Patch coverage is 4.30380% with 378 lines in your changes missing coverage. Please review.
✅ Project coverage is 90%. Comparing base (52dee5f) to head (688dcfa).
⚠️ Report is 3 commits behind head on blasthttp-integration-clean.

Files with missing lines	Patch %	Lines
bbot/test/benchmarks/_memory_helpers.py	0%	151 Missing ⚠️
...ot/test/benchmarks/_scan_memory_parallel_chains.py	0%	72 Missing ⚠️
bbot/test/benchmarks/_scan_memory_deep_chain.py	0%	67 Missing ⚠️
bbot/test/benchmarks/test_scan_memory.py	27%	48 Missing ⚠️
...bot/test/benchmarks/_scan_memory_subdomain_enum.py	0%	22 Missing ⚠️
bbot/test/benchmarks/_scan_memory_web_crawl.py	0%	18 Missing ⚠️

Additional details and impacted files

@@                     Coverage Diff                      @@
##           blasthttp-integration-clean   #3083    +/-   ##
============================================================
- Coverage                           91%     90%    -0%     
============================================================
  Files                              439     442     +3     
  Lines                            37860   38231   +371     
============================================================
+ Hits                             34159   34170    +11     
- Misses                            3701    4061   +360     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.