Skip to content

route TRACE log entries to debug.log, not error.log#3123

Merged
liquidsec merged 3 commits into
devfrom
error-log-no-trace
May 30, 2026
Merged

route TRACE log entries to debug.log, not error.log#3123
liquidsec merged 3 commits into
devfrom
error-log-no-trace

Conversation

@liquidsec

Copy link
Copy Markdown
Collaborator

Summary

error.log is currently a misnomer — its filter matches levelno == TRACE or levelno >= ERROR, so every self.trace() call (which modules use for "API responded with a non-2xx, here's the body" diagnostic noise) ends up in error.log alongside actual errors. A sample 24-hour scan produced 145K lines of error.log made up almost entirely of JSONDecodeErrors from crt/anubisdb/subdomaincenter parsing non-JSON error pages, 401 Incorrect API Key traces from leakix, Cloudflare-challenge HTML dumps from skymem, etc. — all already gracefully handled by the modules' try/except blocks. This drowns out anything genuinely worth surfacing.

This change scopes error.log to ERROR + CRITICAL only. TRACE entries continue to flow into debug.log (whose filter is levelno >= DEBUG, and TRACE=49 is above DEBUG=10), so no diagnostic information is lost — it just moves to the right file.

Note on the explicit != TRACE clause

bbot defines TRACE = 49 (just below CRITICAL = 50), so a plain >= ERROR filter would still match TRACE. The explicit and x.levelno != logging.TRACE excludes it. Same approach used by main_handler and the core logger's debug_handler in bbot/core/config/logger.py.

Comment thread bbot/test/test_step_2/module_tests/test_module_excavate.py Fixed
Comment thread bbot/test/test_step_2/module_tests/test_module_excavate.py Fixed
@github-actions

github-actions Bot commented May 22, 2026

Copy link
Copy Markdown
Contributor

📊 Performance Benchmark Report

Comparing dev (baseline) vs error-log-no-trace (current)

📈 Detailed Results (All Benchmarks)

📋 Complete results for all benchmarks - includes both significant and insignificant changes

🧪 Test Name 📏 Base 📏 Current 📈 Change 🎯 Status
Bloom Filter Dns Mutation Tracking Performance 4.23ms 4.30ms +1.7%
Bloom Filter Large Scale Dns Brute Force 17.45ms 17.70ms +1.4%
Large Closest Match Lookup 342.42ms 344.13ms +0.5%
Realistic Closest Match Workload 181.77ms 180.77ms -0.5%
Event Memory Medium Scan 1394 B/event 1394 B/event +0.0%
Event Memory Large Scan 1519 B/event 1519 B/event +0.0%
Event Validation Full Scan Startup Small Batch 370.06ms 369.60ms -0.1%
Event Validation Full Scan Startup Large Batch 501.33ms 506.36ms +1.0%
Make Event Autodetection Small 19.70ms 19.73ms +0.1%
Make Event Autodetection Large 201.71ms 201.38ms -0.2%
Make Event Explicit Types 8.30ms 8.25ms -0.6%
Excavate Single Thread Small 3.104s 3.082s -0.7%
Excavate Single Thread Large 8.580s 8.620s +0.5%
Excavate Parallel Tasks Small 3.340s 3.304s -1.1%
Excavate Parallel Tasks Large 5.837s 5.822s -0.2%
Intercept Throughput Small 999.93ms 1.024s +2.4%
Intercept Throughput Medium 982.76ms 974.07ms -0.9%
Is Ip Performance 2.26ms 2.26ms -0.2%
Make Ip Type Performance 202.32µs 205.43µs +1.5%
Mixed Ip Operations 2.34ms 2.34ms +0.2%
Memory Use Web Crawl 391.9 MB 390.8 MB -0.3%
Memory Use Subdomain Enum 29.2 MB 29.2 MB +0.0%
Memory Use Deep Chain 8.5 MB 8.5 MB +0.1%
Memory Use Parallel Chains 22.3 MB 22.5 MB +0.9%
Scan Throughput 100 2.836s 2.828s -0.3%
Scan Throughput 1000 20.011s 20.387s +1.9%
Typical Queue Shuffle 5.47µs 5.57µs +1.9%
Priority Queue Shuffle 26.92µs 27.52µs +2.2%

🎯 Performance Summary

No significant performance changes detected (all changes <10%)


🐍 Python Version 3.11.15

@liquidsec liquidsec force-pushed the error-log-no-trace branch from 1211d37 to 8e83e91 Compare May 22, 2026 15:06
@codecov

codecov Bot commented May 22, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90%. Comparing base (61e3aaf) to head (9093957).
⚠️ Report is 8 commits behind head on dev.

Additional details and impacted files
@@          Coverage Diff          @@
##             dev   #3123   +/-   ##
=====================================
- Coverage     90%     90%   -0%     
=====================================
  Files        441     441           
  Lines      38758   38758           
=====================================
- Hits       34679   34676    -3     
- Misses      4079    4082    +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@liquidsec liquidsec self-assigned this May 25, 2026
@liquidsec liquidsec requested a review from ausmaster May 25, 2026 15:16

@ausmaster ausmaster left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is good, in conjunction with #3127. Ensure immediately after this merge, rebase/merge dev into #3127 and merge that one as well.

@liquidsec liquidsec merged commit a43c280 into dev May 30, 2026
20 checks passed
@liquidsec liquidsec mentioned this pull request May 30, 2026
@liquidsec liquidsec mentioned this pull request Jun 9, 2026
@ausmaster ausmaster deleted the error-log-no-trace branch June 11, 2026 01:27
@ausmaster ausmaster added this to the BBOT 3.0 - blazed_elijah milestone Jun 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants