Skip to content

put an upper bound on the blasthttp version - #3389

Merged
liquidsec merged 1 commit into
devfrom
blasthttp-decode-handling
Aug 21, 2026
Merged

put an upper bound on the blasthttp version#3389
liquidsec merged 1 commit into
devfrom
blasthttp-decode-handling

Conversation

@liquidsec

@liquidsec liquidsec commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator

Puts an upper bound on the blasthttp dependency (>=0.10.0,<0.11), matching how the other dependencies are pinned.

Also backs out the decode-error tag from #3385. Nothing read it, so an undecoded body is just not passed along as content.

A body that fails to decode is no longer offered as `body` or parsed for a
`title`; `decode_error` carries the reason instead. Drops the `decode-error`
tag, which nothing read.

Also bounds the blasthttp pin at <0.11, matching the other dependencies.
@liquidsec liquidsec changed the title don't treat an undecoded response body as content put an upper bound on the blasthttp version Aug 21, 2026
@codecov

codecov Bot commented Aug 21, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90%. Comparing base (91e49ca) to head (162af48).
⚠️ Report is 1 commits behind head on dev.

Additional details and impacted files
@@          Coverage Diff          @@
##             dev   #3389   +/-   ##
=====================================
+ Coverage     90%     90%   +1%     
=====================================
  Files        454     454           
  Lines      46935   46937    +2     
=====================================
+ Hits       42196   42199    +3     
+ Misses      4739    4738    -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions

Copy link
Copy Markdown
Contributor

📊 Performance Benchmark Report

Comparing dev (baseline) vs blasthttp-decode-handling (current)

📈 Detailed Results (All Benchmarks)

📋 Complete results for all benchmarks - includes both significant and insignificant changes

🧪 Test Name 📏 Base 📏 Current 📈 Change 🎯 Status
Bloom Filter Dns Mutation Tracking Performance 3.86ms 3.84ms -0.6%
Bloom Filter Large Scale Dns Brute Force 15.85ms 15.17ms -4.3%
Large Closest Match Lookup 334.11ms 343.53ms +2.8%
Realistic Closest Match Workload 182.19ms 179.98ms -1.2%
Event Memory Medium Scan 1404 B/event 1405 B/event +0.1%
Event Memory Large Scan 1527 B/event 1527 B/event +0.0%
Event Validation Full Scan Startup Small Batch 394.11ms 395.44ms +0.3%
Event Validation Full Scan Startup Large Batch 508.11ms 504.15ms -0.8%
Make Event Autodetection Small 20.70ms 19.92ms -3.8%
Make Event Autodetection Large 208.28ms 205.52ms -1.3%
Make Event Explicit Types 8.55ms 8.59ms +0.5%
Excavate Single Thread Small 3.730s 3.658s -1.9%
Excavate Single Thread Large 9.336s 9.212s -1.3%
Excavate Parallel Tasks Small 3.859s 3.830s -0.8%
Excavate Parallel Tasks Large 5.941s 5.890s -0.9%
Intercept Throughput Small 880.79ms 865.08ms -1.8%
Intercept Throughput Medium 867.52ms 868.24ms +0.1%
Dns Throughput Quiet 2.637s 2.559s -2.9%
Dns Throughput Loaded 1.710s 1.706s -0.3%
Dns Throughput Inherited 2.195s 2.296s +4.6%
Is Ip Performance 2.34ms 2.28ms -2.5%
Make Ip Type Performance 210.32µs 210.06µs -0.1%
Mixed Ip Operations 2.45ms 2.36ms -3.4%
Memory Use Web Crawl 1.0 MB 1.0 MB +1.0%
Memory Use Subdomain Enum 28.3 MB 28.3 MB +0.0%
Memory Use Deep Chain 5.0 MB 5.0 MB +0.0%
Memory Use Parallel Chains 13.2 MB 11.0 MB -16.6% 🟢🟢 🚀
Scan Throughput 100 2.818s 2.685s -4.7%
Scan Throughput 1000 18.763s 18.302s -2.5%
Typical Queue Shuffle 5.49µs 5.47µs -0.3%
Priority Queue Shuffle 26.79µs 27.07µs +1.1%

🎯 Performance Summary

+ 1 improvement 🚀
  30 unchanged ✅

🔍 Significant Changes (>10%)

  • Memory Use Parallel Chains: 16.6% 🚀 less memory

🐍 Python Version 3.11.16

@liquidsec
liquidsec merged commit 9d516e8 into dev Aug 21, 2026
19 checks passed
@liquidsec
liquidsec deleted the blasthttp-decode-handling branch August 21, 2026 20:20
liquidsec added a commit that referenced this pull request Aug 24, 2026
* trufflehog: stream HTTP body via stdin, include source url on findings

* add js-audit preset

* bump trufflehog _module_threads to 2

* Update trufflehog

* [create-pull-request] automated change

* Bump pre-commit from 4.6.0 to 4.6.1

Bumps [pre-commit](https://github.com/pre-commit/pre-commit) from 4.6.0 to 4.6.1.
- [Release notes](https://github.com/pre-commit/pre-commit/releases)
- [Changelog](https://github.com/pre-commit/pre-commit/blob/main/CHANGELOG.md)
- [Commits](pre-commit/pre-commit@v4.6.0...v4.6.1)

---
updated-dependencies:
- dependency-name: pre-commit
  dependency-version: 4.6.1
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump actions/setup-python from 6 to 7 in the github-actions group

Bumps the github-actions group with 1 update: [actions/setup-python](https://github.com/actions/setup-python).


Updates `actions/setup-python` from 6 to 7
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](actions/setup-python@v6...v7)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump fastapi from 0.139.2 to 0.140.0

Bumps [fastapi](https://github.com/fastapi/fastapi) from 0.139.2 to 0.140.0.
- [Release notes](https://github.com/fastapi/fastapi/releases)
- [Commits](fastapi/fastapi@0.139.2...0.140.0)

---
updated-dependencies:
- dependency-name: fastapi
  dependency-version: 0.140.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump pytest-env from 1.6.0 to 1.7.0

Bumps [pytest-env](https://github.com/pytest-dev/pytest-env) from 1.6.0 to 1.7.0.
- [Release notes](https://github.com/pytest-dev/pytest-env/releases)
- [Commits](pytest-dev/pytest-env@1.6.0...1.7.0)

---
updated-dependencies:
- dependency-name: pytest-env
  dependency-version: 1.7.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump mmh3 from 5.2.0 to 5.2.1

Bumps [mmh3](https://github.com/hajimes/mmh3) from 5.2.0 to 5.2.1.
- [Release notes](https://github.com/hajimes/mmh3/releases)
- [Changelog](https://github.com/hajimes/mmh3/blob/master/CHANGELOG.md)
- [Commits](hajimes/mmh3@v5.2.0...v5.2.1)

---
updated-dependencies:
- dependency-name: mmh3
  dependency-version: 5.2.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump cachetools from 7.1.4 to 7.1.6

Bumps [cachetools](https://github.com/tkem/cachetools) from 7.1.4 to 7.1.6.
- [Changelog](https://github.com/tkem/cachetools/blob/master/CHANGELOG.rst)
- [Commits](tkem/cachetools@v7.1.4...v7.1.6)

---
updated-dependencies:
- dependency-name: cachetools
  dependency-version: 7.1.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump orjson from 3.11.7 to 3.11.9

Bumps [orjson](https://github.com/ijl/orjson) from 3.11.7 to 3.11.9.
- [Release notes](https://github.com/ijl/orjson/releases)
- [Changelog](https://github.com/ijl/orjson/blob/master/CHANGELOG.md)
- [Commits](ijl/orjson@3.11.7...3.11.9)

---
updated-dependencies:
- dependency-name: orjson
  dependency-version: 3.11.9
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump regex from 2026.7.10 to 2026.7.19

Bumps [regex](https://github.com/mrabarnett/mrab-regex) from 2026.7.10 to 2026.7.19.
- [Changelog](https://github.com/mrabarnett/mrab-regex/blob/hg/changelog.txt)
- [Commits](mrabarnett/mrab-regex@2026.7.10...2026.7.19)

---
updated-dependencies:
- dependency-name: regex
  dependency-version: 2026.7.19
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Fix docker_pull crash on failed registry request

* Bump uvicorn from 0.39.0 to 0.51.0

Bumps [uvicorn](https://github.com/Kludex/uvicorn) from 0.39.0 to 0.51.0.
- [Release notes](https://github.com/Kludex/uvicorn/releases)
- [Changelog](https://github.com/Kludex/uvicorn/blob/main/docs/release-notes.md)
- [Commits](Kludex/uvicorn@0.39.0...0.51.0)

---
updated-dependencies:
- dependency-name: uvicorn
  dependency-version: 0.51.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* ci: pass benchmark refs through env and quote them (#3354)

Co-authored-by: Paul Mueller <paul.mueller08@gmail.com>

* Reinstall module deps when they're missing from the environment (#3338)

* Reinstall module deps when they're missing from the environment

The setup status cache recorded a module's deps as installed and never
rechecked them, so a rebuilt virtualenv (e.g. `uv sync`) silently left
modules broken until ~/.bbot was wiped by hand. Verify a module's pip
packages are actually present before honoring a cache hit.

Also promote `packaging` to a direct dependency; it was only present
transitively via ansible.

* Remove dead else branch, honor custom pip constraints

The else branch in install()'s dependency loop was unreachable structure;
the preceding branch always continues. Collapsing it also flattens the
nested if/else into elif.

pip_install() overwrote custom constraints with bbot's own instead of
using them as a fallback. No module sets deps_pip_constraints today, so
this changes no current behavior.

* Bump packaging from 26.0 to 26.2 (#3365)

Bumps [packaging](https://github.com/pypa/packaging) from 26.0 to 26.2.
- [Release notes](https://github.com/pypa/packaging/releases)
- [Changelog](https://github.com/pypa/packaging/blob/main/CHANGELOG.rst)
- [Commits](pypa/packaging@26.0...26.2)

---
updated-dependencies:
- dependency-name: packaging
  dependency-version: '26.2'
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump pytest-httpserver from 1.1.4 to 1.1.5 (#3364)

Bumps [pytest-httpserver](https://github.com/csernazs/pytest-httpserver) from 1.1.4 to 1.1.5.
- [Release notes](https://github.com/csernazs/pytest-httpserver/releases)
- [Changelog](https://github.com/csernazs/pytest-httpserver/blob/master/CHANGES.rst)
- [Commits](csernazs/pytest-httpserver@1.1.4...1.1.5)

---
updated-dependencies:
- dependency-name: pytest-httpserver
  dependency-version: 1.1.5
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump lxml from 6.1.0 to 6.1.1 (#3362)

Bumps [lxml](https://github.com/lxml/lxml) from 6.1.0 to 6.1.1.
- [Release notes](https://github.com/lxml/lxml/releases)
- [Changelog](https://github.com/lxml/lxml/blob/master/CHANGES.txt)
- [Commits](lxml/lxml@lxml-6.1.0...lxml-6.1.1)

---
updated-dependencies:
- dependency-name: lxml
  dependency-version: 6.1.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump pip from 26.1.2 to 26.2 (#3360)

Bumps [pip](https://github.com/pypa/pip) from 26.1.2 to 26.2.
- [Changelog](https://github.com/pypa/pip/blob/main/NEWS.rst)
- [Commits](pypa/pip@26.1.2...26.2)

---
updated-dependencies:
- dependency-name: pip
  dependency-version: '26.2'
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump uvicorn from 0.51.0 to 0.52.0 (#3363)

Bumps [uvicorn](https://github.com/Kludex/uvicorn) from 0.51.0 to 0.52.0.
- [Release notes](https://github.com/Kludex/uvicorn/releases)
- [Changelog](https://github.com/Kludex/uvicorn/blob/main/docs/release-notes.md)
- [Commits](Kludex/uvicorn@0.51.0...0.52.0)

---
updated-dependencies:
- dependency-name: uvicorn
  dependency-version: 0.52.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump fastapi from 0.140.0 to 0.141.1 (#3361)

Bumps [fastapi](https://github.com/fastapi/fastapi) from 0.140.0 to 0.141.1.
- [Release notes](https://github.com/fastapi/fastapi/releases)
- [Commits](fastapi/fastapi@0.140.0...0.141.1)

---
updated-dependencies:
- dependency-name: fastapi
  dependency-version: 0.141.1
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump pip from 26.2 to 26.2.1 (#3369)

Bumps [pip](https://github.com/pypa/pip) from 26.2 to 26.2.1.
- [Changelog](https://github.com/pypa/pip/blob/main/NEWS.rst)
- [Commits](pypa/pip@26.2...26.2.1)

---
updated-dependencies:
- dependency-name: pip
  dependency-version: 26.2.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump packaging from 26.2 to 26.3 (#3371)

Bumps [packaging](https://github.com/pypa/packaging) from 26.2 to 26.3.
- [Release notes](https://github.com/pypa/packaging/releases)
- [Changelog](https://github.com/pypa/packaging/blob/main/CHANGELOG.rst)
- [Commits](pypa/packaging@26.2...26.3)

---
updated-dependencies:
- dependency-name: packaging
  dependency-version: '26.3'
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Parallelize test suite with pytest-xdist (#3353)

* test: parallelize suite with pytest-xdist

Test servers bound fixed ports (8888/9999/5556) and 58 test files hardcoded
those in URLs, so the suite could only ever run serially. CI spent ~58 min in
a single pytest call while using one of four cores.

- bbot/test/ports.py: resolve server ports per xdist worker
- conftest.py: tolerate listener=None (workers inherit _BBOT_LOGGING_SETUP)
- 57 test files: hardcoded host:port -> constants from bbot.test.ports

localhost and 127.0.0.1 spellings are kept distinct; virtualhost tests assert
on the Host header, where they are not interchangeable. test_regexes.py is
untouched, its literals are regex fixtures, not references to the test server.

* test: isolate per-worker state for parallel runs

Beyond the server ports, three things were process-global and would corrupt
each other under xdist:

- BBOT home (/tmp/.bbot_test): shared caches and scan output, and
  pytest_sessionfinish rmtree's it. The first worker to finish would delete
  the directory out from under every worker still running.
- uvicorn (8978) and websockets (8765) servers: bound fixed ports.
- docker-backed tests (kafka, elastic, mongo, mysql, nats, postgres,
  rabbitmq): fixed container names and fixed host port bindings, so they
  genuinely cannot run concurrently. They are now pinned to a single worker
  via xdist_group, keyed off the skip_distro_tests flag they already set.

Renamed ports.py -> worker.py since it now covers more than ports. Port bases
were chosen with distinct mod-100 residues; verified collision-free and clear
of the ephemeral range for up to 64 workers.

* ci: run tests in parallel

pytest-xdist added to the dev group; workflow runs with -n logical
--dist loadgroup. loadgroup (not loadfile) is required so the xdist_group
marker on the docker-backed tests is honored.

* test: bound docker service waits and size worker count by memory

Two problems surfaced running the suite 16-way on a 16-core/15GB box.

Elasticsearch was OOM-killed (exit 137): 16 workers hold ~10GB, leaving
under 3GB, and it wants ~2GB of heap. The test then waited for it in an
unbounded 'while True', so a container that was already dead hung the run
at 99% until the 1200s global timeout. Same unbounded loop in the mongo
test. Both now stop after CONTAINER_READY_TIMEOUT (180s, overridable) and
say what happened, including the exit-137 hint.

'-n logical' is the wrong worker count when cores outnumber GB. worker_count.py
bounds by cores AND by available memory, keeping 5GB free for the docker-backed
services. A 4-core/16GB github runner is unaffected (still 4); this box goes
16 -> 15. Override with BBOT_TEST_WORKERS.

* test: fix parallel-run regressions missed by the first port pass

Running the suite under xdist surfaced failures that were my own, not
pre-existing. Four separate gaps:

Logging. Workers inherit _BBOT_LOGGING_SETUP from the pytest parent, so
BBOTLogger skipped setup entirely and debug.log was never written. The
previous 'if listener is not None' guard hid that instead of fixing it,
and log-reading tests failed with 'assert -retries 1 in ""'. Workers now
clear the flag so each sets up its own logging, and conftest raises if a
process somehow still has no listener.

Port literals the first pass did not match, because it only looked for
127.0.0.1:PORT and localhost:PORT:
  * synthetic hostnames carrying the port (test.example:8888, bad.dns:8888)
  * the bbot_other_httpservers fixture, six servers hardcoded to 8888/8889
  * config values like {"ports": "8888"}, which are not URLs
  * 127.0.0.0:9999 (the .0 address, excluded along with .1)
  * filename/slug spellings: 127.0.0.1.8888 and http-127-0-0-1-8888
  * mocked masscan JSON, whose port is what the module 'discovers'

Hardcoded /tmp/.bbot_test paths, which my own per-worker home broke.

Adds BBOT_TEST_PORT_OFFSET so a second copy of the suite can run beside
one that already owns the base ports, instead of the two colliding and
looking like a worker-isolation bug.

* test: remove ordering dependency and fix integer port comparisons

Full-suite A/B surfaced the last regressions from parallelization.

asset_inventory: TestAsset_InventoryEmitPrevious runs with use_previous=True,
which reads the asset-inventory.csv left behind by an earlier scan of the same
name. It was relying on TestAsset_Inventory having already run. That is
order-dependent, and xdist guarantees neither order nor worker affinity, so it
failed whenever it was scheduled first or landed on another worker (each worker
has its own BBOT home). It now seeds the CSV itself in setup_before_prep, which
makes it independent and drops its runtime from ~20s to ~4s since it no longer
needs the portscan.

Port comparisons written as bare integers (e.port == 8888) in test_scope.py and
test_module_fingerprintx.py. The earlier passes only rewrote string literals, so
these were missed.

Also groups tests that share a scan_name onto one worker, alongside the existing
docker grouping. Not sufficient on its own for the ordering problem above, but
correct for co-locating tests that share a scan home.

* test: stop hardcoding the BBOT home directory name in path assertions

git_clone writes cloned repos to git_repos/<bbot home name>/<repo>, so the
home directory's *basename* shows up inside the paths these tests assert on.
They spelled it '.bbot_test', which stopped matching once each xdist worker
got its own home ('.bbot_test_gw0').

Export BBOT_TEST_DIR_NAME from worker.py and use it instead.

Worth noting these reproduce with '-n 1', a single worker with no concurrency
at all. They were never race conditions, just the per-worker home rename, and
checking '-n 1' before '-n 4' would have found that much faster.

* test: fix port in a regex literal and a self-poisoning cli precondition

test_module_robots matched the server URL with a raw regex string, where the
port sits behind escaped dots (r"http://127\.0\.0\.1:8888/..."). The earlier
rewrites looked for plain literals and skipped it, so the test failed whenever
it landed on a worker whose port was not 8888.

test_cli_presets asserts a scan output dir does not exist yet, and removes it
at the end of the test. Any earlier failure or interrupted run leaves it in
place, and every later run against that BBOT home then fails on the assertion
rather than on the real problem. It now clears the directory up front, so the
precondition is established instead of assumed.

Both showed up as intermittent (~25% of runs) rather than deterministic, which
is what made them the last two to be found.

* test: cut dead code and de-duplicate the container wait

Net removal pass over the parallelization work.

Removed:
- BBOT_TEST_PORT_OFFSET and port_offset(). Added for running two copies of the
  suite side by side while benchmarking. Nothing outside worker.py used it.
- LOCALHOST_SSL_HOSTPORT, the BASE_* port aliases, and the worker_index()
  wrapper. All unused or single-use indirection.
- OTHER_HTTPSERVER_PORT, which was literally 'OTHER_HTTPSERVER_PORT =
  HTTPSERVER_PORT'. Renamed to HTTPSERVER_PORT/HTTPSERVER_PORT_ALT at its one
  call site.
- The scan_name xdist grouping. It compensated for asset_inventory's ordering
  dependency, which the previous commit removed by having the test seed its own
  input, so the grouping no longer does anything. Verified: 3x green at 4
  workers without it.
- worker_count's -v flag and MIN_WORKERS floor. The floor guaranteed 2 workers
  even when the memory arithmetic said there was room for none, which is an
  assumption I never validated; a 2-core/4GB box now correctly gets 1.
- A dead 'import time' the refactor exposed.

De-duplicated:
- The container readiness loop was copy-pasted between elastic (sync) and mongo
  (async), each with its own deadline handling and OOM hint. Now one
  wait_for_container() helper that takes either kind of callable.

worker.py 154 -> 100 lines, worker_count.py 64 -> 38, conftest hook 40 -> 13.

* test: fix gowitness hardcoded ports and shared home directory

Caught by CI on this PR, not locally: gowitness needs its binary installed, so
both classes error out on my machine and never reached the assertions.

Two bugs:

The multiport test asserted on bare port substrings ('8888' in url) rather than
URL-shaped literals, so the earlier rewrites did not match them. On any worker
but the first the ports are 9188/10299 and the assertion could never pass.

Both classes also pinned an explicit home directory under /tmp and rmtree'd it
at class-definition time, which every worker does to the same path. Workers
deleted each other's dependency installs mid-run, which is where the
FileNotFoundError on the depsinstaller artifact came from. Now worker-scoped.

Worth noting these run with deps.behavior=force_install, so each worker
installs its own copy of chrome, ~650MB. That is fine on a runner where /tmp is
disk-backed, but it multiplies with worker count.

* test: fix a hardcoded port in the virtualhost canary regex

The force_basehost wildcard-canary check still matched a literal 8888, so on
any worker but gw0 it never matched. Both that branch and the fallthrough
return 404, which is why it did not show up as a failure.

* test: give the remaining fixed /tmp paths a per-worker name

Same class of bug as the gowitness home directory that CI caught: paths that
are fine serially but shared once workers run concurrently.

- test_python_api used one BBOT home across two tests, so the dep installer
  wrote and read the same artifacts directory from two workers
- test_presets wrote a fixed custom preset dir
- test_command and test_web wrote fixed scratch files, then read them back

* ci: skip the benchmark PR comment on fork pull requests

A pull_request run from a fork gets a read-only GITHUB_TOKEN regardless of the
permissions block, so github-script fails with

  HttpError: Resource not accessible by integration (403)
  POST /repos/.../issues/{n}/comments

and takes the whole job red. Every fork PR has a failing Performance Benchmarks
check for a reason that has nothing to do with performance, which trains people
to ignore a red check.

Skip the comment step when the head repo is not this repo. The benchmark still
runs and the report is still uploaded as an artifact, so nothing is lost except
a comment that could never have been posted.

* test: cover the worker count and isolation helpers

Codecov flagged worker_count.py at 0 percent: the workflow runs it as a script,
so pytest never imported it. worker.py was only covered incidentally.

These two modules decide how many workers run and how each one gets its own
ports and directories, so a silent regression there either overloads the runner
or lets workers collide.

Verified by mutation: removing the max(1, ...) clamp, dropping the memory cap,
and swapping sched_getaffinity for os.cpu_count each fail these tests.

* Bump uvicorn from 0.52.0 to 0.52.1 (#3368)

Bumps [uvicorn](https://github.com/Kludex/uvicorn) from 0.52.0 to 0.52.1.
- [Release notes](https://github.com/Kludex/uvicorn/releases)
- [Changelog](https://github.com/Kludex/uvicorn/blob/main/docs/release-notes.md)
- [Commits](Kludex/uvicorn@0.52.0...0.52.1)

---
updated-dependencies:
- dependency-name: uvicorn
  dependency-version: 0.52.1
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Shane Engelman <haroboy876@gmail.com>

* Bump cachetools from 7.1.6 to 7.1.7 (#3372)

Bumps [cachetools](https://github.com/tkem/cachetools) from 7.1.6 to 7.1.7.
- [Changelog](https://github.com/tkem/cachetools/blob/master/CHANGELOG.rst)
- [Commits](tkem/cachetools@v7.1.6...v7.1.7)

---
updated-dependencies:
- dependency-name: cachetools
  dependency-version: 7.1.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Shane Engelman <haroboy876@gmail.com>

* Bump websockets from 16.0 to 16.1.1 (#3312)

Bumps [websockets](https://github.com/python-websockets/websockets) from 16.0 to 16.1.1.
- [Release notes](https://github.com/python-websockets/websockets/releases)
- [Commits](python-websockets/websockets@16.0...16.1.1)

---
updated-dependencies:
- dependency-name: websockets
  dependency-version: 16.1.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Shane Engelman <haroboy876@gmail.com>

* Bump ruff from 0.15.22 to 0.16.1 (#3370)

* Bump ruff from 0.15.22 to 0.16.1

Bumps [ruff](https://github.com/astral-sh/ruff) from 0.15.22 to 0.16.1.
- [Release notes](https://github.com/astral-sh/ruff/releases)
- [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md)
- [Commits](astral-sh/ruff@0.15.22...0.16.1)

---
updated-dependencies:
- dependency-name: ruff
  dependency-version: 0.16.1
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* style: ruff format for ruff 0.16

ruff 0.16 formats python code blocks inside markdown by default, so the
existing docs fail 'ruff format --check' under the bumped version. Apply
the formatter (10 files, docs only) to make the lint step pass.

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Update nuclei (#3367)

Co-authored-by: Shane Engelman <haroboy876@gmail.com>

* [create-pull-request] automated change (#3373)

* Fix custom YARA rules matching but emitting nothing (#3374)

* fix custom YARA rule emission and rule splitting

CustomExtractor.process() called .get() on YaraRuleSettings, which has no
get(), so every custom rule match raised AttributeError before the event was
emitted. The scan logged one error per match and reported zero findings.

Replace the regex-based rule splitter with a brace-depth-aware parser that
skips comments, string literals, and regex literals, so braces inside them no
longer truncate or drop rules. Rule names now tolerate arbitrary whitespace,
import statements are collected and preserved, and unparseable input fails
loudly instead of silently dropping a rule.

* test: use per-worker HTTPSERVER_URL in custom YARA tests

The two custom-YARA module tests hardcoded http://127.0.0.1:8888/ as
their target. Under pytest-xdist the httpserver port is offset per
worker (worker_port(8888) in bbot/test/worker), so 8888 is only correct
on gw0. On any other worker the scan hit the wrong (or no) server, so no
HTTP_RESPONSE was produced, the custom rule never matched, and the test
failed with 'produced no FINDING, got: []'. TestExcavateYaraCustomEdgeCases
passed only because it happened to land on gw0.

Point both at HTTPSERVER_URL like every other test in this file. Verified
green under -n 2 --dist loadgroup, which reproduced the original failure.

---------

Co-authored-by: Shane Engelman <shane.engelman@blacklanternsecurity.com>

* Raise process pool max_tasks_per_child from 25 to 250 (#3375)

Under spawn, retiring a worker every 25 tasks forces a cold re-import of
the callback module (~742ms for badsecrets). badsecrets runs on every
HTTP_RESPONSE and leaks ~0 memory (RSS flat to 5000 tasks), so 25 costs a
~25x throughput drop and ~130x p95 blowup for no benefit. 250 recovers
full throughput while keeping the safety net for the heavy infrequent
tenants (kreuzberg, bevigil, waf_bypass).

* Update trufflehog (#3376)

* Bump pip from 26.2 to 26.2.1 (#3380)

Bumps [pip](https://github.com/pypa/pip) from 26.2 to 26.2.1.
- [Changelog](https://github.com/pypa/pip/blob/main/NEWS.rst)
- [Commits](pypa/pip@26.2...26.2.1)

---
updated-dependencies:
- dependency-name: pip
  dependency-version: 26.2.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump tldextract from 5.3.1 to 5.3.2 (#3381)

Bumps [tldextract](https://github.com/john-kurkowski/tldextract) from 5.3.1 to 5.3.2.
- [Release notes](https://github.com/john-kurkowski/tldextract/releases)
- [Changelog](https://github.com/john-kurkowski/tldextract/blob/master/CHANGELOG.md)
- [Commits](john-kurkowski/tldextract@5.3.1...5.3.2)

---
updated-dependencies:
- dependency-name: tldextract
  dependency-version: 5.3.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump packaging from 26.2 to 26.3 (#3378)

Bumps [packaging](https://github.com/pypa/packaging) from 26.2 to 26.3.
- [Release notes](https://github.com/pypa/packaging/releases)
- [Changelog](https://github.com/pypa/packaging/blob/main/CHANGELOG.rst)
- [Commits](pypa/packaging@26.2...26.3)

---
updated-dependencies:
- dependency-name: packaging
  dependency-version: '26.3'
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump ruff from 0.16.1 to 0.16.2 (#3379)

Bumps [ruff](https://github.com/astral-sh/ruff) from 0.16.1 to 0.16.2.
- [Release notes](https://github.com/astral-sh/ruff/releases)
- [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md)
- [Commits](astral-sh/ruff@0.16.1...0.16.2)

---
updated-dependencies:
- dependency-name: ruff
  dependency-version: 0.16.2
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump pre-commit from 4.6.1 to 4.6.2 (#3383)

Bumps [pre-commit](https://github.com/pre-commit/pre-commit) from 4.6.1 to 4.6.2.
- [Release notes](https://github.com/pre-commit/pre-commit/releases)
- [Changelog](https://github.com/pre-commit/pre-commit/blob/main/CHANGELOG.md)
- [Commits](pre-commit/pre-commit@v4.6.1...v4.6.2)

---
updated-dependencies:
- dependency-name: pre-commit
  dependency-version: 4.6.2
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump xxhash from 3.8.1 to 4.0.0 (#3382)

Bumps [xxhash](https://github.com/ifduyue/python-xxhash) from 3.8.1 to 4.0.0.
- [Release notes](https://github.com/ifduyue/python-xxhash/releases)
- [Changelog](https://github.com/ifduyue/python-xxhash/blob/master/CHANGELOG.rst)
- [Commits](ifduyue/python-xxhash@v3.8.1...v4.0.0)

---
updated-dependencies:
- dependency-name: xxhash
  dependency-version: 4.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* bump blasthttp to 0.10.0 (#3385)

Surface Response.decode_error on HTTP_RESPONSE events and tag them
decode-error, so nothing treats an undecoded body as content.

Pass redirect_cookies through WebHelper.request().

* Bump version to 3.0.2 (#3387)

* don't treat an undecoded response body as content (#3389)

A body that fails to decode is no longer offered as `body` or parsed for a
`title`; `decode_error` carries the reason instead. Drops the `decode-error`
tag, which nothing read.

Also bounds the blasthttp pin at <0.11, matching the other dependencies.

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: blsaccess <info@blacklanternsecurity.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Kobi Hikri <kobi.hikri@gmail.com>
Co-authored-by: Shane Engelman <haroboy876@gmail.com>
Co-authored-by: BLS <114500254+blsaccess@users.noreply.github.com>
Co-authored-by: Shane Engelman <shane.engelman@blacklanternsecurity.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants