Skip to content

v0.4.2-beta

Choose a tag to compare

View all tags
@Lekssays Lekssays released this 10 Jun 08:28
· 187 commits to main since this release
v0.4.2-beta
549f6ee

🚀 codebadger — v0.4.2-beta

🎯 Highlights

v0.4.2-beta is a reliability-under-load release. After v0.4.1-beta made the stack production-shaped, large batches (hundreds of CVEs, high client concurrency) still hit a cluster of load-tier failures — CPGs that built fine but wouldn't reactivate, ports that raced Docker's teardown, queue rejections under fan-out, and "ready" codebases whose Joern server had quietly died. This release fixes those: a collision-free CPG loader, build-time overlay persistence so memory-capped workers load big CPGs reliably, rotating port allocation, a decoupled queue depth, event-loop-managed server restarts, and classified, API-visible build failures. Net effect: clean RAM no longer means mysterious failures — batches run to completion and the ones that don't tell you why.

⚠️ Breaking Changes

  • None. v0.4.2-beta is drop-in over v0.4.1-beta. All new behavior is additive and defaults are safe; the new knobs below only need touching for unattended/batch drivers.

📦 What's New

Collision-Free CPG Loading (fixes the "No projects loaded" load-tier failures)

  • Every CPG file is literally named cpg.bin, so letting Joern derive the project name from the filename collided when a worker imported a second CPG or reused a workspace — importCpg then left no project open ("No projects loaded"), failing reactivation of a perfectly good build.
  • load_cpg now imports under an explicit, collision-free project name (workspace.reset; importCpg(path, name); open(name)), then runs a readiness poll that distinguishes three outcomes: loaded-and-non-empty, genuinely empty build (0 user-defined methods → fail with a distinct reason, no pointless retry), and no-project race (→ re-import once before giving up). A registration race right after importCpg no longer reads as a permanent failure.

Build-Time Overlay Persistence (memory safety on reactivation)

  • Joern applies the dataflow overlay (ReachingDefPass) the first time a CPG is opened and re-saves it into cpg.bin. Doing that on every load inside a memory-capped query worker OOMed the worker on large C/C++ trees — surfacing, again, as "No projects loaded".
  • CPG generation now applies and persists the overlays once, in the large-heap build container. Later importCpg calls just deserialize ("Overlay dataflowOss already exists – skipping"), so even a tiny tier-S (2 GB) worker loads a large CPG reliably. Best-effort: on failure the base CPG is kept and the worker falls back to recompute-on-load.
  • Build JVM is now sized from CPG_BUILD_HEAP_GB with G1GC + string dedup (-Xmx{heap}G -Xms2G -XX:+UseG1GC -XX:+UseStringDeduplication), replacing the tiny -Xmx2G default that OOMed the overlay pass.

Rotating Port Allocation (fixes "failed to become ready / connection refused" pile-ups)

  • Always handing back the lowest free port republished a just-released host port on the very next spawn — racing Docker's teardown of the old mapping (docker-proxy/iptables DNAT) and the kernel's TIME_WAIT. Failures concentrated on the first port (e.g. 14000).
  • Both PortManager (in-process) and RedisPoolStore (pool mode, via an atomic INCR cursor under the admit lock) now rotate across the whole range, giving a freed port time to fully release before reuse. A new best-effort _wait_host_port_free waits out a lingering mapping before publishing a worker.

Decoupled Build-Queue Depth (fixes ~30% generation rejections under load)

  • Pending-queue depth was tied to build_workers (workers * 4 = 8), so a 12+-way client got ~30% of generations rejected with queue_full — even though only build_workers builds ever run at once.
  • New CPG_QUEUE_MAXSIZE (default 64) sizes only the waiting room; concurrent builds — and thus build memory — stay capped at CPG_BUILD_WORKERS. Raising it does not increase memory. <=0 falls back to the old build_workers * 4.

Event-Loop-Managed Server Restarts (fixes the restart-fail churn)

  • A "ready" codebase whose Joern server had died entered a retry → fail → repeat loop. Sync MCP tools (e.g. get_cpg_status) run in worker threads with no running event loop, so background restarts had nowhere to schedule.
  • The main server loop is now captured at startup; sync tools schedule Joern server restarts onto it via run_coroutine_threadsafe. If a reload fails, the codebase is marked FAILED instead of left "ready with a dead server." New tests cover zombie/restart scenarios.

Classified, API-Visible Build Failures

  • get_cpg_status now surfaces the failure cause (error_code + human-readable error) on a failed build, instead of a bare "failed" that forced digging through container logs.
  • New _classify_cpg_build_failure distinguishes an out-of-memory build (the dominant large-project failure) from a generic frontend error and from timeouts, with the frontend output tail attached.

Large-Project Guard (opt-out for batch drivers)

  • generate_cpg now declines a local source above a size/LOC threshold — returning a large_project_warning instead of silently committing to a giant full-project build — unless the caller passes force=True.
  • Thresholds are deliberately high (default 2 GB / 2 M LOC) so only enormous trees warn. New knobs: CPG_LARGE_PROJECT_GUARD (set false for unattended/batch/eval harnesses that always intend to build and can't pass force per call), CPG_LARGE_PROJECT_MAX_MB, CPG_LARGE_PROJECT_MAX_LOC — wired through config.example.yaml, .env.example, and docker-compose.yml.

Symlink-Safe Local Copy

  • New _copy_local_source_tree skips symlinks that escape the source root when staging a local source, closing a path-escape gap in local-source ingestion.

🐳 Deployment & Infrastructure

  • docker-compose.yml now surfaces the build-sizing and scale knobs inline: CPG_BUILD_WORKERS (default 4), CPG_BUILD_HEAP_GB (default 6), MAX_MCP_CONNECTIONS (default 16, 503 past it), MAX_REPO_SIZE_MB (default 1024), plus the large-project-guard vars.
  • Pool-mode invariant unchanged: CPG_BUILD_WORKERS * CPG_BUILD_HEAP_GB ≤ JOERN_MEM_LIMIT (the build container's cap). Run python scripts/recommend_config.py before launching on a new host.

🧪 Testing

  • New suites for collision-free project naming (test_joern_client_load.py), server restart / zombie handling (test_restart_scheduling.py), large-project guard behavior (test_cpg_generator.py, test_mcp_tools.py), symlink-safe copy, and worker-pool port rotation (test_worker_pool.py). 503 tests collected (~77 new).

⚠️ Notes

  • This remains a beta release.
  • Dedicate the host to codebadger — the MCP container mounts the Docker socket (root-equivalent on the host) and uses host networking. The MCP HTTP endpoint has no built-in auth; front it with a reverse proxy / network policy.

Full Changelog: v0.4.1-beta...v0.4.2-beta

Assets 2
Loading