Fix a batch of correctness, concurrency, and resource bugs across all modules

[jbachorik]

Summary

A wide bug review across the whole codebase (core, runtime, agent/instr, compiler, client, dtrace, boot, MCP server) turned up a set of correctness, concurrency, and resource-management bugs. This PR fixes them. Each change was compiled per-module against the real dependencies (ASM 9.10.1, jctools 4.0.6, lanterna 3.1.5, slf4j, JDK compiler APIs) and the changed files are formatted with google-java-format.

Two consolidated analysis reports with the full findings (including deferred/latent items) are included under internal/analysis/.

⚠️ Action required before merge — regenerate instrumentation golden files.
The three instrumentor fixes in the last commit change the bytecode the instrumentors emit, so the btrace-instr golden data under btrace-instr/src/test/resources/instrumentorTestData/ will no longer match. Please regenerate and commit it:

./gradlew test -PupdateTestData
# then commit the updated files under btrace-instr/src/test/resources/instrumentorTestData/

I couldn't run this in my environment (the full distribution build needs the JDK 8/11 toolchains, and Gradle distribution / toolchain downloads were blocked). Everything else was verified by direct compilation and, where feasible, standalone behavioral tests.

What's fixed

Correctness — silently wrong results

• @Level gating never worked. The MethodHandle guard read the thread-local runtime (the level-0 dummy at guard time) instead of the owning probe's runtime. Guards now resolve the probe's runtime by name. (BTraceRuntimes, HandlerRepositoryImpl)
• @OnError handlers never ran and errors were swallowed. handleException invoked guarded error handlers while the thread was still entered, so their enter() prologue failed. They're now dispatched with the runtime escaped, like handleEvent. (BTraceRuntimeImplBase)
• Profiler: snapshotAndReset() never reset, reset() corrupted the recorder when threads were mid-method (NPE), and direct recursion double-counted wall time. Fixed and covered by MethodInvocationRecorderTest.
• @OnLowMemory was broken end to end (reflective arity in LowMemoryHandler, the runtime call site, and the Preprocessor codegen: constructor descriptor + trackUsage, array element type, null-LDC).
• Compiler hung forever on any enum-declared method call (e.g. TimeUnit.NANOSECONDS.toMillis(x)) — the verifier's parent-type loop never advanced. (VerifierVisitor)
• @Duration on a Kind.CALL handler always received 0, and adaptive-sampled CALL sites lost their timing feedback — onAfterCallMethod shadowed the timing context configured by onBeforeCallMethod. (Instrumentor)
• Nested new (new Outer(new Inner())) missed Kind.NEW/@Return — a single boolean collapsed under nesting; now tracked on a stack. (ObjectAllocInstrumentor)
• LineNumberInstrumentor reported line-1 instead of the actual previous line for non-consecutive line tables.
• @TLS short / @Export short fields failed with a VerifyError (TypeUtils.isPrimitive omitted short).

Protocol / wire

• V1 could not decode LIST_FAILED_EXTENSIONS (a command the client sends first), killing the session on both ends; also now throws IOException instead of a bare RuntimeException on unknown types.
• V1 ObjectOutputStream reset/flush were unsynchronized against the command write, tearing the stream under concurrent writers.
• Agent negotiation had no read timeout, so one idle/half-open connection wedged the single accept thread for all clients.
• DTraceDropCommand.write serialized the stream instead of the event (out.writeObject(out)).
• -Dio.btrace.core.cmdQueueLimit was silently dropped by the client (malformed agent-arg segment).

Concurrency

• CommandQueue.enqueue cleared the producing application thread's interrupt status and could drop commands while reporting success.
• SharedSettings.GLOBAL leaked across clients (settings and a sticky trusted escalation); each remote client now gets its own copy, and the fields are volatile.
• CircularBuffer re-delivered already-consumed commands after wrap-around and had no synchronization; rewritten and covered by tests.
• ExtensionLoaderImpl re-opened/re-appended the API jar on every call (FD/path leak) and used plain HashMaps across threads.
• Agent Client.WRITER_MAP (plain HashMap) was mutated from multiple threads; now guarded.
• MethodTracker sampling arrays made volatile (hot readers vs. resize).

Resource management

• Agent leaked a socket per non-instrument connection (-lp, -le, EXIT, malformed handshake).
• MCP server crashed the whole process on one malformed JSON line, treated blank lines as EOF, and leaked client sockets in the list/exit/detach handlers.
• CompilerHelper permanently set GLOBAL trusted and leaked a URLClassLoader per compiled class.

Robustness / API

• Perf counters (perfInt/perfString) were unreachable — the jvmstat availability probe used a broken relative resource path and the reader class was package-private. Fixed the probe and made it instantiable.
• JDK 11 JFR factory returned null where 8/9 return the EMPTY factory, NPE-ing probes on JFR-less JDK 11+.
• @OnProbe-mapped Location was shared with a process-wide-cached template and mutated in place across sessions; now deep-copied.
• BTraceTransformer regex-probe unregistration used a mismatched cache key, leaking filter entries.
• Oneliner args[N] == null filter NPE, client CLI hang on an empty argument token, and a PolicyFile.save NPE on a bare relative path.

Deferred (documented in the analysis reports, not in this PR)

The Maven/Gradle plugin extension-packaging mismatches are packaging changes best validated with an actual plugin build; they're described in internal/analysis/2026-07-05-remaining-modules-analysis.md.

Commits

1. Wide analysis report for core & runtime
2. High-severity core/runtime fixes (16)
3. Fixes across agent/instr, compiler, client, dtrace, MCP (16) + remaining-modules report
4. Instrumentor fixes (CALL @Duration, nested-new alloc, line-number off-by-one) — requires golden-file regeneration

Testing

• Per-module compilation against real dependencies on JDK 21 (all touched modules clean).
• New JUnit test: MethodInvocationRecorderTest (profiler reset/recursion semantics).
• New CircularBufferTest cases (no stale re-delivery, partial consumption, doNext).
• Standalone behavioral verification of the CircularBuffer, profiler recorder, @Level MethodHandle composition, and nested-new allocation logic.
• Not yet run: the full instrumentation golden-file suite — see the action-required note above.

🤖 Generated with Claude Code

---

This change is 

[jbachorik]

The Code Quality bot flagged btrace-core/src/main/java/io/btrace/extension/impl/ExtensionLoaderImpl.java:360 saying the JarFile is "not always closed on method exit". This is a false positive — the JarFile is intentionally retained in a class field, not leaked. The full lifecycle is:

1. Field declaration (line 59, with explanatory comment):

// Bootstrap-append bookkeeping, all guarded by bootstrapLock. appendedApiJars dedups the
// appendToBootstrapClassLoaderSearch calls so a hot path (one call per @Injected field per
// submitted script) does not reopen the same JAR and grow the bootstrap search path without
// bound; openApiJars keeps the JarFiles open for the loader's lifetime.
private final Object bootstrapLock = new Object();
private final Set<String> appendedApiJars = new HashSet<>();
private final List<JarFile> openApiJars = new ArrayList<>();

2. The method's own Javadoc:

The opened JarFile is retained for the loader's lifetime because HotSpot may need the descriptor to read class bytes after the append.

3. The class's close() method (lines 445–451) — which explicitly does not close the JarFiles:

@Override
public void close() {
  // openApiJars were registered with the bootstrap classloader via
  // appendToBootstrapClassLoaderSearch and must not be closed; the JVM may
  // continue reading class bytes from them after registration. The OS reclaims
  // file descriptors at JVM exit.
  openApiJars.clear();
}

This is a well-known idiom for Instrumentation.appendToBootstrapClassLoaderSearch(JarFile): the JarFile argument must remain open for the lifetime of the JVM, and closing it would corrupt the classloader's view of the bootstrap search path. The pre-PR code had the same pattern (the JarFile was created inline in ensureApiOnBootstrap and added to openApiJars); the ef953ac0 commit simply extracted that pattern into the new appendApiJarToBootstrap helper. The "leak" warning is identical in both versions.

The bot's static-analysis rule fires on new JarFile(...) without a try-with-resources in the same scope, but it cannot see that the JarFile is intentionally retained in a class field whose lifecycle is managed at the class level. Resolving as "not applicable — JarFile is deliberately retained in openApiJars for the loader's lifetime (and HotSpot requires this for bootstrap classloader search entries)".