Architecture and Telemetry

Architecture & Telemetry

This document details the internal architecture of LoomDoctor and the technical mechanics of Java 21+ Virtual Thread concurrency.

---

1. Virtual Thread Mechanics in Project Loom

Virtual threads (java.lang.Thread.ofVirtual()) are user-mode threads managed directly by the Java Virtual Machine rather than the host operating system.

+--------------------------------------------------------------------+
|                      Virtual Threads (M)                           |
|  [VT 1]  [VT 2]  [VT 3]  [VT 4]  ...  [VT 10,000]  ...  [VT 1,000,000]
+--------------------------------------------------------------------+
                                | (Mount / Unmount)
                                v
+--------------------------------------------------------------------+
|               ForkJoinPool Carrier Threads (N)                     |
|            [Worker 1]    [Worker 2]    [Worker 3]                  |
+--------------------------------------------------------------------+
                                | (1:1 mapping)
                                v
+--------------------------------------------------------------------+
|                      OS Kernel Threads (N)                         |
+--------------------------------------------------------------------+

• M : N Scheduling: Millions of virtual threads ($M$) are multiplexed over a small pool of OS carrier threads ($N$, typically matching Runtime.getRuntime().availableProcessors()).
• Non-blocking Unmount: When a virtual thread performs standard Java I/O (e.g., Socket.read(), Thread.sleep()), the JVM unmounts the virtual thread, preserving its stack in Java heap memory and freeing the carrier thread to execute another task.

---

2. Carrier Pinning Mechanics

Why Pinning Occurs

A virtual thread is said to be pinned to its carrier thread when it cannot be unmounted during a blocking operation. This occurs under two primary circumstances:

1. Synchronized Blocks & Methods: The virtual thread enters a monitor lock (synchronized (lock) { ... }) and then executes a blocking call (such as JDBC I/O or network calls). The current JVM implementation cannot unmount virtual threads holding object monitors.
2. Native Calls: When executing blocking operations inside JNI or foreign functions (Project Panama FFM).

Impact on Throughput

When a carrier thread is pinned:

• The carrier thread remains occupied waiting for the blocking call to finish.
• Other ready virtual threads waiting in the ForkJoinPool queue cannot make progress on that carrier thread.
• If all carrier threads become pinned simultaneously, the application experiences complete throughput collapse.

Pinning Diagnostics in LoomDoctor

LoomDoctor monitors pinning using a thread-safe circular buffer:

• Captures the pinned carrier thread name (ForkJoinPool-1-worker-3), the virtual thread descriptor, the duration in milliseconds, and the exact source code location.
• Provides event callbacks (PinningListener) to integrate with enterprise alerting or log aggregators.

---

3. Downstream Resource Starvation

The "Infinite Client" Illusion

Before virtual threads, platform thread pools (e.g., 200 Tomcat worker threads) acted as a natural concurrency throttle. With virtual threads, an application can effortlessly accept 100,000 simultaneous incoming HTTP requests.

However, if each request requires a database connection from a fixed-size pool (e.g., HikariCP with 20 connections):

• 20 virtual threads acquire connections.
• 99,980 virtual threads queue in memory waiting for a connection lease.
• Queue wait times spike exponentially, causing connection timeouts, cascading circuit-breaker trips, and client-side HTTP 504 Gateway Timeouts.

Starvation Ratio Algorithm

LoomDoctor evaluates pool congestion using the queue-to-capacity ratio:

$$\text{Wait Ratio} = \frac{\text{Queued Virtual Threads}}{\max(1, \text{Active Connections})}$$

• When $\text{Wait Ratio} \ge \text{threshold}$ (default: 2.0), a StarvationAlert is triggered, and LoomDoctor.diagnose() elevates system health to CRITICAL.

---

4. Remediation Strategies Recommended by LoomDoctor

1. Replace synchronized with ReentrantLock

• Before (Causes Pinning):

  public synchronized byte[] fetchExternalData() {
      return httpClient.send(...); // Pinned to carrier thread!
  }

• After (Yields Cleanly):

  private final ReentrantLock lock = new ReentrantLock();
  
  public byte[] fetchExternalData() {
      lock.lock();
      try {
          return httpClient.send(...); // Virtual thread unmounts cleanly
      } finally {
          lock.unlock();
      }
  }

2. Apply Bulkheads and Semaphores

Rather than allowing unbounded virtual threads to bombard connection pools, protect downstream resources with a java.util.concurrent.Semaphore:

private final Semaphore dbPermits = new Semaphore(50);

public Order processOrder(OrderRequest req) {
    dbPermits.acquire();
    try {
        return orderRepository.save(req);
    } finally {
        dbPermits.release();
    }
}