Skip to content

Watchdog Cleanup Improvements

Jump to bottom
Sathishkumar edited this page Aug 14, 2026 · 2 revisions

Watchdog Cleanup Improvements

Overview

The current watchdog implementation successfully provides request timeout monitoring but introduces an unintended side effect during disconnect. Because the watchdog relies on periodic sleep_for() intervals, shutdown responsiveness becomes dependent on the configured timeout interval rather than the actual cleanup work being performed.

This proposal introduces a minimal change to the existing implementation by replacing timer-based sleeping with a condition-variable-based timed wait. The solution preserves the existing timeout monitoring behavior while allowing the watchdog thread to be awakened immediately during shutdown.

Current Behavior

Today, the watchdog thread spends most of its lifetime waiting inside:

std::this_thread::sleep_for(timeoutInterval);

During disconnect:

  1. Gateway requests shutdown.
  2. Gateway attempts to join the watchdog thread.
  3. Watchdog may still be sleeping.
  4. Disconnect waits until sleep expires.
  5. Watchdog wakes and exits.
  6. Cleanup completes.
flowchart TD

    A --> B[Set Shutdown Flag]

    B --> C[join Watchdog Thread]

    C --> D[Watchdog Sleeping]

    D --> E[Wait For Timeout Interval]

    E --> F[Watchdog Wakes]

    F --> G[Observe Shutdown]

    G --> H[Thread Exit]

    H --> I[Disconnect Complete]
Loading

Limitation

The delay is unrelated to pending requests, queue size, or cleanup work.

Even when the system is completely idle, disconnect may still wait for the watchdog's timer to expire before cleanup can proceed.

Proposed Improvement

Replace the existing timer-based sleep with a condition-variable-based timed wait.

cv.wait_for(lock, timeoutInterval);

The watchdog continues waking periodically to process request timeouts, but it can now also be awakened immediately when disconnect is initiated.

During shutdown:

shutdownRequested = true;
cv.notify_all();

This immediately interrupts the wait and allows the watchdog thread to terminate without waiting for the timeout interval to expire.

Proposed Lifecycle

sequenceDiagram

    participant Gateway
    participant Watchdog
    participant ConditionVariable

    Gateway->>Watchdog: Start Thread

    loop Timeout Monitoring
        Watchdog->>ConditionVariable: wait_for(timeoutInterval)

        alt Timeout Expired
            ConditionVariable-->>Watchdog: Wake
            Watchdog->>Watchdog: Scan Pending Requests
        end
    end

    Gateway->>ConditionVariable: notify_all()

    ConditionVariable-->>Watchdog: Immediate Wake

    Watchdog->>Watchdog: Detect Shutdown

    Watchdog-->>Gateway: Exit
Loading

Disconnect Flow After Improvement

flowchart TD

    A[disconnect]
        --> B[Set Shutdown Flag]

    B --> C[notify_all]

    C --> D[Watchdog Immediately Wakes]

    D --> E[Observe Shutdown]

    E --> F[Thread Exit]

    F --> G[join Returns]

    G --> H[Disconnect Complete]
Loading

Benefits

Minimal Code Changes

The existing watchdog architecture, timeout processing logic, correlation queues, and request lifecycle management remain unchanged.

Immediate Shutdown Response

The watchdog no longer needs to wait for the timeout interval to expire before observing a shutdown request.

Preserves Existing Behavior

Request timeout monitoring continues to function exactly as it does today.

Reduced Disconnect Latency

Disconnect completion is no longer tied to the watchdog polling interval.

Low Risk Implementation

The change is localized to the watchdog waiting mechanism and can be introduced without impacting public APIs or transport semantics.

Tradeoffs

While this improvement eliminates the disconnect delay, it does not address the broader architectural concerns identified in Architectural Challenges.

The Gateway remains responsible for:

  • Promise management
  • Request correlation
  • Timeout monitoring
  • Watchdog lifecycle management
  • Synchronous API abstraction

As a result, this proposal should be viewed as an incremental improvement that resolves the shutdown responsiveness issue while preserving the existing architecture.

Why This Isn't Enough

This proposal improves disconnect responsiveness but does not reduce the overall complexity of the synchronous request model.

The Gateway is still responsible for maintaining Promise/Future state, request correlation, timeout tracking, and watchdog processing for every in-flight operation.

These broader concerns are addressed in Firebolt CPP 2.0, where timeout handling and request lifecycle management can potentially be simplified through asynchronous APIs.

Clone this wiki locally