fix(tunnel): implement ping pong #3491

MasterPtato · 2025-11-19T21:20:09Z

No description provided.

vercel · 2025-11-19T21:20:15Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Preview	Comments	Updated (UTC)
rivetkit-serverless	Ready	Preview	Comment	Nov 20, 2025 4:53am

3 Skipped Deployments

Project	Deployment	Preview	Updated (UTC)
rivet-cloud	Ignored		Nov 20, 2025 4:53am
rivet-inspector	Ignored	Preview	Nov 20, 2025 4:53am
rivet-site	Ignored	Preview	Nov 20, 2025 4:53am

MasterPtato · 2025-11-19T21:20:23Z

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

fix(tunnel): implement ping pong #3491 : 2 dependent PRs (#3475 , #3494 ) 👈 (View in Graphite)
chore(pegboard-gateway): add new message id format & add deprecated tunnel ack #3492
chore(pegboard-gateway): remove TunnelAck #3488
chore(engine-runner): remove echo close on ToClientWebSocketClose #3487
chore(rivetkit): implement new hibernating ws protocol #3464 : 1 other dependent PR (#3452 )
fix(actor-kv): fix listing keys #3484
fix(guard): handle actor stopped during ws/req #3481
chore: rename retry -> hibernate in runner protocol #3473
fix(udb): update backoff algo #3469
chore: write/update docs #3468
chore: add debug scripts #3467
main

How to use the Graphite Merge Queue

Add the label merge-queue to this PR to add it to the merge queue.

You must have a Graphite account in order to use the merge queue. Sign up using this link.

_{An organization admin has enabled the Graphite Merge Queue in this repository.} _{Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.}

This stack of pull requests is managed by Graphite. Learn more about stacking.

claude · 2025-11-19T21:22:33Z

Pull Request Review: Implement Ping/Pong for Tunnel

Summary

This PR implements a ping/pong mechanism for the pegboard tunnel to monitor connection health and measure RTT between gateway and runner. Overall, this is a solid implementation that addresses tunnel health monitoring.

✅ Strengths

Good Architecture: The refactoring into separate task modules (ping_task.rs, tunnel_to_ws_task.rs, ws_to_tunnel_task.rs) significantly improves code organization and readability.
Proper Protocol Design: The ping/pong protocol is well-designed:
- Gateway sends ToRunnerPing with timestamp
- Runner responds with ToGatewayPong echoing the timestamp
- Gateway calculates RTT and tracks last_pong for timeout detection
Metrics Integration: Good use of OpenTelemetry metrics to track RTT (rivet_gateway_tunnel_ping_duration) for observability.
Timeout Handling: The TUNNEL_PING_TIMEOUT (30s) with UPDATE_PING_INTERVAL (3s) provides reasonable connection monitoring with multiple ping attempts before timeout.

🔍 Issues & Suggestions

1. Potential Race Condition in Ping Timeout Check ⚠️

Location: engine/packages/pegboard-gateway/src/shared_state.rs:237-240

if now.saturating_sub(req.last_pong) > TUNNEL_PING_TIMEOUT {
    tracing::warn!("tunnel timeout");
    return Err(WebSocketServiceTimeout.build());
}

Issue: The ping task checks for timeout BEFORE sending the ping, but last_pong is only updated when a pong is received. On the very first ping after connection establishment, last_pong is initialized to util::timestamp::now() (shared_state.rs:119), which is fine. However, there's a subtle timing issue:

If the runner is slow to respond for exactly TUNNEL_PING_TIMEOUT, the next ping check will immediately fail
The timeout check happens before sending a new ping, so we don't give the current ping a chance to respond

Suggestion: Consider checking the timeout AFTER a failed ping response or implementing a separate "last ping sent" timestamp to track when we last attempted communication.

2. Inconsistent RTT Calculation Between Gateway and Runner

Location: engine/packages/pegboard-runner/src/ws_to_tunnel_task.rs:94-109

The runner calculates RTT as:

let delta = now.saturating_sub(ping.ts);
let rtt = delta * 2;  // Assuming symmetric delta

But the gateway calculates it as:

let rtt = now.saturating_sub(pong.ts);  // shared_state.rs:285

Issue: These measure different things:

Runner: Measures one-way latency × 2 (from gateway → runner)
Gateway: Measures actual round-trip time (gateway → runner → gateway)

The gateway's calculation is the true RTT, while the runner's is an estimate assuming symmetric latency. This might cause confusion when debugging or monitoring.

Suggestion: Add a comment explaining this difference, or consider having the runner echo back both timestamps so the gateway can calculate true RTT on both sides.

3. Missing Error Context in Pong Handler

Location: engine/packages/pegboard-gateway/src/shared_state.rs:271-282

Ok(protocol::ToGateway::ToGatewayPong(pong)) => {
    let Some(mut in_flight) = self.in_flight_requests.get_async(&pong.request_id).await
    else {
        tracing::debug!(
            request_id=?Uuid::from_bytes(pong.request_id),
            "in flight has already been disconnected, dropping ping"
        );
        continue;
    };

Minor Issue: This silently drops pongs for disconnected requests. While this is probably fine, it could indicate a timing issue or bug if it happens frequently.

Suggestion: Consider adding a metric to track dropped pongs to help identify potential issues in production.

4. Ping Task Continues After Error

Location: engine/packages/pegboard-gateway/src/ping_task.rs:21

shared_state.send_and_check_ping(request_id).await?;

The ping task will exit immediately if send_and_check_ping returns an error (including timeout). This is correct behavior, but ensure that the error propagates properly to trigger connection cleanup.

Verification Needed: Confirm that when the ping task returns an error, it properly triggers cleanup of the other tasks and the connection. From the code in lib.rs:446-467, it looks like this is handled correctly via the abort channels.

5. Clock Skew Handling

Location: engine/packages/pegboard-runner/src/ws_to_tunnel_task.rs:94-106

if ping.ts <= now {
    let delta_ms = now.saturating_sub(ping.ts);
    delta_ms.min(u32::MAX as i64) as u32
} else {
    tracing::warn!("ping timestamp is in the future, possibly due to clock skew");
    0
}

Good: The code handles clock skew gracefully by defaulting to 0 RTT when timestamps are in the future.

Suggestion: Consider setting a flag or metric when clock skew is detected, as this could indicate a systemic issue that needs attention.

6. Metrics Bucket Configuration

Location: engine/packages/pegboard-gateway/src/metrics.rs:10-12

pub static ref TUNNEL_PING_DURATION: Histogram<f64> = METER.f64_histogram("rivet_gateway_tunnel_ping_duration")
    .with_description("RTT of messages from gateway to pegboard.")
    .with_boundaries(BUCKETS.to_vec())

Question: What are the bucket boundaries defined in BUCKETS? For RTT measurements, typical values might range from 1ms to several seconds. Ensure the buckets are appropriate for the expected RTT range.

7. Missing Tracing Instrumentation

Location: engine/packages/pegboard-gateway/src/ping_task.rs

The ping task function is not instrumented with #[tracing::instrument], unlike other tasks.

Suggestion: Add instrumentation for consistency:

#[tracing::instrument(skip_all, fields(request_id=?Uuid::from_bytes(request_id)))]
pub async fn task(...)

🧪 Testing Concerns

No Test Coverage: This PR doesn't include any tests for the new ping/pong functionality. Consider adding:

Unit tests for timeout calculations
Integration tests for ping/pong message flow
Tests for edge cases like clock skew, dropped pongs, and timeout scenarios

Per CLAUDE.md guidelines, when running tests: "pipe the test to a file in /tmp/ then grep it in a second step."

🔒 Security

No security concerns identified. The ping/pong mechanism doesn't introduce new attack vectors.

⚡ Performance

Minor Concern: The ping task runs every 3 seconds for every active WebSocket connection. For high connection counts, this could generate significant background traffic.

Metrics: The ping interval and timeout values seem reasonable:

UPDATE_PING_INTERVAL: 3 seconds
TUNNEL_PING_TIMEOUT: 30 seconds (allows ~10 ping attempts before timeout)

Consider monitoring the overhead in production with many concurrent connections.

📝 Code Quality

Good:

Follows existing error handling patterns using RivetError
Proper use of structured logging with tracing
Clean separation of concerns with dedicated task modules
Follows workspace dependency conventions

Minor Issues:

The ordering of serde and serde_json in Cargo.toml was changed (alphabetization) - this is fine but inconsistent with the note to "check existing imports before adding new ones"
Some #[allow(dead_code)] attributes added to struct fields in shared_state.rs:454-461 - consider if these fields are actually needed

✨ Suggestions for Future Work

Add configurable ping interval and timeout via environment variables
Consider adaptive ping intervals based on connection stability
Add more detailed metrics (e.g., histogram of pong response times, dropped pong counter)
Consider implementing exponential backoff for ping attempts before timeout

Verdict

This is a solid implementation with good architectural decisions. The main concerns are:

The potential race condition in timeout checking (minor)
Lack of test coverage (should be addressed)
Missing observability for edge cases (dropped pongs, clock skew)

Recommendation: ✅ Approve with suggestions - The code is production-ready, but would benefit from additional tests and minor improvements to error handling/observability.

Review conducted following repository conventions from CLAUDE.md

claude · 2025-11-19T21:46:24Z

Code Review: Ping-Pong Implementation for Tunnel Protocol

This PR implements a ping-pong mechanism to detect and handle tunnel timeouts. Overall, the implementation is solid with good separation of concerns.

Strengths

Good Architecture: The code properly separates ping logic into dedicated task modules on both gateway and runner sides
Metrics Integration: Added proper OpenTelemetry metrics (TUNNEL_PING_DURATION) to track RTT
Clean Refactoring: The main websocket handler logic was successfully extracted into separate task modules, improving code organization
Proper Tracing: Added tracing::instrument attributes to key functions for better observability
Structured Logging: Follows the repo's logging conventions with structured fields

Potential Issues

1. Ping Timeout Timing Mismatch (Critical)

The ping is sent every 3 seconds (UPDATE_PING_INTERVAL), but the timeout is 30 seconds (TUNNEL_PING_TIMEOUT). This means it would take 10 missed pings before timing out. Consider:

Is this intentional for network resilience?
Should there be a more aggressive timeout (e.g., 3-4x the ping interval = 9-12 seconds)?

2. Initial Ping Timing

In pegboard-gateway/src/ping_task.rs:13-21, the first ping is sent after waiting for UPDATE_PING_INTERVAL. This delays initial connection validation by 3 seconds. Consider sending the first ping immediately before entering the loop.

3. RTT Calculation Asymmetry

In pegboard-runner/src/ws_to_tunnel_task.rs:109-110, the code assumes symmetric network latency (rtt = delta * 2). While common, this may be inaccurate in asymmetric network conditions. The comment is good, but this limitation should be noted.

4. Error Handling in Ping Task

If send_and_check_ping returns an error, the ping task terminates without specific logging about why. Consider adding error context before returning.

5. Potential Race Condition

In shared_state.rs:266-267, there's a potential race between the ping task reading last_pong and the pong handler updating it. However, this is likely acceptable since the check is conservative (only leads to false negatives).

Code Quality Observations

1. Duplicate Abort Logic

The abort signal handling is duplicated across three join arms (lib.rs:435-473). Consider extracting to a helper function.

2. Magic Number in Metrics

The 0.001 constant in shared_state.rs:270 converts milliseconds to seconds. Consider adding a named constant for clarity.

Test Coverage

Missing: No tests were added for the new ping-pong functionality. Consider adding:

Unit tests for timeout detection logic
Integration tests for ping-pong flow
Tests for edge cases (clock skew, missed pongs, timeout scenarios)

Security & Performance

Security: No concerns identified. Proper timestamp validation and clock skew handling.

Performance: 3-second ping intervals are reasonable. Minimal metrics overhead. Good use of concurrent tasks.

Summary

Well-structured implementation that adds necessary timeout detection. Main concerns:

Action Required: Verify ping timeout values are intentional
Recommended: Send first ping immediately, add tests
Nice-to-Have: Improve error logging, reduce code duplication

The code follows repository conventions well and demonstrates good async Rust patterns.

claude · 2025-11-19T21:46:34Z

Pull Request Review: Implement Ping-Pong for Tunnel

Overall Assessment

This is a well-structured implementation that adds a critical ping-pong mechanism to detect tunnel connection failures. The refactoring to separate tasks improves code organization and maintainability.

Strengths

Good Code Organization: The refactoring into separate task modules significantly improves code readability and maintainability.
Proper Timeout Handling: The implementation includes timeout detection with TUNNEL_PING_TIMEOUT (30s) to identify stale connections.
Metrics Integration: The addition of TUNNEL_PING_DURATION metric provides valuable observability for tunnel health monitoring.
Structured Logging: Good use of structured logging with tracing attributes throughout the code.

Issues and Concerns

1. Potential Race Condition in Ping Timeout Check (High Priority)

Location: engine/packages/pegboard-gateway/src/shared_state.rs:221

The ping task sends a ping every 3 seconds and immediately checks if the last pong was more than 30 seconds ago. However, the ping check happens AFTER sending the ping message which could lead to race conditions with message reordering or delivery delays.

Recommendation: Consider checking the timeout BEFORE sending the ping, not after. This makes the logic clearer.

2. Ping Task Doesn't Send First Ping Immediately (Medium Priority)

Location: engine/packages/pegboard-gateway/src/ping_task.rs:14-21

The ping task sleeps for 3 seconds before sending the first ping. If the connection is already dead, it takes 30 seconds to detect rather than 3 seconds.

Recommendation: Consider sending the first ping immediately before entering the sleep loop.

3. RTT Calculation Assumes Symmetric Latency (Low Priority)

Location: engine/packages/pegboard-runner/src/ws_to_tunnel_task.rs:109-110

The code multiplies delta by 2 assuming symmetric latency, which may not hold in real-world networks.

Recommendation: Document this assumption more prominently or use one-way latency directly.

4. Missing Error Context in Metrics (Low Priority)

Location: engine/packages/pegboard-gateway/src/shared_state.rs:270

The metric is recorded with no attributes, making it difficult to debug issues for specific gateways or runners.

Recommendation: Consider adding attributes for better observability.

5. Protocol Version Compatibility (Medium Priority)

The protocol change from ToRunnerKeepAlive to ToRunnerPing breaks backward compatibility.

Recommendation: Document the minimum required protocol version for this feature and consider adding compatibility checks.

6. Inconsistent Import Ordering (Style)

Location: engine/packages/pegboard-gateway/src/shared_state.rs:11

Per CLAUDE.md conventions, imports should be at the top of the file.

Test Coverage

Missing test scenarios:

Ping timeout detection (connection goes stale for >30s)
Ping task abort when other tasks fail
Concurrent ping/pong with high message volume
Clock skew handling
Metrics recording under various RTT conditions

Recommendation: Add integration tests to cover these scenarios.

Security Considerations

No security concerns identified. The ping-pong mechanism doesn't introduce any new attack surfaces.

Performance Considerations

Ping Frequency: 3-second intervals seem reasonable for tunnel health checks
Timeout Duration: 30 seconds is conservative but appropriate for production
Metrics Overhead: Minimal

Summary

This is a solid implementation that adds necessary tunnel health monitoring. The main concerns are:

High Priority: Review the ping timeout check logic to avoid race conditions
Medium Priority: Send first ping immediately to detect dead connections faster
Medium Priority: Document protocol version compatibility requirements

The code is well-structured and follows the project's conventions. With the above issues addressed, this will be a valuable addition to the codebase.

Great work overall!

claude · 2025-11-19T21:55:51Z

Code Review: Ping/Pong Implementation for Tunnels

Summary

This PR implements a ping/pong mechanism to monitor tunnel health between the gateway and pegboard runners. The implementation is well-structured with good separation of concerns through dedicated task modules. Overall, this is solid work with good architectural decisions.

✅ Strengths

Good code organization - Splitting the monolithic lib.rs into separate task modules (ping_task.rs, tunnel_to_ws_task.rs, ws_to_tunnel_task.rs) significantly improves readability and maintainability.
Proper metrics tracking - The RTT measurement and OpenTelemetry integration in metrics.rs provides good observability.
Consistent protocol design - The ping/pong protocol additions follow the existing patterns with ToRunnerPing and ToGatewayPong types.
Task coordination - The abort mechanism with watch channels properly coordinates the three concurrent tasks.

🐛 Issues & Concerns

High Priority

1. Potential Race Condition in last_pong Update (shared_state.rs:119, shared_state.rs:267)

In start_in_flight_request(), you initialize last_pong when creating a new request, but when re-using an existing Entry::Occupied, you don't reset last_pong. This means if a request is being restarted/reconnected, it could immediately timeout if the old last_pong timestamp is too old.

Entry::Occupied(mut entry) => {
    entry.receiver_subject = receiver_subject;
    entry.msg_tx = msg_tx;
    entry.drop_tx = drop_tx;
    entry.opened = false;
    entry.message_index = 0;
    // Missing: entry.last_pong = util::timestamp::now();

Recommendation: Reset last_pong in the Occupied branch as well.

2. Ping Task Error Handling Could Cause Silent Failures (ping_task.rs:21)

The ping task calls send_and_check_ping() which can return an error (including timeout errors). If this happens, the error propagates and terminates the ping task, which then terminates all other tasks. However, this might be too aggressive - a transient network issue could kill an otherwise healthy connection.

shared_state.send_and_check_ping(request_id).await?;

Recommendation: Consider whether timeout detection should be handled more gracefully, perhaps with retry logic or exponential backoff before terminating the connection.

3. Missing RTT Tracking on Runner Side (tunnel_to_ws_task.rs:57-80)

The runner responds to pings but doesn't track its own RTT or connection health. This means the runner has no way to detect if the gateway has gone away or if the connection is degraded.

Recommendation: Consider implementing symmetric ping/pong where both sides track RTT and can detect connection issues.

Medium Priority

4. Hardcoded Timeout Constants Need Documentation

const TUNNEL_PING_TIMEOUT: i64 = util::duration::seconds(30);
const UPDATE_PING_INTERVAL: Duration = Duration::from_secs(3);

These constants define critical behavior but lack documentation explaining:

Why 30 seconds was chosen
What the expected RTT range is
How this relates to UPDATE_PING_INTERVAL (3 seconds → ~10 missed pings before timeout)

Recommendation: Add doc comments explaining the rationale and relationship between these values.

5. Inconsistent Timeout Type (i64 vs Duration) (shared_state.rs:19)

TUNNEL_PING_TIMEOUT uses i64 (milliseconds), while UPDATE_PING_INTERVAL uses Duration. This inconsistency makes it harder to understand the relationship between them.

Recommendation: Use Duration consistently and convert to millis only when needed for timestamp comparisons.

6. Metrics Missing Attributes (metrics.rs:9)

The comment says "Has no expected attributes" but adding attributes like gateway_id or aggregated connection state would make debugging much easier.

Recommendation: Consider adding optional attributes to correlate metrics with specific gateways or connection states.

7. GC Debug Messages Use #[allow(dead_code)] (shared_state.rs:444-447)

WebSocketMessageNotAcked {
    #[allow(dead_code)]
    first_msg_index: u16,
    #[allow(dead_code)]
    last_msg_index: u16,
},

These fields are only used in debug output but marked as dead code. This suggests the debug formatting might not be as useful as intended.

Recommendation: Either use these in structured logging or remove them if not providing value.

🔍 Minor Issues

8. Code Style: Import Ordering (lib.rs:12-13)

use rivet_guard_core::{
    WebSocketHandle,
    custom_serve::{CustomServeTrait, HibernationResult},

According to CLAUDE.md, imports should follow a consistent pattern. The WebSocketHandle import should be grouped with other imports alphabetically.

9. Commented Out Dead Code (shared_state.rs:543-545)

// fn wrapping_lt(a: u16, b: u16) -> bool {
//     b.wrapping_sub(a) < u16::MAX / 2
// }

Per CLAUDE.md: "Avoid backwards-compatibility hacks like... adding // removed comments for removed code, etc. If something is unused, delete it completely."

Recommendation: Remove the commented code.

10. Pattern Match Result Type Collision (tunnel_to_ws_task.rs:48)

let msg = match versioned::ToRunner::deserialize_with_embedded_version(&ups_msg.payload) {
    Result::Ok(x) => x,

Using Result::Ok instead of Ok suggests a potential type collision. This works but is unusual.

Recommendation: Verify if this is necessary or if standard Ok(x) would work.

✨ Suggestions

Add integration tests - This ping/pong mechanism is critical for connection health. Consider adding tests that simulate:
- Normal ping/pong exchange
- Missed pings leading to timeout
- Recovery from transient failures
Consider backpressure - If the runner is overloaded and slow to respond to pings, should the gateway handle this differently than a dead connection?
Logging improvements - Consider adding structured logging for ping RTT at regular intervals (not just in metrics) to aid debugging.

📊 Test Coverage

The PR doesn't include tests for the ping/pong mechanism. Given this is a critical reliability feature, I'd strongly recommend adding:

Unit tests for timeout detection logic
Integration tests for the full ping/pong cycle
Tests for edge cases (connection interruption, delayed pongs, etc.)

Security & Performance

✅ No security concerns identified
✅ Performance looks good - Ping interval of 3 seconds is reasonable and shouldn't cause overhead

Conclusion

This is a solid implementation that significantly improves tunnel reliability. The main concerns are:

The race condition in last_pong initialization (should be fixed)
Error handling strategy in the ping task (consider retry logic)
Missing test coverage

Once the last_pong race condition is addressed, this would be ready to merge. Great work on the refactoring and separation of concerns!

claude · 2025-11-19T21:57:50Z

PR Review: fix(tunnel): implement ping pong

Summary

This PR implements a ping/pong mechanism for tunnel connections between the gateway and pegboard runner to detect connection failures and measure round-trip time (RTT). The implementation adds active health monitoring to replace the previous passive keep-alive approach.

✅ Strengths

1. Good Code Organization

Excellent refactoring that extracts tunnel_to_ws_task, ws_to_tunnel_task, and ping_task into separate modules, improving code readability and maintainability
Clear separation of concerns between the three concurrent tasks

2. Proper Concurrency Handling

Well-structured use of tokio::select! with appropriate abort channels
Correct handling of task lifecycle with LifecycleResult::Aborted pattern
Good use of watch channels for coordinating task shutdown

3. Observability

Added metrics for tunnel ping duration (TUNNEL_PING_DURATION) which is valuable for monitoring
Comprehensive tracing with structured logging using request IDs

4. Protocol Design

Clean protocol extension with ToRunnerPing and ToGatewayPong messages
Preserves timestamp from ping in pong for accurate RTT calculation

⚠️ Issues & Concerns

1. Potential Race Condition in Ping Timeout (Medium Severity)

In shared_state.rs:210-221, the ping task sends a ping but checks last_pong before sending:

let now = util::timestamp::now();

// Verify ping timeout
if now.saturating_sub(req.last_pong) > TUNNEL_PING_TIMEOUT {
    tracing::warn!("tunnel timeout");
    return Err(WebSocketServiceTimeout.build());
}

// Send message
let message = protocol::ToRunner::ToRunnerPing(...)

Issue: If the connection has been dead for exactly 30 seconds, this will send one final ping before timing out. This is wasteful and could mask the actual timeout condition.

Recommendation: Check the timeout after attempting to send, or skip sending if we're about to timeout.

2. Missing last_pong Initialization Validation (Low-Medium Severity)

In shared_state.rs:119, new in-flight requests initialize last_pong: util::timestamp::now(). However, there's no validation that the first pong is received within a reasonable timeframe.

Scenario: A connection could be established, but if the first ping/pong exchange fails, it would take 30 seconds to detect the failure (from TUNNEL_PING_TIMEOUT).

Recommendation: Consider adding an initial handshake validation or reducing the effective timeout for the first ping/pong exchange.

3. Integer Overflow in Timestamp Arithmetic (Low Severity)

shared_state.rs:260 uses saturating_sub correctly:

let rtt = now.saturating_sub(pong.ts);

However, this could theoretically return 0 if clocks are skewed. While unlikely in practice with monotonic timestamps, consider:

Adding validation that rtt >= 0 and logging warnings for negative RTT
Using a more robust time source or documenting clock sync requirements

4. Inconsistent Error Handling in update_runner_ping (Low Severity)

In pegboard-runner/src/ping_task.rs:33-35, if the workflow doesn't exist, the code logs an error but returns Ok(()):

else {
    tracing::error!(?conn.runner_id, "workflow does not exist");
    return Ok(());
};

Question: Should this be a fatal error that stops the ping task, or is continuing silently the intended behavior? The current approach could hide genuine issues.

Recommendation: Consider returning an error or at least using tracing::warn! instead of tracing::error! if this is expected behavior.

5. Task Coordination Logic Could Be Simplified (Code Quality)

In lib.rs:426-448, the abort logic is duplicated with multiple abort channel clones:

let tunnel_to_ws_abort_tx2 = tunnel_to_ws_abort_tx.clone();
let ws_to_tunnel_abort_tx2 = ws_to_tunnel_abort_tx.clone();
let ping_abort_tx2 = ping_abort_tx.clone();

Recommendation: Consider using a shared abort token or a single broadcast channel to simplify the coordination logic. This would make the shutdown logic more maintainable.

6. Missing Tracing Instrumentation (Minor)

The new task functions in tunnel_to_ws_task.rs and ws_to_tunnel_task.rs lack #[tracing::instrument] attributes that are present elsewhere in the codebase (e.g., ping_task.rs:9).

Recommendation: Add consistent instrumentation:

#[tracing::instrument(skip_all, fields(request_id=?tunnel_id::request_id_to_string(&request_id)))]
pub async fn task(...)

7. Import Organization (Minor - Style)

guard-core/src/custom_serve.rs:6 shows an import being moved:

+use pegboard::tunnel::id::RequestId;
 
 use crate::WebSocketHandle;
 use crate::proxy_service::ResponseBody;
 use crate::request_context::RequestContext;
 
-use pegboard::tunnel::id::RequestId;

According to CLAUDE.md: "Always add imports at the top of the file inside of inline within the function." This change is correct, but ensure consistency across all modified files.

🧪 Testing Concerns

1. Missing Test Coverage

There are no new test files added for:

Ping/pong message exchange
Timeout behavior when pongs stop arriving
RTT calculation accuracy
Task coordination and abort scenarios

Recommendation: Add integration tests covering:

- Happy path: ping sent, pong received, RTT recorded
- Timeout path: no pong received within TUNNEL_PING_TIMEOUT
- Task cleanup: all tasks properly abort on connection close

2. No Validation of Metrics

The new TUNNEL_PING_DURATION metric is recorded but there's no test validating:

Metric is actually recorded
RTT values are reasonable (not negative, not absurdly large)

🔒 Security Considerations

1. No Major Security Issues Detected

The ping/pong implementation doesn't introduce obvious security vulnerabilities. However:

Timestamp Trust: The gateway trusts the timestamp from the runner's pong. In a malicious scenario, a runner could send fake timestamps to manipulate RTT metrics. This is likely acceptable given the trust model.
DoS via Ping Flooding: The UPDATE_PING_INTERVAL (3 seconds) is reasonable and won't cause DoS, but ensure rate limiting exists at the protocol level if not already present.

📊 Performance Considerations

1. Reasonable Ping Interval

UPDATE_PING_INTERVAL = 3 seconds is a good balance between responsiveness and overhead.

2. Metrics Collection

The histogram collection (metrics::TUNNEL_PING_DURATION) has minimal overhead, using pre-defined buckets.

3. Lock Contention

shared_state.rs uses SCC's HashMap which is lock-free, so the additional last_pong field won't introduce contention.

📝 Recommendations Summary

High Priority:

Fix the race condition in ping timeout checking

Medium Priority:
2. Add validation for initial ping/pong handshake
3. Add integration tests for ping/pong behavior
4. Clarify error handling in update_runner_ping

Low Priority:
5. Add tracing instrumentation to task functions
6. Consider simplifying task abort coordination
7. Add validation for RTT calculation edge cases

✅ Conclusion

This is a solid implementation of ping/pong for tunnel health monitoring. The code is well-structured and follows good Rust practices. The main concerns are around edge case handling and test coverage. With the recommended fixes, this will be a robust addition to the codebase.

Recommended Action: Address the high/medium priority items before merging, especially the ping timeout race condition and test coverage.

Great work on the refactoring! The separation into task modules makes the code much more maintainable. 🚀

claude · 2025-11-19T21:57:54Z

Code Review: Ping-Pong Implementation for Tunnel

Summary

This PR implements a WebSocket ping-pong mechanism between the pegboard gateway and runner to detect stale connections and measure round-trip time (RTT). The implementation adds proper connection health monitoring to the tunnel protocol.

✅ Strengths

Good Architecture: The refactoring of monolithic async tasks into separate modules (ping_task.rs, tunnel_to_ws_task.rs, ws_to_tunnel_task.rs) significantly improves code organization and maintainability.
Proper Protocol Evolution: The protocol changes from ToRunnerKeepAlive to ToRunnerPing/ToGatewayPong with timestamps enable RTT measurement, which is more useful than a simple keepalive.
Metrics Integration: Adding TUNNEL_PING_DURATION histogram provides valuable observability into tunnel health (gateway/src/metrics.rs:10-13).
Consistent Timeout Handling: The TUNNEL_PING_TIMEOUT of 30 seconds with UPDATE_PING_INTERVAL of 3 seconds provides a reasonable 10x safety margin.
Good Error Recovery: The ping task refactoring in runner (runner/src/ping_task.rs:27-66) properly extracts update_runner_ping into a separate function with better error handling.

🔍 Issues & Concerns

1. Potential Ordering Issue in Abort Logic ⚠️

Location: engine/packages/pegboard-gateway/src/lib.rs:435-436, 450-451, 465-466

The abort signals are sent using .send() which can fail, but errors are ignored:

let _ = ping_abort_tx.send(());
let _ = ws_to_tunnel_abort_tx.send(());

Issue: If these sends fail (e.g., receiver already dropped), the tasks won't be properly aborted. This could lead to resource leaks or zombie tasks.

Recommendation:

Consider logging when abort signals fail to send
Or verify that ignoring failures is intentional (e.g., receiver already completed)

2. Timestamp Overflow Potential ⚠️

Location: engine/packages/pegboard-gateway/src/shared_state.rs:218

if now.saturating_sub(req.last_pong) > TUNNEL_PING_TIMEOUT {

The comparison uses saturating_sub, which is good, but TUNNEL_PING_TIMEOUT is defined as i64 milliseconds (30,000ms). If now or req.last_pong are corrupted or uninitialized, this could fail silently.

Recommendation: Add validation that req.last_pong is reasonable (e.g., not zero, not far in the future).

3. Race Condition in Ping Update ⚠️

Location: engine/packages/pegboard-gateway/src/shared_state.rs:265-267

let now = util::timestamp::now();
in_flight.last_pong = now;

The last_pong timestamp is updated in the pong handler, but there's no synchronization with the ping checker. If a ping timeout check happens between receiving the pong message and updating last_pong, it could trigger a false timeout.

Recommendation: Consider atomic operations or ensure the update happens before any timeout checks can observe the old value.

4. Missing Error Context

Location: engine/packages/pegboard-gateway/src/shared_state.rs:215

.context("request not in flight")?;

This error message doesn't include the request_id, making debugging difficult.

Recommendation:

.with_context(|| format!("request not in flight: {:?}", tunnel_id::request_id_to_string(&request_id)))?;

5. Incomplete Close Frame Handling

Location: engine/packages/pegboard-gateway/src/lib.rs:476-486

The logic to determine the final lifecycle_res from three tasks prefers the first non-aborted result:

(Ok(res), Ok(LifecycleResult::Aborted), _) => Ok(res),
(Ok(LifecycleResult::Aborted), Ok(res), _) => Ok(res),
// Unlikely case
(res, _, _) => res,

Issue: If the ping task times out (returns error) while the other two tasks complete successfully, the error takes precedence. This might close the connection even if data transfer was successful.

Recommendation: Consider whether ping timeout should take precedence over successful close from other tasks, or if it should only abort if both other tasks are still running.

6. Protocol Breaking Change ⚠️

Location: engine/sdks/schemas/runner-protocol/v3.bare

The protocol changed from:

type ToRunnerKeepAlive void

to:

type ToRunnerPing struct {
    gatewayId: GatewayId
    requestId: RequestId
    ts: i64
}

Issue: This is a breaking protocol change. Old runners won't understand the new ping format.

Question: Is there a protocol version negotiation? Should this be v4 instead of modifying v3?

🎯 Security Considerations

DoS via Timestamp Manipulation: If an attacker could manipulate timestamps, they could cause false timeouts. The implementation uses util::timestamp::now() which should be monotonic, but verify it's using a trusted clock source.
Resource Exhaustion: The ping task runs every 3 seconds for each connection. For many connections, this could generate significant load. Consider:
- Rate limiting ping operations
- Batching ping updates
- Monitoring ping task count

📊 Performance Considerations

Metrics Overhead: Each ping generates a histogram sample. For high-connection-count scenarios, verify this doesn't impact performance (gateway/src/metrics.rs:10-13).
GC Impact: The TUNNEL_PING_TIMEOUT affects GC behavior in shared_state.rs:482. Document the relationship between ping timeout and GC cycles.
Lock Contention: in_flight_requests is accessed on every ping/pong. For high throughput, consider if this becomes a bottleneck (shared_state.rs:216).

🧪 Test Coverage

Critical Issue: No tests were added for the new ping-pong functionality.

Recommendations:

Add unit tests for send_and_check_ping timeout logic
Add integration tests for ping-pong flow
Test race conditions (concurrent ping timeout and pong receipt)
Test protocol serialization/deserialization for new message types
Test abort signal handling in all three tasks

📝 Documentation

Missing:

No documentation on the ping-pong protocol behavior
No comments explaining timeout values (why 3s and 30s?)
The relationship between ping timeout and hibernation needs documentation

Recommendation: Add module-level docs explaining:

The purpose of ping-pong (connection health, RTT measurement)
Timeout values and their rationale
How ping interacts with hibernation

🔧 Code Style

Good:

Follows Rust conventions
Proper use of structured logging with tracing
Good error propagation with anyhow

Minor Issues:

Unused #[allow(dead_code)] attributes in shared_state.rs:444-445 suggest incomplete implementation or unnecessary fields
Comment in runner protocol (v3.bare:18) says "TODO: Rename to update_ts" - should this be addressed?

💡 Suggestions

Add Exponential Backoff: If ping fails transiently, exponential backoff could reduce unnecessary connection drops.
Configurable Timeouts: Make UPDATE_PING_INTERVAL and TUNNEL_PING_TIMEOUT configurable rather than hardcoded constants.
Ping Jitter: Add small random jitter to ping intervals to avoid thundering herd with many connections.
Connection State Machine: Consider formalizing the connection states (CONNECTED, PINGING, TIMEOUT, etc.) for better debugging.

🎬 Conclusion

This is a solid implementation that adds necessary connection health monitoring. The code refactoring improves maintainability significantly. However, there are several important issues to address before merging:

Must Fix:

Clarify protocol version compatibility (breaking change concern)
Add tests for ping-pong functionality
Review abort signal error handling

Should Fix:

Add error context with request IDs
Document timeout values and behavior
Consider race condition in last_pong update

Nice to Have:

Add configurability for timeouts
Add ping jitter
Improve metrics documentation

Overall: Approve with changes - The implementation is sound but needs test coverage and protocol version clarification before merging.

claude · 2025-11-19T23:28:39Z

Code Review: Implement Ping/Pong for Tunnel Connection

Summary

This PR implements a ping/pong mechanism to detect and handle stale tunnel connections between the pegboard gateway and runner. The implementation adds a periodic ping task that sends ping messages every 3 seconds and tracks RTT (Round Trip Time) metrics.

✅ Positive Observations

Clean Code Organization: The refactoring into separate task modules (ping_task.rs, tunnel_to_ws_task.rs, ws_to_tunnel_task.rs) improves code maintainability and separation of concerns.
Proper Error Handling: The implementation correctly uses Result types and handles errors appropriately throughout the new code.
Good Observability: The addition of RTT metrics (TUNNEL_PING_DURATION) provides valuable insights into connection health.
Protocol Evolution: The protocol changes are well-structured, replacing the deprecated ToRunnerKeepAlive with proper ToRunnerPing/ToGatewayPong messages.
Task Lifecycle Management: Proper use of watch channels for task coordination and abort signaling.

🔍 Issues and Concerns

1. Critical: Ping Timing Issue ⚠️

Location: engine/packages/pegboard-gateway/src/ping_task.rs:13-22

The ping task sleeps BEFORE checking the ping, which means:

The first ping happens after 3 seconds instead of immediately
If a connection starts unhealthy, it won't be detected for 3 seconds

loop {
    tokio::select! {
        _ = tokio::time::sleep(UPDATE_PING_INTERVAL) => {}
        _ = ping_abort_rx.changed() => {
            return Ok(LifecycleResult::Aborted);
        }
    }
    shared_state.send_and_check_ping(request_id).await?;
}

Recommendation: Send the first ping immediately, then sleep:

loop {
    shared_state.send_and_check_ping(request_id).await?;
    
    tokio::select! {
        _ = tokio::time::sleep(UPDATE_PING_INTERVAL) => {}
        _ = ping_abort_rx.changed() => {
            return Ok(LifecycleResult::Aborted);
        }
    }
}

2. Ping Timeout Mismatch ⚠️

Location: engine/packages/pegboard-gateway/src/shared_state.rs:18 and src/lib.rs:40

const TUNNEL_PING_TIMEOUT: i64 = util::duration::seconds(30);  // 30 seconds
const UPDATE_PING_INTERVAL: Duration = Duration::from_secs(3); // 3 seconds

The timeout is 10x the ping interval, which is reasonable, but consider:

With the current sleep-first implementation, you could have up to 33 seconds before timeout detection
A 30-second timeout might be too long for time-sensitive operations

Recommendation: Document the timeout-to-interval ratio reasoning or consider reducing the timeout to 15 seconds for faster failure detection.

3. Import Organization (Code Style)

Location: Multiple files (e.g., engine/packages/guard-core/src/custom_serve.rs)

The PR fixes import ordering in some places but introduces it in others:

use tokio_tungstenite::tungstenite::protocol::frame::CloseFrame;
use pegboard::tunnel::id::RequestId;  // Moved to top

This is good, but ensure consistency across all new files. Per CLAUDE.md:

Always add imports at the top of the file inside of inline within the function.

4. Missing Documentation

Location: engine/packages/pegboard-gateway/src/shared_state.rs:210-244

The new send_and_check_ping method lacks documentation explaining:

What the timeout verification does
Why ping and timeout check are combined in one method
The expected behavior when timeout is exceeded

Recommendation: Add doc comments:

/// Sends a ping message to the runner and verifies the connection hasn't timed out.
///
/// This method performs two operations:
/// 1. Checks if the last pong was received within TUNNEL_PING_TIMEOUT
/// 2. Sends a new ping message with the current timestamp
///
/// # Errors
/// Returns `WebSocketServiceTimeout` if no pong received within timeout window
#[tracing::instrument(skip_all, fields(request_id=?tunnel_id::request_id_to_string(&request_id)))]
pub async fn send_and_check_ping(&self, request_id: RequestId) -> Result<()> {

5. Potential Race Condition

Location: engine/packages/pegboard-gateway/src/shared_state.rs:210-219

pub async fn send_and_check_ping(&self, request_id: RequestId) -> Result<()> {
    let req = self.in_flight_requests.get_async(&request_id).await
        .context("request not in flight")?;
    let now = util::timestamp::now();
    
    // Verify ping timeout
    if now.saturating_sub(req.last_pong) > TUNNEL_PING_TIMEOUT {

Between getting the request and checking the timestamp, the last_pong could be updated by the pong handler. While unlikely to cause issues (would only make timeout detection more lenient), this is technically a TOCTOU issue.

Recommendation: This is probably acceptable as-is since it only affects timeout detection precision, but consider documenting this behavior.

6. Dead Code Warning (Minor)

Location: engine/packages/pegboard-gateway/src/shared_state.rs:442-447

WebSocketMessageNotAcked {
    #[allow(dead_code)]
    first_msg_index: u16,
    #[allow(dead_code)]
    last_msg_index: u16,
},

These fields are marked as dead_code but are present in the enum. If they're only for debugging:

Either use them in logging (remove #[allow(dead_code)])
Or remove them if truly unused

Recommendation: Use them in the tracing when garbage collecting:

tracing::warn!(
    request_id=?tunnel_id::request_id_to_string(&request_id),
    ?reason,
    first_msg_index=reason.first_msg_index,
    last_msg_index=reason.last_msg_index,
    "removing in-flight request due to gc"
);

🧪 Testing Considerations

Test Coverage: No tests were added for the new ping/pong functionality. Consider adding tests for:
- Ping timeout detection
- RTT metric recording
- Proper pong handling
- Task abort behavior
Integration Testing: Verify the 30-second timeout works correctly in real-world scenarios with network latency.

🔒 Security Assessment

✅ No security concerns identified. The implementation:

Doesn't introduce injection vulnerabilities
Properly validates message types
Uses safe timeout mechanisms
Doesn't expose sensitive information in metrics

📊 Performance Considerations

Metric Recording: The histogram recording happens on every pong, which is fine for 3-second intervals.
Memory: The last_pong field adds 8 bytes per in-flight request, which is negligible.
CPU: Additional periodic ping tasks are lightweight (3-second intervals).

Summary Rating

Overall: 8/10 - Good implementation with one critical timing issue and several minor improvements needed.

Action Items:

🔴 Must Fix: Ping timing issue (send immediately, then sleep)
🟡 Should Fix: Add documentation for send_and_check_ping
🟡 Should Consider: Use or remove dead_code fields in gc logging
🟢 Nice to Have: Add tests for ping/pong behavior

Great work on improving connection reliability! The refactoring into separate task modules makes the code much more maintainable.

claude · 2025-11-19T23:28:53Z

Pull Request Review: Implement Ping-Pong for Tunnel Connections

Overview

This PR implements a ping-pong mechanism for tunnel connections between the gateway and pegboard runners. The implementation adds RTT (Round-Trip Time) tracking and timeout detection for WebSocket tunnels.

Code Quality & Best Practices

Positives:

✅ Good separation of concerns with dedicated task modules (ping_task.rs, tunnel_to_ws_task.rs, ws_to_tunnel_task.rs)
✅ Proper use of structured logging with tracing instrumentation
✅ Consistent error handling patterns using anyhow::Result
✅ Good use of watch channels for task cancellation/abort signals
✅ Metrics collection for RTT monitoring

Issues:

Import organization (engine/packages/guard-core/src/custom_serve.rs:4-8)

Import moved from bottom to top, but mixing local and external imports. Consider grouping:

use bytes::Bytes;
use http_body_util::Full;
use hyper::{Request, Response};
use tokio_tungstenite::tungstenite::protocol::frame::CloseFrame;

use pegboard::tunnel::id::RequestId;
use crate::{WebSocketHandle, proxy_service::ResponseBody, request_context::RequestContext};

Hardcoded constants - The timeout value TUNNEL_PING_TIMEOUT is set to 30 seconds, which should work for most cases, but consider making this configurable or documenting why this specific value was chosen.

Potential Bugs & Issues

Clock Skew Handling (engine/packages/pegboard-runner/src/ws_to_tunnel_task.rs:92-111)
- Good: The code handles clock skew by checking if ping.ts > now
- Issue: When clock skew is detected, RTT is set to 0, which could give misleading metrics. Consider:
  - Using a sentinel value (e.g., u32::MAX) to indicate invalid RTT
  - Or logging this as a warning and not updating RTT at all
RTT Calculation Assumption (engine/packages/pegboard-runner/src/ws_to_tunnel_task.rs:110)
```
// Assuming symmetric delta
let rtt = delta * 2;
```
This assumes symmetric network paths. While generally reasonable, consider documenting this assumption more prominently in a comment or module-level doc.
Race Condition in Pong Handling (engine/packages/pegboard-gateway/src/shared_state.rs:255-270)
- The code updates in_flight.last_pong = now instead of using the original ping timestamp
- This is actually correct for timeout checking, but the comment at line 261 says "dropping ping" when it should say "dropping pong"
Missing Error Context (engine/packages/pegboard-gateway/src/ping_task.rs:21)
- If send_and_check_ping fails, the error propagates and terminates the ping task
- Consider whether transient errors should be retried rather than terminating the entire WebSocket connection

Performance Considerations

Ping Interval - UPDATE_PING_INTERVAL is set to 3 seconds
- With TUNNEL_PING_TIMEOUT at 30 seconds, this gives ~10 missed pings before timeout
- This seems reasonable, but under high load, consider whether 3-second intervals might create unnecessary network traffic
- ✅ Good: The interval uses tokio::time::sleep which is efficient
Metrics Recording (engine/packages/pegboard-gateway/src/shared_state.rs:270)
```
metrics::TUNNEL_PING_DURATION.record(rtt as f64 * 0.001, &[]);
```
- Converting milliseconds to seconds (multiplying by 0.001)
- ✅ Recording happens without blocking
Task Abort Coordination (engine/packages/pegboard-gateway/src/lib.rs:447-469)
- The abort logic properly coordinates between three tasks (tunnel_to_ws, ws_to_tunnel, ping)
- ✅ Good use of tokio::join! for concurrent task completion
- Minor: The cloning of abort senders (tunnel_to_ws_abort_tx2, etc.) could be cleaner with better naming or restructuring

Security Concerns

Timeout Enforcement - ✅ Good: The timeout check properly prevents zombie connections
Message Validation - ✅ The protocol uses versioned serialization with proper error handling
Resource Cleanup - ✅ Tasks properly clean up on abort/timeout

No critical security issues identified.

Test Coverage

⚠️ Missing Test Coverage:

No test files are included in this PR
Consider adding tests for:
1. Ping timeout detection (happy path and timeout path)
2. Clock skew handling
3. RTT calculation correctness
4. Task abort coordination
5. Metrics recording

Recommended test structure:

#[cfg(test)]
mod tests {
    #[tokio::test]
    async fn test_ping_timeout_detection() { /* ... */ }
    
    #[tokio::test]
    async fn test_clock_skew_handling() { /* ... */ }
    
    #[tokio::test]
    async fn test_rtt_calculation() { /* ... */ }
}

Protocol Changes

The protocol schema changes look good:

✅ Added ToRunnerPing and ToGatewayPong messages
✅ Properly integrated into the ToRunner and ToGateway unions
✅ Timestamp field (ts: i64) for RTT calculation

Question: Are there any backwards compatibility concerns with older runners/gateways that don't support ping-pong?

Suggestions

Documentation: Add module-level documentation explaining the ping-pong protocol flow
Error Handling: Consider retry logic for transient ping failures
Configuration: Make timeout values configurable via environment variables or config
Logging: The comment at line 261 should be updated from "dropping ping" to "dropping pong"
Tests: Add comprehensive test coverage as noted above

Summary

Overall Assessment: This is a solid implementation of ping-pong for tunnel connections. The code is well-structured and handles most edge cases properly.

Recommendation: Approve with minor fixes:

Fix the "dropping ping" comment (should be "pong")
Consider retry logic for transient ping errors
Add test coverage before merging
Document the RTT calculation assumptions

Risk Level: Low - The implementation is defensive and includes proper timeout handling.

claude · 2025-11-20T03:09:00Z

PR Review: fix(tunnel): implement ping pong

Summary

This PR implements a ping/pong mechanism for the tunnel between the pegboard gateway and runner to detect connection failures and measure RTT. The implementation introduces a periodic ping task on both sides and updates the protocol to support ping/pong messages.

Code Quality & Best Practices ✅

Strengths:

Good separation of concerns with new dedicated task modules (ping_task.rs, tunnel_to_ws_task.rs, ws_to_tunnel_task.rs)
Proper use of structured logging with tracing (e.g., tracing::debug!(?close, "server closed websocket"))
Consistent error handling patterns using the project's custom error types
Good use of #[tracing::instrument] for observability
Follows the repository's conventions for imports (at the top of files)

Observations:

The refactoring from inline tasks to separate modules improves code organization significantly
Metrics integration is well-done with proper histogram for RTT measurements

Potential Issues & Bugs 🔍

1. Clock Skew Handling on Gateway Side ⚠️

In pegboard-gateway/src/shared_state.rs:221, the ping timeout check uses:

if now.saturating_sub(req.last_pong) > TUNNEL_PING_TIMEOUT {

However, there's no clock skew protection like in the runner's implementation (pegboard-runner/src/ws_to_tunnel_task.rs:95-107). If the gateway's clock is behind the runner's clock, last_pong could be in the future, and saturating_sub would return 0, never timing out.

Recommendation: Add clock skew detection similar to the runner:

let elapsed = if req.last_pong <= now {
    now.saturating_sub(req.last_pong)
} else {
    tracing::warn!(last_pong = req.last_pong, now, "last_pong is in the future");
    0
};
if elapsed > TUNNEL_PING_TIMEOUT { ... }

2. RTT Calculation Assumption 📊

In pegboard-runner/src/ws_to_tunnel_task.rs:110, the RTT is calculated as delta * 2:

// Assuming symmetric delta
let rtt = delta * 2;

This assumes symmetric network latency, which may not always be accurate. The comment acknowledges this, but this could lead to inaccurate RTT metrics.

Recommendation: Consider documenting this limitation in the metrics description or exploring one-way latency measurement if precision is critical.

3. Ping Interval vs Timeout Mismatch ⏱️

UPDATE_PING_INTERVAL = 3 seconds (gateway sends ping every 3s)
TUNNEL_PING_TIMEOUT = 30 seconds (gateway times out after 30s)

This means ~10 missed pings before timeout, which seems reasonable. However, there's no explicit documentation of this ratio.

Recommendation: Add a comment explaining the relationship between these constants.

4. Metrics Recording Location 📈

The RTT metric is recorded in shared_state.rs:269 on the gateway side when receiving a pong:

let rtt = now.saturating_sub(pong.ts);
metrics::TUNNEL_PING_DURATION.record(rtt as f64 * 0.001, &[]);

However, if the pong arrives late (after the in-flight request is removed), the metric won't be recorded (line 255-262 early returns). This could skew metrics toward successful connections only.

Recommendation: Consider whether to record metrics for late pongs or document this behavior.

5. Missing Error Propagation in GC 🗑️

In shared_state.rs:479-485, when a WebSocket message timeout occurs during GC, the error is logged but the message is just removed. The client won't receive any notification about why the connection was terminated.

This is consistent with the existing pattern, but worth noting that clients may experience unexpected disconnections.

Performance Considerations ⚡

1. Additional Network Traffic

Each active tunnel now sends a ping every 3 seconds
For systems with many concurrent tunnels, this adds constant overhead
Impact: Minimal for most use cases, but worth monitoring in high-scale deployments

2. Task Spawning Overhead

Each WebSocket connection now spawns 3 tasks instead of 2 (added ping task)
Impact: Negligible, as Tokio tasks are lightweight

3. Atomic Operations

The runner's last_rtt uses AtomicU32 with Ordering::Relaxed, which is appropriate for metrics that don't require strict ordering guarantees.

Security Concerns 🔒

No significant security issues identified. The ping/pong mechanism:

Uses existing authentication channels
Doesn't expose sensitive information
Properly validates message timestamps for clock skew

Test Coverage 🧪

Concern: No test files were added in this PR.

Recommendations:

Add integration tests to verify ping/pong behavior
Test timeout scenarios (e.g., runner stops responding to pings)
Test clock skew scenarios
Verify metrics are recorded correctly

Example test cases:

Normal ping/pong flow
Ping timeout triggers connection closure
Clock skew doesn't cause false timeouts
RTT metrics are recorded accurately

Additional Observations

Good Patterns:

Abort Channel Pattern: The use of watch channels for coordinated task shutdown is clean and idiomatic
Error Context: Good use of .context() for error messages
Tracing Fields: Structured logging with request IDs makes debugging easier

Minor Style Issues:

Import Order (guard-core/src/custom_serve.rs:6-7): The import was moved but this is just a reordering - no issue
Cargo.toml Formatting (pegboard-gateway/Cargo.toml:23-24): serde_json and serde were reordered, harmless

Summary & Recommendations

Overall Assessment: ✅ Good implementation with solid architecture

Must Fix:

Add clock skew handling for the gateway's ping timeout check

Should Consider:

Add test coverage for ping/pong functionality
Document the ping interval to timeout ratio
Consider whether late pongs should be recorded in metrics

Nice to Have:

Explore more accurate RTT measurement if precision is important
Monitor network overhead in production

The core implementation is sound and follows the project's conventions well. The main concern is the missing clock skew protection on the gateway side, which could lead to incorrect timeout behavior.

Generated with Claude Code

claude · 2025-11-20T03:12:11Z

Pull Request Review: fix(tunnel): implement ping pong

Overview

This PR implements a ping/pong mechanism for the tunnel protocol between the pegboard gateway and runner to detect connection health and timeouts. The implementation adds a dedicated ping task that runs alongside the existing tunnel message forwarding tasks.

Positive Aspects

Architecture & Design

Good separation of concerns: The refactoring splits the monolithic message handling into dedicated tasks (tunnel_to_ws_task, ws_to_tunnel_task, ping_task) which improves code organization and maintainability
Protocol design: The ping/pong messages are well-designed with timestamps for RTT measurement and proper message types in the BARE schema
Metrics integration: Added observability with TUNNEL_PING_DURATION histogram to track round-trip time

Code Quality

Proper error handling: The ping timeout check correctly returns WebSocketServiceTimeout error when connections are stale
Task coordination: The three-way task coordination with abort channels is implemented correctly
Instrumentation: Good use of tracing attributes for debugging (e.g., request_id formatting)

Issues & Concerns

1. Potential Race Condition in `last_pong` Update (Medium severity)

Location: engine/packages/pegboard-gateway/src/shared_state.rs:256-268

The last_pong field is updated while holding a reference from get_async(), but this doesn't provide atomic guarantees. Between checking the timeout and sending a ping, another thread could update last_pong.

let Some(mut in_flight) = self.in_flight_requests.get_async(&pong.request_id).await
in_flight.last_pong = now;  // Mutation through reference

Recommendation: Consider using an AtomicI64 for last_pong similar to how last_rtt is handled in the runner code, or ensure the concurrent access pattern is safe.

2. Timing Configuration Mismatch (High severity)

Location: engine/packages/pegboard-gateway/src/lib.rs:41 and shared_state.rs:19

const UPDATE_PING_INTERVAL: Duration = Duration::from_secs(3);  // 3 seconds
const TUNNEL_PING_TIMEOUT: i64 = util::duration::seconds(30);    // 30 seconds

With a ping interval of 3 seconds and timeout of 30 seconds, you only allow ~10 consecutive missed pings before declaring a timeout. However, the check happens before sending each ping:

// Verify ping timeout
if now.saturating_sub(req.last_pong) > TUNNEL_PING_TIMEOUT {
    tracing::warn!("tunnel timeout");
    return Err(WebSocketServiceTimeout.build());
}

Issues:

On the very first ping, last_pong is initialized to util::timestamp::now() (line 119), so the initial state is good
However, if network delays or processing delays occur, this could lead to false positives
No jitter or backoff strategy for network congestion

Recommendation:

Add a safety margin (e.g., 45-60 seconds timeout for 3 second interval)
Consider adding jitter to ping intervals to avoid thundering herd
Document the relationship between these constants

3. Missing Error Context (Low severity)

Location: engine/packages/pegboard-gateway/src/shared_state.rs:215-216

.context("request not in flight")?;

This error message doesn't include the request_id, making debugging difficult.

Recommendation:

.with_context(|| format!("request {:?} not in flight", tunnel_id::request_id_to_string(&request_id)))?;

4. Inconsistent Import Organization (Low severity)

Location: Multiple files

The CLAUDE.md states "Always add imports at the top of the file inside of inline within the function," but there are inconsistencies. For example, in lib.rs:

use rivet_guard_core::{
    WebSocketHandle,
    custom_serve::{CustomServeTrait, HibernationResult},
    errors::{ServiceUnavailable, WebSocketServiceUnavailable},
    // ...
};

Some imports are grouped with braces spanning multiple lines with tabs, some with different formatting.

Recommendation: Ensure consistent formatting matches the existing codebase style.

5. Dead Code Attribute (Low severity)

Location: engine/packages/pegboard-gateway/src/shared_state.rs:444-446

WebSocketMessageNotAcked {
    #[allow(dead_code)]
    first_msg_index: u16,
    #[allow(dead_code)]
    last_msg_index: u16,
},

While #[allow(dead_code)] is used here for debugging purposes, the fields should either be used (e.g., in the debug log) or documented why they're kept.

Recommendation: Either use these fields in debugging output or add a comment explaining why they're preserved.

6. Potential Performance Impact (Low severity)

Location: engine/packages/pegboard-gateway/src/shared_state.rs:211-244

The send_and_check_ping method performs:

Async HashMap lookup
Timestamp check
Message serialization
Publish to UPS

This runs every 3 seconds for every active request. For high concurrency scenarios (many simultaneous WebSocket connections), this could create significant overhead.

Recommendation:

Consider batching ping checks if many requests are in flight
Profile under load to ensure acceptable performance
Document expected performance characteristics

7. Lifecycle Result Handling Could Be Clearer (Low severity)

Location: engine/packages/pegboard-gateway/src/lib.rs:472-480

let mut lifecycle_res = match (tunnel_to_ws_res, ws_to_tunnel_res, ping_res) {
    // Prefer error
    (Err(err), _, _) => Err(err),
    (_, Err(err), _) => Err(err),
    (_, _, Err(err)) => Err(err),
    // Prefer non aborted result if both succeed
    (Ok(res), Ok(LifecycleResult::Aborted), _) => Ok(res),
    (Ok(LifecycleResult::Aborted), Ok(res), _) => Ok(res),
    // Unlikely case
    (res, _, _) => res,
};

The comment says "unlikely case" but doesn't explain what the actual case is (all three are Ok and non-aborted, or ping result taking precedence).

Recommendation: Add more detailed comments or restructure for clarity.

Security Considerations

Positive

Timeout protection: The ping/pong mechanism provides good protection against hung connections
No DoS vulnerability: The ping interval is reasonable and won't create excessive traffic

Potential Issue

Resource exhaustion: If many connections timeout simultaneously, the error handling could create a spike in cleanup operations. Consider rate limiting or batching cleanup operations.

Testing Recommendations

This PR lacks test coverage. Recommended tests:

Unit tests:
- Test ping timeout detection with mocked timestamps
- Test pong message handling updates last_pong correctly
- Test task abort coordination
Integration tests:
- Test connection survives normal ping/pong cycles
- Test connection terminates on missing pongs
- Test RTT metrics are recorded correctly
- Test behavior under network delays
Load tests:
- Test performance with many concurrent connections
- Verify no thundering herd problems with synchronized pings

Documentation

Missing:

No comments explaining the ping/pong flow in the main lifecycle code
Constants lack documentation about their relationship
No README or architecture doc updates

Recommendation: Add inline comments explaining the three-task architecture and timing constraints.

Summary

This is a solid implementation of connection health checking with good separation of concerns. The main concerns are:

Must fix: Timing configuration needs review (3s interval / 30s timeout might be too tight)
Should fix: Potential race condition with last_pong updates
Should add: Test coverage for the ping/pong mechanism
Nice to have: Better error messages, documentation, and performance profiling

The refactoring to separate tasks is a good architectural improvement that makes the code more maintainable.

Recommended Changes

Increase TUNNEL_PING_TIMEOUT to 45-60 seconds or add explicit safety margin documentation
Make last_pong atomic or document why the current access pattern is safe
Add test coverage for timeout scenarios
Improve error messages with request IDs
Document the ping/pong mechanism in code comments

claude · 2025-11-20T04:53:31Z

PR Review: Implement Ping Pong for Tunnel

Summary

This PR implements a ping/pong mechanism for the tunnel connection between the pegboard gateway and runner. The implementation adds health checking and RTT (round-trip time) monitoring to detect disconnected or slow connections.

Code Quality & Best Practices ✅

Strengths:

Good modularization: The refactoring splits lib.rs into focused task modules (ping_task.rs, tunnel_to_ws_task.rs, ws_to_tunnel_task.rs)
Proper use of structured logging with tracing attributes
Consistent error handling patterns following the anyhow Result conventions
Good separation of concerns between gateway and runner implementations
Metrics integration for observability (RTT tracking)

Minor Issues:

Import ordering (engine/packages/guard-core/src/custom_serve.rs:7-10): Import reordering changes should ideally be in a separate commit, but this is minor
Dead code warnings: The #[allow(dead_code)] attributes in shared_state.rs:444-447 suggest these fields might be useful for debugging but aren't currently used in production code

Potential Bugs & Issues ⚠️

Ping task starts immediately without initial delay (pegboard-gateway/src/ping_task.rs:13-22): The ping loop sleeps first, which means the first ping happens after UPDATE_PING_INTERVAL (3 seconds). This could cause a false timeout if the connection takes longer than TUNNEL_PING_TIMEOUT (30 seconds) to establish and no messages are exchanged. Consider sending an initial ping before entering the loop.
Race condition in pong handling (pegboard-gateway/src/shared_state.rs:255-270): The last_pong timestamp is updated when receiving a pong, but there's no validation that the pong timestamp matches any sent ping. This could lead to incorrect RTT measurements if messages are reordered or duplicated.
Incomplete diff truncation: The diff appears to be truncated at line 54 in tunnel_to_ws_task.rs. I can see let gateway_reply_to = GatewayReceiverSubject::new(ping.gatewa but the line is cut off. Please verify the complete implementation is correct.
Task coordination on abort (pegboard-gateway/src/lib.rs:425-449): The abort signal coordination uses let _ = abort_tx.send(()) which ignores errors. If a task has already exited, the send will fail silently. While this is likely intentional, a comment explaining this would improve code clarity.

Performance Considerations 🔄

Positive:

Ping interval of 3 seconds is reasonable and shouldn't cause excessive overhead
Using watch channels for abort signaling is efficient
RTT metric recording uses appropriate float conversion (milliseconds to seconds)

Considerations:

Memory allocation in shared_state::send_and_check_ping: Every ping creates a new versioned message and serializes it. At 3-second intervals, this should be fine, but consider message pooling if ping frequency increases
Blocking async operation: In shared_state.rs:221, checking last_pong happens while holding a read lock on the HashMap entry. This is fine for the current simple check, but be aware if this logic becomes more complex

Security Concerns 🔒

Timeout value (TUNNEL_PING_TIMEOUT = 30 seconds): This is quite generous and could allow a zombie connection to persist for 30 seconds after failure. Consider if this timeout aligns with your SLA requirements
No ping flood protection: The ping mechanism doesn't have rate limiting. While the ping task controls the sending rate, a malicious runner could send unsolicited pongs. Consider adding validation or rate limiting on pong messages
Timestamp trust: The implementation trusts timestamps from the runner (pong.ts). While RTT calculation is informational, ensure this can't be exploited to cause overflow or other issues

Test Coverage 📋

Missing Test Coverage:

No unit tests visible for the new ping/pong functionality
No integration tests demonstrating timeout behavior
No tests for RTT metric recording
No tests for the task abort/coordination logic

Recommendations:

Add unit tests for send_and_check_ping timeout detection
Add integration tests that simulate:
- Normal ping/pong flow
- Timeout scenario (no pong received)
- Runner disconnection during ping
Consider adding tests for the task lifecycle (abort signal coordination)

Additional Observations

Documentation:

The new modules lack module-level documentation explaining their purpose
Consider adding doc comments for the LifecycleResult enum and its variants
The metrics module could benefit from examples of expected metric values

Protocol Changes:

The protocol changes (ToRunnerPing, ToGatewayPong) look appropriate
Good that the old ToRunnerKeepAlive is being replaced with actual ping/pong
The TypeScript stringify helpers are thorough and well-structured

Verdict

This is a solid implementation of a necessary feature. The code follows project conventions and the architecture is sound. The main concerns are:

Lack of test coverage for critical timeout logic
Potential race condition in timestamp handling
Missing initial ping before the first interval

I recommend adding tests before merging, particularly around the timeout behavior since that's critical for connection health.

🤖 Generated with Claude Code

claude · 2025-11-20T04:53:36Z

PR Review: Implement ping/pong for tunnel connections

Overview

This PR implements a ping/pong mechanism for tunnel connections between the gateway and pegboard runner, replacing the previous keep-alive system. This improves connection health monitoring and allows tracking of Round-Trip Time (RTT) metrics.

Strengths

✅ Good architectural improvements:

Properly refactored the monolithic websocket handler into separate task modules (ping_task.rs, tunnel_to_ws_task.rs, ws_to_tunnel_task.rs)
Added RTT metrics collection for observability
Implemented proper timeout handling for detecting dead connections

✅ Protocol enhancements:

Replaced ToRunnerKeepAlive with proper ToRunnerPing/ToGatewayPong messages
Bidirectional ping/pong implementation with timestamp tracking

✅ Code quality:

Good use of structured logging with tracing
Proper error handling patterns
Added helpful stringify utilities for debugging in TypeScript SDK

Issues & Concerns

1. Potential Race Condition in Ping Timeout Logic ⚠️

Location: engine/packages/pegboard-gateway/src/shared_state.rs:221

if now.saturating_sub(req.last_pong) > TUNNEL_PING_TIMEOUT {
    tracing::warn!("tunnel timeout");
    return Err(WebSocketServiceTimeout.build());
}

Issue: The timeout check happens in send_and_check_ping, but last_pong is initialized to util::timestamp::now() when the request starts. This means:

If the first ping is sent immediately after initialization, the timeout check may never trigger until 30 seconds pass
There's a window where the connection could be dead but not detected for up to TUNNEL_PING_TIMEOUT + UPDATE_PING_INTERVAL (33 seconds)

Suggestion: Consider initializing last_pong to 0 or adding a flag to track if we've received at least one pong.

2. Missing Error Handling for Pong Messages ⚠️

Location: engine/packages/pegboard-gateway/src/shared_state.rs:255-270

When a pong is received, if the request is not in flight, it's logged and dropped. However, there's no handling for:

Pongs with mismatched timestamps (replay attacks or clock skew issues)
Pongs received out of order

While this may be acceptable for the current use case, consider adding validation or at least documenting this behavior.

3. Task Abortion Logic Complexity 🤔

Location: engine/packages/pegboard-gateway/src/lib.rs:425-471

The task abortion logic with multiple watch channels is complex:

let (tunnel_to_ws_abort_tx, tunnel_to_ws_abort_rx) = watch::channel(());
let (ws_to_tunnel_abort_tx, ws_to_tunnel_abort_rx) = watch::channel(());
let (ping_abort_tx, ping_abort_rx) = watch::channel(());

let tunnel_to_ws_abort_tx2 = tunnel_to_ws_abort_tx.clone();
let ws_to_tunnel_abort_tx2 = ws_to_tunnel_abort_tx.clone();
let ping_abort_tx2 = ping_abort_tx.clone();

Concern: The abortion logic requires careful cloning and sending to multiple channels. While it appears correct, the _tx2 naming pattern makes it hard to track which clone is used where.

Suggestion: Consider using more descriptive names or adding comments explaining the ownership pattern, e.g.:

let tunnel_to_ws_abort_tx_for_ws = tunnel_to_ws_abort_tx.clone();

4. Inconsistent Result Handling Pattern 🤔

Location: engine/packages/pegboard-gateway/src/lib.rs:460-465

let mut lifecycle_res = match (tunnel_to_ws_res, ws_to_tunnel_res, ping_res) {
    (Err(err), _, _) => Err(err),
    (_, Err(err), _) => Err(err),
    (_, _, Err(err)) => Err(err),
    (Ok(res), Ok(LifecycleResult::Aborted), _) => Ok(res),
    (Ok(LifecycleResult::Aborted), Ok(res), _) => Ok(res),
    (res, _, _) => res,
};

The comment says "Unlikely case" for the last arm, but this doesn't handle the case where ping succeeds with a non-Aborted result while the other two are Aborted. The logic might be correct, but it's not immediately clear.

Suggestion: Add more explicit pattern matching or documentation explaining the precedence rules.

5. Magic Number Constants 📝

Location: engine/packages/pegboard-gateway/src/shared_state.rs:18

const TUNNEL_PING_TIMEOUT: i64 = util::duration::seconds(30);

Location: engine/packages/pegboard-gateway/src/lib.rs:40

const UPDATE_PING_INTERVAL: Duration = Duration::from_secs(3);

These constants are related but defined in different files. Consider:

Documenting why 30 seconds was chosen for timeout
Explaining the relationship between ping interval (3s) and timeout (30s) - appears to allow for ~10 missed pings

6. Import Organization ℹ️

Location: Multiple files

Several files have imports that don't follow the project's style of grouping. For example:

engine/packages/guard-core/src/custom_serve.rs:4-10

use pegboard::tunnel::id::RequestId;  // Added after other imports

use crate::WebSocketHandle;
use crate::proxy_service::ResponseBody;
use crate::request_context::RequestContext;

Note: Per CLAUDE.md: "Always add imports at the top of the file inside of inline within the function." The imports should be properly grouped (std, external crates, internal crates).

Performance Considerations

✅ Efficient ping interval: 3 seconds is reasonable and won't cause excessive overhead

⚠️ Metrics overhead: Recording metrics on every pong is fine, but ensure the histogram buckets (BUCKETS) are appropriately configured for expected RTT values (typically 1-500ms for cloud environments)

Security Considerations

✅ No major security concerns - The ping/pong mechanism is internal and doesn't expose sensitive data

ℹ️ Clock skew: The implementation uses util::timestamp::now() on both sides. If there's significant clock skew between gateway and runner, RTT calculations could be inaccurate (but this is a monitoring issue, not a security one).

Testing

❓ Missing information: The PR doesn't show any test changes. Consider:

Unit tests for ping timeout logic
Integration tests for the ping/pong mechanism
Tests for edge cases (connection drops during ping, etc.)

Minor Issues

TypeScript stringify.ts: The new file is well-structured but adds 336 lines. Consider if all these stringify functions are necessary or if a more generic approach could reduce code duplication.
Dead code annotation: engine/packages/pegboard-gateway/src/shared_state.rs:443-446
```
#[allow(dead_code)]
first_msg_index: u16,
#[allow(dead_code)]
last_msg_index: u16,
```
These fields are useful for debugging but marked as dead code. Consider using them in the log message or removing them.
Logging consistency: Some logs use tracing::debug! while others use tracing::warn!. Ensure consistent severity levels (e.g., shared_state.rs:261 logs "dropping ping" as debug, which seems appropriate).

Recommendations

Before merging:

✅ Review and address the ping timeout initialization concern
✅ Add inline documentation for the task abortion logic
✅ Consider adding tests for the ping/pong mechanism
✅ Clean up import organization to match project standards

Nice to have:

Document the relationship between ping interval and timeout constants
Use the first_msg_index/last_msg_index in the GC log message or remove the fields
Consider consolidating the TypeScript stringify utilities if possible

Conclusion

This is a solid implementation that improves connection health monitoring. The main concerns are around edge cases in the timeout logic and code maintainability. With the suggested improvements, this will be a good addition to the codebase.

Overall assessment: ✅ Approve with minor suggested improvements

This was referenced Nov 19, 2025

chore(engine-runner): remove echo close on ToClientWebSocketClose #3487

Open

chore(pegboard-gateway): remove TunnelAck #3488

Open

vercel bot had a problem deploying to Preview – rivetkit-serverless November 19, 2025 21:20 Failure

NathanFlurry changed the base branch from 11-19-chore_pegboard-gateway_remove_tunnelack_ to graphite-base/3491 November 19, 2025 21:21

NathanFlurry mentioned this pull request Nov 19, 2025

chore(pegboard-gateway): add new message id format & add deprecated tunnel ack #3492

Open

NathanFlurry force-pushed the 11-19-fix_tunnel_implement_ping_pong branch from 11b75bb to 73e19ae Compare November 19, 2025 21:42

NathanFlurry force-pushed the graphite-base/3491 branch from 759df3e to e47241f Compare November 19, 2025 21:42

NathanFlurry changed the base branch from graphite-base/3491 to 11-19-chore_pegboard-gateway_add_new_message_id_format_add_deprecated_tunnel_ack November 19, 2025 21:42

NathanFlurry marked this pull request as ready for review November 19, 2025 21:42

vercel bot had a problem deploying to Preview – rivetkit-serverless November 19, 2025 21:43 Failure

MasterPtato force-pushed the 11-19-fix_tunnel_implement_ping_pong branch from 73e19ae to 1340648 Compare November 19, 2025 21:54

MasterPtato force-pushed the 11-19-chore_pegboard-gateway_add_new_message_id_format_add_deprecated_tunnel_ack branch from e47241f to 6521b98 Compare November 19, 2025 21:54

vercel bot had a problem deploying to Preview – rivetkit-serverless November 19, 2025 21:54 Failure

MasterPtato force-pushed the 11-19-fix_tunnel_implement_ping_pong branch from 1340648 to 633be91 Compare November 19, 2025 21:55

vercel bot had a problem deploying to Preview – rivetkit-serverless November 19, 2025 21:56 Failure

MasterPtato mentioned this pull request Nov 19, 2025

fix(pb): simplify runner wf #3483

Draft

NathanFlurry force-pushed the 11-19-chore_pegboard-gateway_add_new_message_id_format_add_deprecated_tunnel_ack branch from 6521b98 to d6a2e79 Compare November 19, 2025 23:25

NathanFlurry force-pushed the 11-19-fix_tunnel_implement_ping_pong branch from 633be91 to 4c35d85 Compare November 19, 2025 23:25

vercel bot deployed to Preview – rivetkit-serverless November 19, 2025 23:28 View deployment

fix(tunnel): implement ping pong

f0f270e

MasterPtato force-pushed the 11-19-chore_pegboard-gateway_add_new_message_id_format_add_deprecated_tunnel_ack branch from d6a2e79 to adeb05d Compare November 20, 2025 03:07

MasterPtato force-pushed the 11-19-fix_tunnel_implement_ping_pong branch from 4c35d85 to 879f607 Compare November 20, 2025 03:07

vercel bot deployed to Preview – rivetkit-serverless November 20, 2025 03:08 View deployment

NathanFlurry force-pushed the 11-19-fix_tunnel_implement_ping_pong branch from 879f607 to f0f270e Compare November 20, 2025 04:51

NathanFlurry force-pushed the 11-19-chore_pegboard-gateway_add_new_message_id_format_add_deprecated_tunnel_ack branch from adeb05d to d6a2e79 Compare November 20, 2025 04:51

NathanFlurry mentioned this pull request Nov 20, 2025

chore(rivetkit): fix hibernation implementation #3494

Open

vercel bot deployed to Preview – rivetkit-serverless November 20, 2025 04:53 View deployment

fix(tunnel): implement ping pong #3491

Are you sure you want to change the base?

fix(tunnel): implement ping pong #3491

Uh oh!

Conversation

MasterPtato commented Nov 19, 2025

Uh oh!

vercel bot commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MasterPtato commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How to use the Graphite Merge Queue

Uh oh!

claude bot commented Nov 19, 2025

Pull Request Review: Implement Ping/Pong for Tunnel

Summary

✅ Strengths

🔍 Issues & Suggestions

1. Potential Race Condition in Ping Timeout Check ⚠️

2. Inconsistent RTT Calculation Between Gateway and Runner

3. Missing Error Context in Pong Handler

4. Ping Task Continues After Error

5. Clock Skew Handling

6. Metrics Bucket Configuration

7. Missing Tracing Instrumentation

🧪 Testing Concerns

🔒 Security

⚡ Performance

📝 Code Quality

✨ Suggestions for Future Work

Verdict

Uh oh!

claude bot commented Nov 19, 2025

Code Review: Ping-Pong Implementation for Tunnel Protocol

Strengths

Potential Issues

Code Quality Observations

Test Coverage

Security & Performance

Summary

Uh oh!

claude bot commented Nov 19, 2025

Pull Request Review: Implement Ping-Pong for Tunnel

Overall Assessment

Strengths

Issues and Concerns

1. Potential Race Condition in Ping Timeout Check (High Priority)

2. Ping Task Doesn't Send First Ping Immediately (Medium Priority)

3. RTT Calculation Assumes Symmetric Latency (Low Priority)

4. Missing Error Context in Metrics (Low Priority)

5. Protocol Version Compatibility (Medium Priority)

6. Inconsistent Import Ordering (Style)

Test Coverage

Security Considerations

Performance Considerations

Summary

Uh oh!

claude bot commented Nov 19, 2025

Code Review: Ping/Pong Implementation for Tunnels

Summary

✅ Strengths

🐛 Issues & Concerns

High Priority

Medium Priority

🔍 Minor Issues

✨ Suggestions

📊 Test Coverage

Security & Performance

Conclusion

Uh oh!

claude bot commented Nov 19, 2025

PR Review: fix(tunnel): implement ping pong

Summary

✅ Strengths

⚠️ Issues & Concerns

🧪 Testing Concerns

🔒 Security Considerations

📊 Performance Considerations

📝 Recommendations Summary

vercel bot commented Nov 19, 2025 •

edited

Loading

MasterPtato commented Nov 19, 2025 •

edited

Loading