fix(guard): use moka cache for in flight reqs

[MasterPtato]

No description provided.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

4 Skipped Deployments

Project	Deployment	Preview	Comments	Updated (UTC)
rivet-cloud	Ignored			Dec 11, 2025 8:34pm
rivet-inspector	Ignored			Dec 11, 2025 8:34pm
rivet-site	Ignored			Dec 11, 2025 8:34pm
rivetkit-serverless	Skipped			Dec 11, 2025 8:34pm

[MasterPtato]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• chore: add websocket close logs #3633
• chore: clean up scripts #3630
• fix(guard): use moka cache for in flight reqs #3629 👈 (View in Graphite)
• fix: add debug logs for config loading #3628
• fix(backport): fix serverless #3622
• fix: add serverless backfill #3588 : 1 other dependent PR (#3617 )
• chore: fix runner config creation #3580
• main

---

How to use the Graphite Merge Queue

Add the label merge-queue to this PR to add it to the merge queue.

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

This stack of pull requests is managed by Graphite. Learn more about stacking.

[claude]

Pull Request Review

Summary

This PR replaces Arc<Mutex<HashSetprotocol::RequestId>> with Cache<protocol::RequestId, ()> for tracking in-flight requests. This is a good improvement that leverages Moka cache for thread-safe tracking with automatic cleanup.

---

Positive Changes

1. Improved Concurrency

• Before: Used Arc<Mutex> which required acquiring a lock for all operations
• After: Uses Cache<RequestId, ()> which provides lock-free concurrent access patterns
• This significantly reduces lock contention when many requests are generating/releasing IDs simultaneously

2. Automatic Cleanup

• The Moka cache will automatically evict old entries when it reaches capacity (10M entries)
• Prevents unbounded memory growth if request IDs are leaked
• The previous HashSet implementation had no eviction policy

3. Consistent Pattern

• Aligns with existing code that already uses Moka caches for rate_limiters and in_flight_counters
• Reduces cognitive overhead by using the same pattern throughout

---

Potential Concerns

1. Cache Eviction and ID Collision Risk

The cache is configured with max_capacity(10_000_000) but no TTL (time-to-live).

Issue: If eviction occurs before a request completes:

1. An in-flight request ID could be evicted from the cache
2. A new request could generate the same ID (collision)
3. The collision would go undetected since the evicted ID is no longer in the cache

Impact: Low probability but possible under high load (approaching 10M concurrent requests).

Recommendation: Consider adding a TTL to ensure entries expire after a reasonable request timeout period. This ensures IDs are held for the expected request lifetime even if not explicitly invalidated.

2. Import Cleanup

The import collections::HashSet was removed correctly. The remaining StdHashMap is still used in the code, so the change is appropriate.

3. No Test Coverage for the Change

The PR does not include tests to verify:

• Concurrent ID generation under load
• ID uniqueness guarantees
• Cache eviction behavior

Recommendation: Add integration tests that verify ID uniqueness under concurrent load.

---

Performance Considerations

Throughput

• Improvement: Lock-free cache operations should improve throughput significantly
• Moka uses advanced concurrency techniques (segmented locks internally) that scale better than a single Mutex

Memory

• Before: HashSet - minimal overhead
• After: Moka cache has additional overhead for LRU tracking and internal data structures
• Verdict: Acceptable tradeoff for better concurrency

---

Security Considerations

No security issues identified. The change maintains the same collision detection logic and does not introduce new vulnerabilities.

---

Code Quality

Style

• Follows existing patterns in the codebase
• Clear and readable
• Consistent with CLAUDE.md conventions

Documentation

• No comments were needed to be updated
• The purpose of the field is clear from context

---

Recommendations

High Priority

1. Add TTL to cache to prevent premature eviction causing potential ID collisions

Medium Priority

2. Add integration tests for concurrent request ID generation
3. Consider metrics - track cache hit/miss rates and eviction counts for monitoring

Low Priority

4. Document the 10M capacity choice (why this specific number?)

---

Verdict

Approve with suggested improvements

This is a solid improvement that reduces lock contention and aligns with existing patterns. The main concern is the lack of TTL which could lead to rare but problematic ID collisions under extreme load. Adding a TTL would make this change production-ready with high confidence.

---

Estimated Impact:

• Code Quality: Improved
• Performance: Improved (reduced lock contention)
• Reliability: Slightly improved (automatic cleanup) but consider TTL
• Maintainability: Improved (consistent patterns)

[pkg-pr-new]

More templates

• actor-actions
• example-ai-agent
• example-ai-and-user-generated-actors-freestyle
• chat-room
• chat-room-next-js
• example-cloudflare-workers
• example-cloudflare-workers-hono
• example-cloudflare-workers-inline-client
• cross-actor-actions
• example-cursors
• example-cursors-raw-websocket
• custom-serverless
• example-deno
• example-sqlite
• example-elysia
• example-express
• example-hono
• hono-bun
• hono-react
• example-kitchen-sink
• multi-region
• native-websockets
• next-js
• node
• example-raw-fetch-handler
• example-raw-websocket-handler
• example-raw-websocket-handler-proxy
• example-react
• scheduling
• state
• example-stream
• example-trpc

@rivetkit/cloudflare-workers

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/cloudflare-workers@3629

@rivetkit/db

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/db@3629

@rivetkit/framework-base

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/framework-base@3629

@rivetkit/next-js

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/next-js@3629

@rivetkit/react

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/react@3629

rivetkit

pnpm add https://pkg.pr.new/rivet-dev/rivet/rivetkit@3629

@rivetkit/sql-loader

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/sql-loader@3629

@rivetkit/engine-runner

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/engine-runner@3629

@rivetkit/engine-runner-protocol

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/engine-runner-protocol@3629

commit: 80ffed1

[graphite-app]

Merge activity

• Dec 12, 12:48 AM UTC: NathanFlurry added this pull request to the Graphite merge queue.
• Dec 12, 12:49 AM UTC: CI is running for this pull request on a draft pull request (#3634) due to your merge queue CI optimization settings.
• Dec 12, 12:49 AM UTC: Merged by the Graphite merge queue via draft PR: #3634.