fix: gracefully handle prom failure for pb topo

[MasterPtato]

Changes

Summary by CodeRabbit

• Bug Fixes
  • Improved error handling for Prometheus query failures, ensuring the system gracefully falls back to an empty dataset instead of returning an error.
  • Added a safeguard to prevent scaling actions when Prometheus data is unavailable, avoiding unintended operations.
  • Updated Prometheus query time range selectors to use an open-ended range starting 3 hours ago, enhancing metric data accuracy.

[coderabbitai]

Walkthrough

The update enhances Prometheus data handling by changing error propagation to explicit matching with error logging and a fallback empty response in pegboard_client_usage_get. It also modifies Prometheus query time range selectors from [3h] to [3h:]. Additionally, it adds an early-exit in calculate_diff to skip scaling if Prometheus data was not fetched, preventing further processing.

Changes

File(s)	Change Summary
.../pegboard.rs	Changed Prometheus query handling to log errors and return a default empty response instead of propagating error.
.../topology_get/mod.rs	Modified Prometheus query time range selectors from [3h] to [3h:] in CPU and network bandwidth queries.
.../scale.rs	Added early-exit in calculate_diff to log error and return empty output if Prometheus data is unavailable.

Poem

When Prometheus trips, we softly log the fall,
No panic, no crash, we answer the call.
If data's not fetched, we quietly refrain,
The cluster stays steady, no scaling strain.
🐇💡🌿

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 Clippy (1.86.0)

Updating crates.io index
Updating git repository `https://github.com/rivet-gg/serde_array_query`

error: failed to get serde_array_query as a dependency of package api-helper v25.5.2 (/packages/common/api-helper/build)

Caused by:
failed to load source for dependency serde_array_query

Caused by:
Unable to update https://github.com/rivet-gg/serde_array_query?rev=b9f8bfa#b9f8bfad

Caused by:
failed to create directory /usr/local/git/db/serde_array_query-49d74d117e610833

Caused by:
Permission denied (os error 13)

---

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between c1a3bf7 and 85f5b63.

📒 Files selected for processing (3)

• packages/core/services/cluster/src/ops/datacenter/topology_get/mod.rs (2 hunks)
• packages/core/services/cluster/src/ops/datacenter/topology_get/pegboard.rs (1 hunks)
• packages/core/services/cluster/src/workflows/datacenter/scale.rs (1 hunks)

✅ Files skipped from review due to trivial changes (1)

• packages/core/services/cluster/src/ops/datacenter/topology_get/mod.rs

🚧 Files skipped from review as they are similar to previous changes (2)

• packages/core/services/cluster/src/ops/datacenter/topology_get/pegboard.rs
• packages/core/services/cluster/src/workflows/datacenter/scale.rs

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)

• GitHub Check: cargo-check-main
• GitHub Check: claude-review
• GitHub Check: Cloudflare Pages: rivet-studio

✨ Finishing Touches

• 📝 Generate Docstrings

---

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Explain this complex logic.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai explain this code block.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and explain its main purpose.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR.
• @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[MasterPtato]

• fix(toolchain): fix compat with deploying using podman #2571 : 2 dependent PRs (#2710 , #2713 )
• fix(toolchain): dont upgrade for deploys by default #2743
• fix(workflows): fix signal publish/listen race condition #2742
• fix: upgrade actors by build name #2741
• fix: gracefully handle prom failure for pb topo #2731 👈 (View in Graphite)
• main

---

How to use the Graphite Merge Queue

Add the label merge-queue to this PR to add it to the merge queue.

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

This stack of pull requests is managed by Graphite. Learn more about stacking.

[claude]

Claude finished @MasterPtato's task —— View job

---

Code Review Complete ✅

Reviewing PR: fix: gracefully handle prom failure for pb topo

Todo List:

• Read the modified file to understand the changes
• Analyze code quality and best practices
• Check for potential bugs or issues
• Evaluate performance considerations
• Review security concerns
• Assess test coverage implications
• Provide comprehensive feedback

---

📝 Review Summary

Change: Line 61 modified from .await?; to .await; with added error handling (lines 63-71) to gracefully handle Prometheus failures.

✅ Positive Aspects

1. Excellent Error Handling: The change implements proper graceful degradation by catching prometheus errors and falling back to empty data instead of failing the entire operation.

2. Good Logging: The error is properly logged with structured logging (tracing::error!(?err, ...)) which will help with debugging and monitoring.

3. Consistent with Early Check: The change aligns well with the existing prometheus availability check at lines 27-32, creating a consistent pattern of graceful handling when prometheus is unavailable.

4. Preserves API Contract: The function still returns the expected Output structure, just with an empty clients HashMap when prometheus fails.

🔍 Code Quality Assessment

Score: 8.5/10

• Style: Follows Rust idioms with proper match expressions and error handling
• Readability: Clear intent with descriptive comments
• Maintainability: Simple, straightforward change that's easy to understand

🐛 Potential Issues

Minor Concerns:

1. Silent Failures: While graceful degradation is good, consider if there should be any mechanism to surface prometheus availability issues to operators/monitoring systems beyond just logging.

2. Inconsistent Return Types: The Default::default() call on line 69 works but could be more explicit. Consider using Vec::new() for clarity.

🚀 Performance Considerations

Score: 10/10

• No Performance Impact: The change adds minimal overhead (just a match expression)
• Fail-Fast: Prometheus errors are handled immediately without additional processing
• Memory Efficient: Empty response uses minimal memory

🔒 Security Assessment

Score: 10/10

• No Security Concerns: The change doesn't introduce any security vulnerabilities
• Error Information: Logging includes structured error information which is appropriate for debugging
• No Sensitive Data: The change doesn't expose any sensitive information

🧪 Test Coverage Considerations

Recommendations:

1. Unit Tests: Consider adding tests for the prometheus failure scenario:

   #[tokio::test]
   async fn test_prometheus_failure_graceful_handling() {
       // Test that prometheus failures return empty results
   }

2. Integration Tests: Test the complete flow with prometheus unavailable/failing

📋 Recommendations

1. Consider Metrics: Add a counter/gauge to track prometheus failures for operational visibility
2. Documentation: Consider adding a comment explaining the graceful degradation behavior
3. Consistency: Ensure other similar prometheus calls in the codebase follow the same pattern

🎯 Overall Assessment

Approval: ✅ LGTM

This is a solid defensive programming improvement that enhances system resilience. The change is minimal, well-implemented, and follows best practices for error handling. The graceful degradation approach is appropriate for a metrics collection system where temporary prometheus unavailability shouldn't break core functionality.

Risk Level: Low
Merge Recommendation: Approved after addressing optional test coverage suggestions

---

[cloudflare-workers-and-pages]

Deploying rivet with    Cloudflare Pages

Latest commit:	85f5b63
Status:	✅  Deploy successful!
Preview URL:	https://5a6d7791.rivet.pages.dev
Branch Preview URL:	https://07-08-fix-gracefully-handle.rivet.pages.dev

View logs

[claude]

Claude finished @MasterPtato's task —— View job

---

Code Review Complete ✅

Reviewing PR: fix: gracefully handle prom failure for pb topo

Todo List:

• Read the modified file to understand the changes
• Analyze code quality and best practices
• Check for potential bugs or issues
• Evaluate performance considerations
• Review security concerns
• Assess test coverage implications
• Provide comprehensive feedback

---

📝 Review Summary

Change: Line 61 modified from .await?; to .await; with added error handling (lines 63-71) to gracefully handle Prometheus failures.

✅ Positive Aspects

1. Excellent Error Handling: The change implements proper graceful degradation by catching prometheus errors and falling back to empty data instead of failing the entire operation.

2. Good Logging: The error is properly logged with structured logging (tracing::error!(?err, ...)) which will help with debugging and monitoring.

3. Consistent with Early Check: The change aligns well with the existing prometheus availability check at lines 27-32, creating a consistent pattern of graceful handling when prometheus is unavailable.

4. Preserves API Contract: The function still returns the expected Output structure, just with an empty clients HashMap when prometheus fails.

🔍 Code Quality Assessment

Score: 8.5/10

• Style: Follows Rust idioms with proper match expressions and error handling
• Readability: Clear intent with descriptive comments
• Maintainability: Simple, straightforward change that's easy to understand

🐛 Potential Issues

Minor Concerns:

1. Silent Failures: While graceful degradation is good, consider if there should be any mechanism to surface prometheus availability issues to operators/monitoring systems beyond just logging.

2. Inconsistent Return Types: The Default::default() call on line 69 works but could be more explicit. Consider using Vec::new() for clarity.

🚀 Performance Considerations

Score: 10/10

• No Performance Impact: The change adds minimal overhead (just a match expression)
• Fail-Fast: Prometheus errors are handled immediately without additional processing
• Memory Efficient: Empty response uses minimal memory

🔒 Security Assessment

Score: 10/10

• No Security Concerns: The change doesn't introduce any security vulnerabilities
• Error Information: Logging includes structured error information which is appropriate for debugging
• No Sensitive Data: The change doesn't expose any sensitive information

🧪 Test Coverage Considerations

Recommendations:

1. Unit Tests: Consider adding tests for the prometheus failure scenario:

   #[tokio::test]
   async fn test_prometheus_failure_graceful_handling() {
       // Test that prometheus failures return empty results
   }

2. Integration Tests: Test the complete flow with prometheus unavailable/failing

📋 Recommendations

1. Consider Metrics: Add a counter/gauge to track prometheus failures for operational visibility
2. Documentation: Consider adding a comment explaining the graceful degradation behavior
3. Consistency: Ensure other similar prometheus calls in the codebase follow the same pattern

🎯 Overall Assessment

Approval: ✅ LGTM

This is a solid defensive programming improvement that enhances system resilience. The change is minimal, well-implemented, and follows best practices for error handling. The graceful degradation approach is appropriate for a metrics collection system where temporary prometheus unavailability shouldn't break core functionality.

Risk Level: Low
Merge Recommendation: Approved after addressing optional test coverage suggestions

---

[cloudflare-workers-and-pages]

Deploying rivet-studio with    Cloudflare Pages

Latest commit:	85f5b63
Status:	🚫  Build failed.

View logs

[cloudflare-workers-and-pages]

Deploying rivet-hub with    Cloudflare Pages

Latest commit:	85f5b63
Status:	✅  Deploy successful!
Preview URL:	https://79d9436c.rivet-hub-7jb.pages.dev
Branch Preview URL:	https://07-08-fix-gracefully-handle.rivet-hub-7jb.pages.dev

View logs

[NathanFlurry]

log error in wf

[claude]

Claude finished @MasterPtato's task —— View job

---

Code Review Complete ✅

Reviewing PR: fix: gracefully handle prom failure for pb topo

Todo List:

• Read the modified pegboard.rs file to understand the changes
• Read the modified scale.rs file to understand the changes
• Analyze code quality and best practices
• Check for potential bugs or issues
• Evaluate performance considerations
• Review security concerns
• Assess test coverage implications
• Address reviewer feedback about workflow logging
• Provide comprehensive feedback

---

📝 Review Summary

This PR introduces graceful error handling for Prometheus failures in two key areas:

1. pegboard.rs: Lines 61-71 handle prometheus fetch errors by logging and returning empty data instead of failing
2. scale.rs: Lines 210-216 check prometheus_fetched flag and skip scaling when metrics are unavailable

✅ Positive Aspects

1. Excellent Error Handling Strategy: The approach follows the established pattern in the codebase where prometheus failures are handled gracefully with fallback to empty data.

2. Consistent Architecture: Both changes align with the existing prometheus_fetched pattern used in the parent topology_get module, maintaining architectural consistency.

3. Safety-First Approach: In scale.rs, the check prevents scaling decisions when metrics are unavailable, which is crucial for preventing incorrect auto-scaling actions.

4. Proper Logging: Both locations use structured logging with appropriate error context (tracing::error!(?err, ...)) for debugging and monitoring.

5. Preserves API Contracts: Functions still return expected output structures, just with empty data when prometheus fails.

🔍 Code Quality Assessment

Score: 9/10

Strengths:

• Clear intent and readable code
• Follows Rust idioms with proper match expressions
• Consistent error handling patterns
• Good separation of concerns

Areas for improvement:

• The Default::default() call in pegboard.rs could be more explicit as Vec::new() for clarity

🐛 Potential Issues

Minor Concerns:

1. Silent Degradation: While graceful degradation is correct, consider if there should be additional alerting mechanisms beyond logging for prometheus unavailability.

2. Scaling Behavior: The scale.rs change completely skips scaling when prometheus fails. This is likely correct but worth documenting that scaling is disabled during prometheus outages.

🚀 Performance Considerations

Score: 10/10

• Minimal Overhead: Changes add negligible performance impact (just match expressions)
• Fail-Fast Pattern: Errors are handled immediately without additional processing
• Memory Efficient: Empty responses use minimal memory footprint

🔒 Security Assessment

Score: 10/10

• No Security Concerns: No security vulnerabilities introduced
• Appropriate Error Information: Logged errors include necessary debugging context without exposing sensitive data
• Error Boundary: Proper error containment prevents information leakage

📋 Addressing Reviewer Feedback

Re: "log error in wf" from @NathanFlurry

✅ Addressed: The workflow (scale.rs) now properly logs the error at line 211:

tracing::error!("prometheus could not be fetched, not scaling");

This provides clear visibility into scaling decisions and aligns with the request to log errors in the workflow.

🧪 Test Coverage Considerations

Recommendations:

1. Unit Tests for Error Paths: Consider adding tests for both error scenarios:

   #[tokio::test]
   async fn test_pegboard_prometheus_failure_graceful_handling() {
       // Test prometheus failure returns empty results
   }
   
   #[tokio::test]
   async fn test_scaling_skipped_when_prometheus_unavailable() {
       // Test scaling workflow returns empty actions when prometheus_fetched = false
   }

2. Integration Tests: Test end-to-end behavior when prometheus is unavailable

📊 Architecture Alignment

Score: 10/10

The changes perfectly align with the existing architecture:

• Follows the same graceful degradation pattern as the parent topology_get module
• Maintains the prometheus_fetched flag contract
• Preserves the safety-first approach for scaling decisions

🎯 Overall Assessment

✅ LGTM - Approved

This is a well-implemented defensive programming improvement that enhances system resilience. The changes:

• Enhance Reliability: Prevent cascading failures when Prometheus is temporarily unavailable
• Maintain Safety: Preserve important safety checks for scaling decisions
• Follow Best Practices: Implement consistent error handling patterns across the codebase
• Improve Observability: Provide clear logging for debugging and monitoring

Risk Level: Very Low
Merge Recommendation: ✅ Approved

The implementation is solid, follows established patterns, and addresses both the immediate issue and the reviewer's feedback about workflow logging. The graceful degradation approach is appropriate for a metrics collection system where temporary prometheus unavailability shouldn't break core functionality.

---

x

[graphite-app]

Merge activity

• Jul 12, 12:58 AM UTC: NathanFlurry added this pull request to the Graphite merge queue.
• Jul 12, 1:00 AM UTC: CI is running for this pull request on a draft pull request (#2745) due to your merge queue CI optimization settings.
• Jul 12, 1:01 AM UTC: Merged by the Graphite merge queue via draft PR: #2745.