feat: add pb usage metrics, server state #2503

MasterPtato · 2025-05-30T23:34:17Z

Changes

MasterPtato · 2025-05-30T23:34:37Z

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

How to use the Graphite Merge Queue

Add the label merge-queue to this PR to add it to the merge queue.

You must have a Graphite account in order to use the merge queue. Sign up using this link.

_{An organization admin has enabled the Graphite Merge Queue in this repository.} _{Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.}

This stack of pull requests is managed by Graphite. Learn more about stacking.

greptile-apps

PR Summary

This PR introduces server state tracking and usage metrics functionality across the Rivet platform. The changes span both core and edge services, with significant modifications to server management and metrics collection.

Added new ServerState enum with 7 states (Provisioning through Destroyed) in cluster/src/types.rs for tracking server lifecycle
Introduced new Prometheus metrics in pegboard service for tracking CPU/memory usage per environment and client flavor
Created new standalone pegboard-usage-metrics-publish service to handle periodic collection and publishing of resource usage metrics
Moved client usage metrics from usage_get.rs to the new standalone service for better separation of concerns
Critical: Integration tests are missing for the new functionality, with only a TODO placeholder in integration.rs

_{13 file(s) reviewed, 12 comment(s)}
_{Edit PR Review Bot Settings | Greptile}

greptile-apps · 2025-05-30T23:35:10Z

packages/core/services/cluster/src/ops/server/get.rs

+			CASE
+				WHEN s.cloud_destroy_ts IS NOT NULL THEN 6  -- Destroyed
+				WHEN s.taint_ts IS NOT NULL AND s.drain_ts IS NOT NULL THEN 5  -- TaintedDraining
+				WHEN s.drain_ts IS NOT NULL THEN 4  -- Draining
+				WHEN s.taint_ts IS NOT NULL THEN 3  -- Tainted
+				WHEN s.install_complete_ts IS NOT NULL THEN 2  -- Running
+				WHEN s.provision_complete_ts IS NOT NULL THEN 1  -- Installing
+				ELSE 0  -- Provisioning
+			END AS state


syntax: Missing table alias 's' in FROM clause but used in CASE statement. Add 's AS s' to the FROM clause.

Suggested change

CASE

WHEN s.cloud_destroy_ts IS NOT NULL THEN 6 -- Destroyed

WHEN s.taint_ts IS NOT NULL AND s.drain_ts IS NOT NULL THEN 5 -- TaintedDraining

WHEN s.drain_ts IS NOT NULL THEN 4 -- Draining

WHEN s.taint_ts IS NOT NULL THEN 3 -- Tainted

WHEN s.install_complete_ts IS NOT NULL THEN 2 -- Running

WHEN s.provision_complete_ts IS NOT NULL THEN 1 -- Installing

ELSE 0 -- Provisioning

END AS state

FROM db_cluster.servers AS s

greptile-apps · 2025-05-30T23:35:10Z

packages/core/services/cluster/src/ops/server/list.rs

+			CASE
+				WHEN s.cloud_destroy_ts IS NOT NULL THEN 6  -- Destroyed
+				WHEN s.taint_ts IS NOT NULL AND s.drain_ts IS NOT NULL THEN 5  -- TaintedDraining
+				WHEN s.drain_ts IS NOT NULL THEN 4  -- Draining
+				WHEN s.taint_ts IS NOT NULL THEN 3  -- Tainted
+				WHEN s.install_complete_ts IS NOT NULL THEN 2  -- Running
+				WHEN s.provision_complete_ts IS NOT NULL THEN 1  -- Installing
+				ELSE 0  -- Provisioning
+			END AS state


style: State values should be defined as constants in the codebase to avoid magic numbers and ensure consistency across queries

greptile-apps · 2025-05-30T23:35:11Z

packages/core/services/cluster/src/ops/server/get.rs

+			CASE
+				WHEN s.cloud_destroy_ts IS NOT NULL THEN 6  -- Destroyed
+				WHEN s.taint_ts IS NOT NULL AND s.drain_ts IS NOT NULL THEN 5  -- TaintedDraining
+				WHEN s.drain_ts IS NOT NULL THEN 4  -- Draining
+				WHEN s.taint_ts IS NOT NULL THEN 3  -- Tainted
+				WHEN s.install_complete_ts IS NOT NULL THEN 2  -- Running
+				WHEN s.provision_complete_ts IS NOT NULL THEN 1  -- Installing
+				ELSE 0  -- Provisioning
+			END AS state


style: State transitions appear to be mutually exclusive, but consider adding explicit priority with AND NOT conditions to prevent edge cases where multiple timestamps are set.

greptile-apps · 2025-05-30T23:35:17Z

packages/core/services/cluster/src/types.rs

 	pub lan_ip: Option<IpAddr>,
 	pub wan_ip: Option<IpAddr>,
 	pub cloud_destroy_ts: Option<i64>,
+	pub state: ServerState,


style: The Server struct doesn't derive Serialize/Deserialize traits but contains a serializable state field. This may cause issues if Server needs to be serialized

greptile-apps · 2025-05-30T23:35:17Z

packages/core/services/cluster/src/types.rs

+	Running = 2,
+	Tainted = 3,
+	Draining = 4,
+	TaintedDraining = 5,


style: TaintedDraining combines two states - consider if this should be a separate flag instead of a combined state for better state management

greptile-apps · 2025-05-30T23:35:47Z

packages/edge/services/pegboard/src/metrics.rs

+	pub static ref ENV_CPU_USAGE: IntGaugeVec = register_int_gauge_vec_with_registry!(
+		"pegboard_env_cpu_usage",
+		"Total percent of CPU (per core) used by an environment.",
+		&["env_id", "flavor"],
+		*REGISTRY
+	).unwrap();


style: Missing trailing comma after *REGISTRY to maintain consistency with other metric definitions

Suggested change

pub static ref ENV_CPU_USAGE: IntGaugeVec = register_int_gauge_vec_with_registry!(

"pegboard_env_cpu_usage",

"Total percent of CPU (per core) used by an environment.",

&["env_id", "flavor"],

*REGISTRY

).unwrap();

pub static ref ENV_CPU_USAGE: IntGaugeVec = register_int_gauge_vec_with_registry!(

"pegboard_env_cpu_usage",

"Total percent of CPU (per core) used by an environment.",

&["env_id", "flavor"],

*REGISTRY,

).unwrap();

greptile-apps · 2025-05-30T23:35:48Z

packages/edge/services/pegboard/src/metrics.rs

+	pub static ref ENV_MEMORY_USAGE: IntGaugeVec = register_int_gauge_vec_with_registry!(
+		"pegboard_env_memory_usage",
+		"Total MiB of memory used by an environment.",
+		&["env_id", "flavor"],
+		*REGISTRY
+	).unwrap();


style: Missing trailing comma after *REGISTRY to maintain consistency with other metric definitions

Suggested change

pub static ref ENV_MEMORY_USAGE: IntGaugeVec = register_int_gauge_vec_with_registry!(

"pegboard_env_memory_usage",

"Total MiB of memory used by an environment.",

&["env_id", "flavor"],

*REGISTRY

).unwrap();

pub static ref ENV_MEMORY_USAGE: IntGaugeVec = register_int_gauge_vec_with_registry!(

"pegboard_env_memory_usage",

"Total MiB of memory used by an environment.",

&["env_id", "flavor"],

*REGISTRY,

).unwrap();

greptile-apps · 2025-05-30T23:36:11Z

packages/edge/services/pegboard/src/ops/client/usage_get.rs

logic: Complete removal of usage metrics tracking code without a clear replacement could break existing monitoring. Ensure equivalent functionality exists elsewhere before removing this file.

greptile-apps · 2025-05-30T23:36:14Z

packages/edge/services/pegboard/standalone/usage-metrics-publish/Cargo.toml

+tracing = "0.1"
+tracing-subscriber = { version = "0.3", default-features = false, features = ["fmt", "json", "ansi"] }


style: Consider using workspace version for tracing and tracing-subscriber to maintain consistency with other dependencies

greptile-apps · 2025-05-30T23:36:23Z

packages/edge/services/pegboard/standalone/usage-metrics-publish/src/lib.rs

+		if actor.start_ts.is_none() || actor.destroy_ts.is_some() {
+			continue;
+		}


style: Consider logging skipped actors with debug level to help with monitoring and debugging

cloudflare-workers-and-pages · 2025-05-30T23:37:42Z

Deploying rivet with Cloudflare Pages

Latest commit:	`39fc637`
Status:	✅ Deploy successful!
Preview URL:	https://eefe696a.rivet.pages.dev
Branch Preview URL:	https://05-30-feat-add-pb-usage-metr.rivet.pages.dev

View logs

cloudflare-workers-and-pages · 2025-05-31T00:54:40Z

Deploying rivet-studio with Cloudflare Pages

Latest commit:	`39fc637`
Status:	✅ Deploy successful!
Preview URL:	https://6d9ce416.rivet-studio.pages.dev
Branch Preview URL:	https://05-30-feat-add-pb-usage-metr.rivet-studio.pages.dev

View logs

cloudflare-workers-and-pages · 2025-05-31T01:05:58Z

Deploying rivet-hub with Cloudflare Pages

Latest commit:	`39fc637`
Status:	✅ Deploy successful!
Preview URL:	https://639f7abc.rivet-hub-7jb.pages.dev
Branch Preview URL:	https://05-30-feat-add-pb-usage-metr.rivet-hub-7jb.pages.dev

View logs

MasterPtato requested a review from NathanFlurry May 30, 2025 23:34

This was referenced May 30, 2025

fix(guard): add metrics #2480

Closed

fix(guard): replace internal caches with moka #2481

Closed

fix(guard): add more tokio runtime metrics, remove labels from metrics #2485

Closed

MasterPtato mentioned this pull request May 30, 2025

fix(cache): add traces #2489

Closed

greptile-apps bot reviewed May 30, 2025

View reviewed changes

MasterPtato force-pushed the 05-30-feat_add_pb_usage_metrics_server_state branch from 3de4613 to 464160e Compare May 31, 2025 00:36

NathanFlurry force-pushed the 05-29-fix_cache_add_traces branch from d69ce87 to ecbaa48 Compare May 31, 2025 00:39

NathanFlurry force-pushed the 05-30-feat_add_pb_usage_metrics_server_state branch from 464160e to 1eac431 Compare May 31, 2025 00:39

MasterPtato force-pushed the 05-29-fix_cache_add_traces branch from ecbaa48 to 47e00c0 Compare May 31, 2025 00:54

MasterPtato force-pushed the 05-30-feat_add_pb_usage_metrics_server_state branch from 1eac431 to 2f17dc8 Compare May 31, 2025 00:54

MasterPtato mentioned this pull request May 31, 2025

chore: fix guard metrics #2504

Closed

NathanFlurry force-pushed the 05-30-feat_add_pb_usage_metrics_server_state branch from 2f17dc8 to 1eac431 Compare May 31, 2025 01:01

NathanFlurry force-pushed the 05-29-fix_cache_add_traces branch from 47e00c0 to ecbaa48 Compare May 31, 2025 01:01

NathanFlurry mentioned this pull request May 31, 2025

chore: update workspace #2505

Closed

MasterPtato force-pushed the 05-30-feat_add_pb_usage_metrics_server_state branch from 1eac431 to 2f17dc8 Compare May 31, 2025 01:04

MasterPtato force-pushed the 05-29-fix_cache_add_traces branch from ecbaa48 to 47e00c0 Compare May 31, 2025 01:04

NathanFlurry force-pushed the 05-29-fix_cache_add_traces branch from 47e00c0 to ecbaa48 Compare May 31, 2025 01:52

NathanFlurry force-pushed the 05-30-feat_add_pb_usage_metrics_server_state branch from 2f17dc8 to 1eac431 Compare May 31, 2025 01:52

NathanFlurry mentioned this pull request May 31, 2025

feat(guard): add http ping endpoint to edge api hosts #2506

Closed

MasterPtato force-pushed the 05-29-fix_cache_add_traces branch from ecbaa48 to 47e00c0 Compare May 31, 2025 02:05

MasterPtato force-pushed the 05-30-feat_add_pb_usage_metrics_server_state branch from 1eac431 to 2f17dc8 Compare May 31, 2025 02:05

feat: add pb usage metrics, server state

39fc637

MasterPtato force-pushed the 05-29-fix_cache_add_traces branch from 47e00c0 to 90e20fd Compare June 2, 2025 18:13

MasterPtato force-pushed the 05-30-feat_add_pb_usage_metrics_server_state branch from 2f17dc8 to 39fc637 Compare June 2, 2025 18:13

MasterPtato mentioned this pull request Jun 2, 2025

fix: fix pb metrics standalone #2518

Closed

graphite-app bot closed this Jun 3, 2025

graphite-app bot deleted the 05-30-feat_add_pb_usage_metrics_server_state branch June 3, 2025 07:03

github-actions bot mentioned this pull request Jun 3, 2025

chore(main): release 25.5.1 #2423

Merged

		tracing = "0.1"
		tracing-subscriber = { version = "0.3", default-features = false, features = ["fmt", "json", "ansi"] }

Uh oh!

feat: add pb usage metrics, server state #2503

feat: add pb usage metrics, server state #2503

Uh oh!

Conversation

MasterPtato commented May 30, 2025

Changes

Uh oh!

MasterPtato commented May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How to use the Graphite Merge Queue

Uh oh!

greptile-apps bot left a comment

Choose a reason for hiding this comment

PR Summary

Uh oh!

greptile-apps bot May 30, 2025

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot May 30, 2025

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot May 30, 2025

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot May 30, 2025

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot May 30, 2025

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot May 30, 2025

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot May 30, 2025

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot May 30, 2025

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot May 30, 2025

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot May 30, 2025

Choose a reason for hiding this comment

Uh oh!

cloudflare-workers-and-pages bot commented May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying rivet with Cloudflare Pages

Uh oh!

cloudflare-workers-and-pages bot commented May 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying rivet-studio with Cloudflare Pages

Uh oh!

cloudflare-workers-and-pages bot commented May 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying rivet-hub with Cloudflare Pages

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

MasterPtato commented May 30, 2025 •

edited

Loading

cloudflare-workers-and-pages bot commented May 30, 2025 •

edited

Loading

cloudflare-workers-and-pages bot commented May 31, 2025 •

edited

Loading

cloudflare-workers-and-pages bot commented May 31, 2025 •

edited

Loading