fix(pegboard): revise actor rescheduling algorithm, add client metrics #2531

MasterPtato · 2025-06-04T01:34:06Z

Changes

MasterPtato · 2025-06-04T01:34:41Z

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

How to use the Graphite Merge Queue

Add the label merge-queue to this PR to add it to the merge queue.

You must have a Graphite account in order to use the merge queue. Sign up using this link.

_{An organization admin has enabled the Graphite Merge Queue in this repository.} _{Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.}

This stack of pull requests is managed by Graphite. Learn more about stacking.

greptile-apps

PR Summary

This PR enhances Pegboard's actor management and monitoring capabilities by revising the rescheduling algorithm and adding detailed client resource metrics. The changes focus on improving reliability and observability of the system.

Added client resource metrics (CLIENT_MEMORY_TOTAL, CLIENT_CPU_TOTAL) with proper units (MiB/millicores) in /packages/edge/services/pegboard/src/metrics.rs
Implemented retry backoff reset with RETRY_RESET_DURATION_MS (10 min) in /packages/edge/services/pegboard/src/workflows/actor/mod.rs
Introduced structured RescheduleState for better retry tracking in /packages/edge/services/pegboard/src/workflows/actor/runtime.rs
Optimized metrics collection with single transaction fetch in /packages/edge/services/pegboard/src/workflows/client/mod.rs

_{4 file(s) reviewed, 2 comment(s)}
_{Edit PR Review Bot Settings | Greptile}

greptile-apps · 2025-06-04T01:35:14Z

packages/edge/services/pegboard/src/workflows/actor/runtime.rs

 				let now = util::timestamp::now();
-				state.retry_count =
-					if state.last_retry_ts < now - i64::try_from(2 * backoff.current_duration())? {
-						0
-					} else {
-						state.retry_count + 1
-					};
+				state.retry_count = if state.last_retry_ts < now - RETRY_RESET_DURATION_MS {
+					0
+				} else {
+					state.retry_count + 1
+				};
 				state.last_retry_ts = now;


logic: Potential race condition - state.last_retry_ts is updated after the check, which could lead to incorrect retry count calculations if multiple retries happen very close together

greptile-apps · 2025-06-04T01:35:17Z

packages/edge/services/pegboard/src/workflows/client/mod.rs

-		.set(cpu.try_into()?);
+		.set(total_cpu.try_into()?);
+
+	let alllocated_mem = total_mem.saturating_sub(remaining_mem);


syntax: Typo in variable name 'alllocated_mem' (three 'l's)

Suggested change

let alllocated_mem = total_mem.saturating_sub(remaining_mem);

let allocated_mem = total_mem.saturating_sub(remaining_mem);

cloudflare-workers-and-pages · 2025-06-04T01:37:38Z

Deploying rivet with Cloudflare Pages

Latest commit:	`bbbf37e`
Status:	✅ Deploy successful!
Preview URL:	https://45988c62.rivet.pages.dev
Branch Preview URL:	https://06-04-fix-pegboard-revise-ac.rivet.pages.dev

View logs

cloudflare-workers-and-pages · 2025-06-04T19:24:15Z

Deploying rivet-studio with Cloudflare Pages

Latest commit:	`bbbf37e`
Status:	✅ Deploy successful!
Preview URL:	https://19017757.rivet-studio.pages.dev
Branch Preview URL:	https://06-04-fix-pegboard-revise-ac.rivet-studio.pages.dev

View logs

cloudflare-workers-and-pages · 2025-06-04T19:24:18Z

Deploying rivet-hub with Cloudflare Pages

Latest commit:	`bbbf37e`
Status:	✅ Deploy successful!
Preview URL:	https://416232b6.rivet-hub-7jb.pages.dev
Branch Preview URL:	https://06-04-fix-pegboard-revise-ac.rivet-hub-7jb.pages.dev

View logs

MasterPtato · 2025-06-05T21:55:00Z

@greptileai

greptile-apps

PR Summary

Enhanced resource tracking and rescheduling logic in Pegboard service, with improved state management and metrics collection.

Reorganized metric definitions in packages/edge/services/pegboard/src/metrics.rs to group related CPU and memory metrics for better maintainability
Improved metric collection in packages/edge/services/pegboard/src/workflows/client/mod.rs with explicit draining state handling
Enhanced state persistence for actor rescheduling in packages/edge/services/pegboard/src/workflows/actor/runtime.rs with global state tracking

_{4 file(s) reviewed, 2 comment(s)}
_{Edit PR Review Bot Settings | Greptile}

greptile-apps · 2025-06-05T21:55:46Z

packages/edge/services/pegboard/src/workflows/client/mod.rs

+	let (total_mem, total_cpu, remaining_mem, remaining_cpu) =
 		ctx.fdb()


logic: Tuple order in destructuring doesn't match database query results. total_mem and total_cpu are swapped compared to query result order.

Suggested change

let (total_mem, total_cpu, remaining_mem, remaining_cpu) =

ctx.fdb()

let (total_mem, remaining_mem, total_cpu, remaining_cpu) =

ctx.fdb()

greptile-apps · 2025-06-05T21:55:47Z

packages/edge/services/pegboard/src/workflows/client/mod.rs

+					total_mem,
+					remaining_mem,
+					total_cpu,
+					remaining_cpu,


logic: Tuple return order doesn't match variable names above. Should be (total_mem, total_cpu, remaining_mem, remaining_cpu) to match destructuring.

Suggested change

total_mem,

remaining_mem,

total_cpu,

remaining_cpu,

total_mem,

total_cpu,

remaining_mem,

remaining_cpu,

graphite-app · 2025-06-09T16:46:58Z

Merge activity

Jun 9, 4:46 PM UTC: MasterPtato added this pull request to the Graphite merge queue.
Jun 9, 4:48 PM UTC: CI is running for this pull request on a draft pull request (#2574) due to your merge queue CI optimization settings.
Jun 9, 4:49 PM UTC: Merged by the Graphite merge queue via draft PR: #2574.

#2531)  ## Changes

MasterPtato requested a review from NathanFlurry June 4, 2025 01:34

MasterPtato mentioned this pull request Jun 4, 2025

fix: fix build cache key #2530

Closed

greptile-apps bot reviewed Jun 4, 2025

View reviewed changes

MasterPtato force-pushed the 06-04-fix_pegboard_revise_actor_rescheduling_algorithm_add_client_metrics branch from 51a568c to f9ef2e7 Compare June 4, 2025 01:46

This was referenced Jun 4, 2025

fix(pegboard): increase signal timeout, add check for actor exit #2538

Closed

fix(pegboard): wait for actor allocation not creation #2539

Closed

Fix(guard): return error response on router error #2540

Closed

MasterPtato force-pushed the 06-04-fix_pegboard_revise_actor_rescheduling_algorithm_add_client_metrics branch from f9ef2e7 to a52399a Compare June 4, 2025 19:17

MasterPtato force-pushed the 06-04-fix_fix_build_cache_key branch from 31b6aa7 to b7c048a Compare June 4, 2025 19:17

This was referenced Jun 4, 2025

feat(clusters): add margin per pool #2543

Closed

fix(pegboard): increase actor log ttl to 14 days #2544

Closed

MasterPtato force-pushed the 06-04-fix_fix_build_cache_key branch from 4184d72 to e53a26f Compare June 5, 2025 02:10

MasterPtato force-pushed the 06-04-fix_pegboard_revise_actor_rescheduling_algorithm_add_client_metrics branch from 6f7c837 to ec71a65 Compare June 5, 2025 02:10

This was referenced Jun 5, 2025

chore: fix dc wf #2552

Closed

fix: add vector http config to edge configs #2553

Closed

greptile-apps bot reviewed Jun 5, 2025

View reviewed changes

fix(pegboard): revise actor rescheduling algorithm, add client metrics

bbbf37e

MasterPtato force-pushed the 06-04-fix_pegboard_revise_actor_rescheduling_algorithm_add_client_metrics branch from ec71a65 to bbbf37e Compare June 6, 2025 01:36

MasterPtato force-pushed the 06-04-fix_fix_build_cache_key branch from e53a26f to b9dcd49 Compare June 6, 2025 01:36

graphite-app bot closed this Jun 9, 2025

graphite-app bot deleted the 06-04-fix_pegboard_revise_actor_rescheduling_algorithm_add_client_metrics branch June 9, 2025 16:49

github-actions bot mentioned this pull request Jun 9, 2025

chore(main): release 25.5.1 #2423

Merged

MasterPtato mentioned this pull request Jun 9, 2025

fix(pegboard): fix container runner orphaning #2575

Closed

	let alllocated_mem = total_mem.saturating_sub(remaining_mem);
	let allocated_mem = total_mem.saturating_sub(remaining_mem);

		let (total_mem, total_cpu, remaining_mem, remaining_cpu) =
		ctx.fdb()

Uh oh!

fix(pegboard): revise actor rescheduling algorithm, add client metrics #2531

fix(pegboard): revise actor rescheduling algorithm, add client metrics #2531

Uh oh!

Conversation

MasterPtato commented Jun 4, 2025

Changes

Uh oh!

MasterPtato commented Jun 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How to use the Graphite Merge Queue

Uh oh!

greptile-apps bot left a comment

Choose a reason for hiding this comment

PR Summary

Uh oh!

greptile-apps bot Jun 4, 2025

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot Jun 4, 2025

Choose a reason for hiding this comment

Uh oh!

cloudflare-workers-and-pages bot commented Jun 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying rivet with Cloudflare Pages

Uh oh!

cloudflare-workers-and-pages bot commented Jun 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying rivet-studio with Cloudflare Pages

Uh oh!

cloudflare-workers-and-pages bot commented Jun 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying rivet-hub with Cloudflare Pages

Uh oh!

MasterPtato commented Jun 5, 2025

Uh oh!

greptile-apps bot left a comment

Choose a reason for hiding this comment

PR Summary

Uh oh!

greptile-apps bot Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

graphite-app bot commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge activity

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

MasterPtato commented Jun 4, 2025 •

edited

Loading

cloudflare-workers-and-pages bot commented Jun 4, 2025 •

edited

Loading

cloudflare-workers-and-pages bot commented Jun 4, 2025 •

edited

Loading

cloudflare-workers-and-pages bot commented Jun 4, 2025 •

edited

Loading

graphite-app bot commented Jun 9, 2025 •

edited

Loading