feat(worker): add cpu_shares resource control to protect earners under host contention

[GeiserX]

Why

The storj node on watchtower was repeatedly reported offline by satellites (US1 offline emails) while the container was up and healthy. Root cause: satellites dial the node back in, and under heavy host contention (copyparty at 330%+ CPU, load 24-28 during a disk-zeroing run) those inbound connections timed out. To the network, a slow node is indistinguishable from a dead one — and sustained offline status leads to suspension and forfeited held balance.

The resources schema only supported memory controls (mem_limit, mem_reservation, oom_score_adj), so there was no way to keep an earner responsive when the host is saturated.

What

• ResourceSpec.cpu_shares in app/worker_api.py with validation (2..262144 — Docker treats 0 as default, rejects 1, kernel caps at 262144), rejected at the API instead of failing later at container create
• Passthrough in app/orchestrator.py deploy_raw alongside the existing memory kwargs
• Schema documentation in services/_schema.yml
• cpu_shares: 2048 for storj in services/storage/storj.yml — double the 1024 default; relative weight, so it only takes effect under contention and costs nothing on an idle host

Testing

• uv run pytest tests/ — 4539 passed, 6 skipped, 0 failed
• New tests: valid range accepted (2/1024/2048/262144), out-of-range rejected (0, 1, negatives, 262145), forwarding from both pydantic spec and plain catalog dict, omitted-when-absent
• Real-catalog guard test updated to assert storj carries cpu_shares: 2048
• ruff check + ruff format --check clean

Summary by CodeRabbit

• New Features

  • Added support for configuring container CPU scheduling priority with cpu_shares.
  • Added CPU priority settings to the Storj service.
  • Redeployments now preserve recorded resource values while adopting newly configured resource settings.

• Bug Fixes

  • Resource settings now correctly forward CPU scheduling values during deployment.
  • Invalid CPU share values outside the supported range of 2–262144 are rejected with a clear validation error.

• Documentation

  • Documented CPU share limits, contention behavior, and recommended settings for latency-sensitive services.

[coderabbitai]

Warning

Review limit reached

@GeiserX, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 54 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 91d48947-c13e-4fdd-83fc-eb4e54213141

📥 Commits

Reviewing files that changed from the base of the PR and between 1a90ff4 and eeb7693.

📒 Files selected for processing (1)

• tests/test_worker_resources.py

📝 Walkthrough

Walkthrough

The PR adds optional Docker cpu_shares support. It validates values from 2 through 262144, forwards configured values during deployment, documents the setting, configures Storj with 4096 shares, and reconciles recorded and catalog resource keys per key.

Changes

Docker resource limits

Layer / File(s)	Summary
Resource contract and validation app/worker_api.py, services/_schema.yml, tests/test_worker_resources.py	ResourceSpec supports optional cpu_shares. Validation rejects values outside 2 through 262144 with HTTP 400 responses. Service schemas and catalog checks document and validate Docker resource keys.
Recorded resource reconciliation app/main.py, tests/test_deployed_spec.py	Redeployment starts with catalog resources and overlays recorded values. Recorded values remain authoritative per key, including when catalog resources are removed.
Deployment resource forwarding app/orchestrator.py, tests/test_worker_resources.py	deploy_raw forwards explicit cpu_shares values to Docker and logs the applied resource configuration or Docker defaults.
Storj catalog configuration services/storage/storj.yml, tests/test_main_deploy_routes.py	The Storj service sets cpu_shares to 4096. The deployment test verifies the complete resource specification.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Possibly related PRs

• GeiserX/CashPilot#95: Adds the durable Docker resource-limit implementation extended by this PR.
• GeiserX/CashPilot#143: Modifies the same Docker resource and deployment configuration areas for a separate feature.
• GeiserX/CashPilot#146: Modifies deployment-spec reconciliation in app/main.py and related tests.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 21.43% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: adding cpu_shares resource control for workers under host CPU contention.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/cpu-shares-earner-protection

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.58%. Comparing base (03a8c0f) to head (eeb7693).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #317   +/-   ##
=======================================
  Coverage   95.57%   95.58%           
=======================================
  Files          51       51           
  Lines        6685     6697   +12     
=======================================
+ Hits         6389     6401   +12     
  Misses        296      296           

Files with missing lines	Coverage Δ
app/main.py	97.70% <100.00%> (+0.01%)	⬆️
app/orchestrator.py	81.71% <100.00%> (ø)
app/worker_api.py	89.64% <100.00%> (+0.05%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[GeiserX]

Review record (7-lens review + independent re-review of the fix)

Verdict on the original head (41b9595): two findings that mattered, both fixed in 1a90ff4.

1. HIGH — the feature never reached an already-deployed service. _merge_recorded_spec merged resources as an opaque whole with the recorded spec winning, so any deployment recorded before a resource key existed could never receive it on redeploy — proven by executing the real function: the recorded {mem_limit, oom_score_adj} beat the catalog and cpu_shares was dropped, silently (the divergence note is returned in a JSON field no UI renders). The only escape was remove-and-redeploy, which for storj is the identity-risk operation the recorded spec exists to prevent. The PR's guard test passed because it exercised the fresh-deploy path only. Fix: resources now merge per key, exactly like env ten lines below — the catalog supplies keys the record never set, a key the deployment actually ran with still wins, and only a genuine keep is reported. Six regression tests cover the merge path (it had zero before).

2. MEDIUM→value change — 2048 was counterproductive on older hosts. On cgroup v2 an unset container runs at cpu.weight 100. runc ≥ 1.2 maps shares 1024→100, 2048→174, 4096→303; runc < 1.2 maps linearly, where 2048→79 — below the unset default, i.e. the "protected" service would get less CPU than its neighbours. Anything ≥ ~2600 clears 100 on both mappings, so storj now ships 4096 (157 old / 303 new). The schema comment documents both mappings.

3. Root-cause correction. The motivating incident was re-verified live: the node's dial-back failures continue at a constant rate on an uncontended host, and the mechanism is a stale advertised ADDRESS after a WAN failover — a reachability problem no CPU weight can touch. The storj comment is reworded as a defensive rationale that names what the weight does not fix (I/O-wait, wrong/stale ADDRESS). The knob itself remains worth shipping: it is free when idle and real under container CPU contention.

Also in 1a90ff4: a catalog-wide guard (every docker.resources block must use keys ResourceSpec honors — a cpushares typo previously deployed green with the limit silently unset — tied back to _schema.yml, loading through app.catalog so .yaml services stay covered); deploy_raw logs the resource kwargs it applied (deploy responses never echo them, and version-skewed workers drop unknown keys silently); an endpoint-level 400 negative control for cpu_shares; and comment accuracy fixes (kernel MIN_SHARES..MAX_SHARES range, stale key enumerations, a test comment describing a production path that does not exist).

Follow-ups filed separately rather than widening this PR: compose export drops the whole resources block (pre-existing); Desktop needs CPUShares in its Go ResourceLimits + runtime before its catalog copy is synced; kept_from_previous_deployment deserves a UI rendering; and a storj reachability signal (lastPinged/quicStatus from the node's own API) — the check that would have diagnosed the incident in seconds.

Full suite: 4549 passed, 6 skipped, coverage 95.62%. Independent re-review of the fix commit: APPROVE.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/test_worker_resources.py`:
- Around line 379-390: Update test_every_honored_key_is_documented_in_the_schema
to parse _schema.yml as YAML and locate the properties mapping for
docker.resources, then assert every key in DOCUMENTED_RESOURCE_KEYS exists
there. Remove the raw substring check so matches in comments, descriptions, or
unrelated fields cannot satisfy the test.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: e9c7bf52-6401-4954-b9b1-48aa7c55ae26

📥 Commits

Reviewing files that changed from the base of the PR and between 41b9595 and 1a90ff4.

📒 Files selected for processing (8)

• app/main.py
• app/orchestrator.py
• app/worker_api.py
• services/_schema.yml
• services/storage/storj.yml
• tests/test_deployed_spec.py
• tests/test_main_deploy_routes.py
• tests/test_worker_resources.py

🚧 Files skipped from review as they are similar to previous changes (3)

• services/_schema.yml
• app/orchestrator.py
• app/worker_api.py