Skip to content

fix(tests): model watchdog restart lifecycle - #647

Merged
EtanHey merged 1 commit into
mainfrom
fix/red-baseline-watchdog
Aug 4, 2026
Merged

fix(tests): model watchdog restart lifecycle#647
EtanHey merged 1 commit into
mainfrom
fix/red-baseline-watchdog

Conversation

@EtanHey

@EtanHey EtanHey commented Aug 4, 2026

Copy link
Copy Markdown
Owner

Summary

Root cause

The test's fake launchctl print returned state = running with PID 4321 forever, including after the mocked SIGKILL. Production correctly refused to kickstart while that PID appeared alive, waited the real 45-second safety timeout, and returned recovery_failed. This reproduces alone on clean b95e113d, so it is a real #602 test regression rather than an item-24 concurrency artifact.

Verification

  • RED on clean main: named test failed in 45.17s with watcher pid 4321 survived SIGKILL for 45s
  • GREEN: named test passed in 0.05s after the final reviewed diff
  • focused module: 25 passed in 182.94s
  • targeted Ruff check: clean
  • targeted Ruff format check: clean
  • git diff check: clean
  • normal push hook with FD soft limit 4096: BrainLayer test gate passed; isolated hook-routing leg 40 passed, Bun leg 1 passed, FTS5 determinism shell regression passed
  • remote branch verified at exact head 9f0795703b5074c77edaa7729ff9dd597d0a3b9a

Review notes

  • Claude pair-review verdict: ACCEPT
  • local CodeRabbit suggested expanding this alert-framing fake into a complete command-protocol validator and sequence recorder; not taken because dedicated recovery tests already assert the sequence and the brief forbids drive-by widening

Known base-branch state

Repo-wide Ruff is still red only in the three files already fixed by the separate, unlanded 7214eec5 commit. This PR does not duplicate or reformat that work.

Worker endpoint is PR plus review responses. Do not merge from this lane.


Note

Low Risk
Single test mock change only; no production watchdog or recovery logic modified.

Overview
Test-only fix for test_alert_framing_pages_once_per_episode_not_per_kickstart, which was timing out (~45s) because its fake launchctl print always reported PID 4321 as running, even after a mocked SIGKILL—matching production behavior that blocks kickstart until the old PID is gone.

The test’s command_runner now keeps a small process state: launchctl print reflects running vs exited and the current PID, /bin/kill -9 marks the process dead, and launchctl kickstart -k bumps the PID and marks it running again. Alert-framing expectations (one page per episode, suppression within an episode, re-page after sustained health) are unchanged; only the mock matches the recovery sequence other tests already assert.

Reviewed by Cursor Bugbot for commit 9f07957. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Fix watchdog restart lifecycle simulation in test_alert_framing_pages_once_per_episode_not_per_kickstart

The test's command_runner mock previously returned a fixed running state and pid regardless of commands issued. It now tracks process state across the full lifecycle: launchctl print returns state = running or state = exited based on current state, /bin/kill -9 <pid> sets the process to stopped, and launchctl kickstart -k increments the pid and marks the process as running again.

Macroscope summarized 9f07957.

Summary by CodeRabbit

  • Tests
    • Improved alert-framing test coverage for watcher lifecycle transitions, including status reporting, termination, and restart behavior.

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your trial has ended. Reactivate Greptile to resume code reviews.

@EtanHey

EtanHey commented Aug 4, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@cursor

cursor Bot commented Aug 4, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_c9d83052-2f89-47f7-8c1e-3d45125faadc)

@EtanHey

EtanHey commented Aug 4, 2026

Copy link
Copy Markdown
Owner Author

@codex review

@EtanHey

EtanHey commented Aug 4, 2026

Copy link
Copy Markdown
Owner Author

@cursor @BugBot review

@cursor

cursor Bot commented Aug 4, 2026

Copy link
Copy Markdown

You need to increase your spend limit or enable usage-based billing to run background agents. Go to Cursor

@coderabbitai

coderabbitai Bot commented Aug 4, 2026

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai

coderabbitai Bot commented Aug 4, 2026

Copy link
Copy Markdown

Review Change Stack

Important

Review skipped

No new commits to review since the last review.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 9e2aeb67-7870-4496-a7ba-b189210c4504

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

The throughput watchdog test runner now simulates watcher lifecycle transitions. Status checks report the current PID and state. SIGKILL stops the watcher. Kickstart assigns a new PID and restores the running state.

Changes

Watchdog lifecycle testing

Layer / File(s) Summary
Model watcher lifecycle transitions
tests/test_throughput_watchdog.py
The test runner tracks PID and running state across status checks, SIGKILL, and kickstart commands.

Estimated code review effort: 2 (Simple) | ~10 minutes

Poem

A rabbit checks the watcher’s beat,
Then SIGKILL makes it leave its seat.
Kickstart brings a fresh PID,
And running paws resume with glee.
Tests now follow each lifecycle turn.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'fix(tests): model watchdog restart lifecycle' directly corresponds to the PR's main change—modifying the watchdog test to simulate a complete restart lifecycle with proper state transitions.
✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
  • Create stacked PR
  • Commit on current branch
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/red-baseline-watchdog

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. What shall we delve into next?

Reviewed commit: 9f0795703b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@EtanHey
EtanHey merged commit 949334a into main Aug 4, 2026
7 of 9 checks passed

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/test_throughput_watchdog.py`:
- Around line 723-725: Strengthen the kickstart test around the launchctl
handling branch by recording process["pid"] before each replacement and
asserting it differs afterward. Update the assertions in the surrounding
watchdog test so removing the process["pid"] += 1 behavior would fail, while
preserving the existing running-state checks.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 6a1aa1af-02bc-4068-999b-4fd2a34131d3

📥 Commits

Reviewing files that changed from the base of the PR and between b95e113 and 9f07957.

📒 Files selected for processing (1)
  • tests/test_throughput_watchdog.py
📜 Review details
⏰ Context from checks skipped due to timeout. (3)
  • GitHub Check: test (3.13)
  • GitHub Check: test (3.12)
  • GitHub Check: test (3.11)
🧰 Additional context used
📓 Path-based instructions (1)
tests/**/*.py

📄 CodeRabbit inference engine (AGENTS.md)

tests/**/*.py: Run pytest before claiming behavior changes are safe.
Tests must not refresh the production backup heartbeat log; use BRAINLAYER_BACKUP_LOG_PATH and BRAINLAYER_BACKUP_LOG_PROVENANCE=pytest.

Files:

  • tests/test_throughput_watchdog.py
🔇 Additional comments (1)
tests/test_throughput_watchdog.py (1)

712-725: 🎯 Functional Correctness

Run the focused pytest check before approval.

This file matches tests/**/*.py. Run the test with the required test-only backup-log settings before claiming the behavior is safe:

tmpdir="$(mktemp -d)"
trap 'rm -rf "$tmpdir"' EXIT
export BRAINLAYER_BACKUP_LOG_PATH="$tmpdir/backup-heartbeat.log"
export BRAINLAYER_BACKUP_LOG_PROVENANCE=pytest
pytest -q tests/test_throughput_watchdog.py -k 'test_alert_framing_pages_once_per_episode_not_per_kickstart'

Source: Coding guidelines

Comment on lines +723 to +725
elif args[:3] == ["launchctl", "kickstart", "-k"]:
process["pid"] += 1
process["running"] = True

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Assert the replacement PID.

process["pid"] += 1 models a replacement process, but the test never checks it. If Line 724 is removed, the assertions at Lines 747-764 still pass. Record the PID before each kickstart and assert that it changes afterward.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/test_throughput_watchdog.py` around lines 723 - 725, Strengthen the
kickstart test around the launchctl handling branch by recording process["pid"]
before each replacement and asserting it differs afterward. Update the
assertions in the surrounding watchdog test so removing the process["pid"] += 1
behavior would fail, while preserving the existing running-state checks.

EtanHey added a commit that referenced this pull request Aug 4, 2026
The 7 admin merges of 2026-08-04 (#645 #644 #643 #641 #633 #605 #602) bypassed CI,
landing import-order and formatting drift on main. Every PR's lint job inherits it.
Rebased onto #647 (949334a), which cleared the watchdog test blocking this push.

ruff check --fix + ruff format only; no behavioural change.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant