Expand PostHog telemetry to close audited coverage gaps

[NiveditJain]

Summary

• Adds events at every gap surfaced by the May coverage audit except allow-path hook tracking (intentional skip — high volume; deny/instruct already cover the interesting cases).
• 16 new server-side events across the CLI, hook handler, manager, install-prompt, custom-hooks-loader, policy-evaluator, and postinstall script.
• 12 new web-UI events in app/policies/hooks-client.tsx via usePostHog() — tab switches, debounced filter changes, row expand, copy buttons, pagination, CLI selection / Apply / Reinstall, modal open/close, error display, install-from-error.
• All events reuse the existing helpers (trackHookEvent, trackInstallEvent, captureClientEvent) — no new wrappers — and honor FAILPROOFAI_TELEMETRY_DISABLED=1.
• New ~/.failproofai/last-version sentinel file enables first_install vs version_changed discrimination in scripts/postinstall.mjs.

Server-side events

• bin/failproofai.mjs — cli_install_success / cli_install_failure / cli_uninstall_success / cli_uninstall_failure / cli_list_invoked / cli_parse_error / cli_unexpected_error / hook_dispatch_error
• src/hooks/handler.ts — hook_stdin_error, hook_payload_parse_error
• src/hooks/policy-evaluator.ts — policy_evaluation_error (builtin crashes; distinct from existing custom_hook_error)
• src/hooks/manager.ts — custom_policy_validation_failed, policy_params_validation_warning, scope_validation_failed, hook_write_failed, multi_scope_warning_shown, beta_policies_installed
• src/hooks/custom-hooks-loader.ts — custom_hooks_load_error (tagged with is_convention + convention_scope)
• src/hooks/install-prompt.ts — cli_detection_summary (detected vs explicit vs auto-defaulted)
• scripts/postinstall.mjs — first_install, version_changed (with upgrade/downgrade/reinstall direction)

Test plan

• bun run lint — clean (1 pre-existing warning, unrelated)
• bun --bun tsc --noEmit — clean
• bun run test:run — 1628 tests pass (1623 existing + 5 new in __tests__/hooks/new-telemetry.test.ts)
• bun run build:cli — builds clean
• Smoke: --version, policies --garbage (parse_error path), echo 'not json' | --hook PreToolUse (payload parse error path) all behave correctly
• CI: gh run watch after push
• Verify events arrive in PostHog (project "Claudeye") via MCP query after a real install/uninstall cycle

🤖 Generated with Claude Code

Summary by CodeRabbit

Release Notes

• New Features

  • Expanded event tracking across CLI operations, hook management, policy evaluation, and user interactions
  • Installation phase detection and version change tracking
  • Enhanced error reporting for hook operations and policy evaluation issues

• Tests

  • New test coverage for event tracking functionality

[coderabbitai]

Warning

Rate limit exceeded

@NiveditJain has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 10 minutes and 56 seconds before requesting another review.

You’ve run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 8c425959-92ae-454e-b9f5-a57612e0b619

📥 Commits

Reviewing files that changed from the base of the PR and between a43f253 and aff6a7e.

📒 Files selected for processing (3)

• CHANGELOG.md
• app/policies/hooks-client.tsx
• scripts/postinstall.mjs

📝 Walkthrough

Walkthrough

This PR adds comprehensive PostHog telemetry instrumentation across the CLI, Web-UI, and hook system. It captures CLI command success/failure outcomes, hook processing errors with classification, installation history, and user interactions in the policies dashboard, while respecting the FAILPROOFAI_TELEMETRY_DISABLED=1 environment variable.

Changes

PostHog telemetry coverage

Layer / File(s)	Summary
Telemetry test suite __tests__/hooks/new-telemetry.test.ts	New Vitest test file with module mocks establishing test patterns for manager-level scope validation, custom policy loading, parameter validation failures, and install-prompt CLI detection telemetry assertions.
Web-UI event capture app/policies/hooks-client.tsx	React component wires usePostHog() capture throughout: copy clicks, activity filter/pagination/row expansion, policy modal open/close, CLI checkbox toggles, tab switching, error displays, and install/reinstall submissions; adds centralized fireActionError helper for error telemetry.
CLI lifecycle and error telemetry bin/failproofai.mjs	Main CLI entry point adds lazy-loaded track() helper with telemetry for --hook dispatch errors, successful install/uninstall/list outcomes (capturing scope, CLI counts, flags), and top-level error handling that emits install/uninstall/parse/unexpected failure events differentiated by subcommand context.
Hook system instrumentation src/hooks/install-prompt.ts, src/hooks/custom-hooks-loader.ts, src/hooks/handler.ts, src/hooks/manager.ts, src/hooks/policy-evaluator.ts	Install-prompt emits CLI detection summary with resolution mode; custom-hooks-loader classifies load errors by type; handler tracks stdin oversizing and payload parse failures; manager validates scopes/policies/writes with error classification and tracks beta/multi-scope scenarios; policy-evaluator captures non-custom policy evaluation errors.
Installation lifecycle tracking scripts/postinstall.mjs	Postinstall now reads/writes ~/.failproofai/last-version, emits first_install on new installations or version_changed on upgrades/downgrades/reinstalls with platform/arch metadata via compareSemver helper.
Documentation CHANGELOG.md	Documents new PostHog telemetry events across CLI, hook errors, installation detection, and Web-UI interactions; notes telemetry disabling behavior.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

• FailproofAI/failproofai#374: Both PRs modify src/hooks/handler.ts's handleHookEvent—one removes enqueue/relay-daemon spawning, while the other adds/extends PostHog telemetry (stdin/payload parsing and related events)—so the changes are directly related.

Poem

🐰 Hop into the data,
Events tracked at every bound—
From CLI to Web, the hooks now hum,
With PostHog whispers all around!
(DISABLED respected too, of course)
🎯📊

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 64.71% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely summarizes the main change: expanding PostHog telemetry to address coverage gaps identified by an audit.
Description check	✅ Passed	The PR description is comprehensive and well-structured, covering summary, event details, and test plan; however, it lacks the explicit template structure with checkbox markers for Type of Change and Checklist.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 4

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)

app/policies/hooks-client.tsx (2)

212-245: 🛠️ Refactor suggestion | 🟠 Major | 🏗️ Heavy lift

Add unit tests for the new Web-UI telemetry triggers.

This file introduces multiple new client telemetry behaviors, but the provided new test file only covers manager/install-prompt server-side events. Please add focused unit tests under __tests__/hooks/ for the new UI capture points.

As per coding guidelines, "Always add unit tests for new behaviour. Place tests in tests/. Unit tests live in tests/hooks/, e2e tests in tests/e2e/hooks/."

Also applies to: 364-473, 988-1004, 1448-1460, 1562-1595

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@app/policies/hooks-client.tsx` around lines 212 - 245, Add unit tests under
__tests__/hooks/ that verify the new client-side telemetry capture calls in this
file: specifically test the CopyButton component's handleCopy behavior (import
CopyButton and simulate click) to assert usePostHog.capture is called with
"activity_copy_clicked" and the correct field value (and default "unknown"),
covering both navigator.clipboard.writeText success and the textarea fallback
path; similarly add focused tests for every other UI capture point introduced in
this file (the captures at the ranges noted) by mocking usePostHog, simulating
the UI interaction, and asserting the exact event name and payload are emitted.

---

1142-1153: ⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Track modal close telemetry on save path too.

policy_config_modal_closed is emitted for cancel/close, but not when Save closes the modal. Add a close event for the save-driven close so opened/closed metrics stay consistent.

Also applies to: 1177-1180

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@app/policies/hooks-client.tsx` around lines 1142 - 1153, handleSaveParams
currently closes the modal by calling setConfiguringPolicy(null) after a
successful save but does not emit the "policy_config_modal_closed" telemetry
event, causing open/close metrics to be inconsistent; update handleSaveParams to
emit the same telemetry event used for cancel/close (call the telemetry emitter
with "policy_config_modal_closed" and any required payload) immediately when the
modal is closed on save (after awaiting updatePolicyParamsAction and
before/after setConfiguringPolicy as your convention dictates), and apply the
same change to the analogous save handler around lines 1177-1180 so both
save-driven closes and cancel-driven closes emit the same metric; reference
functions: handleSaveParams, updatePolicyParamsAction, setConfiguringPolicy,
reload, and fireActionError to locate where to add the telemetry call.

🧹 Nitpick comments (1)

app/policies/hooks-client.tsx (1)

478-488: ⚡ Quick win

Keep state updater callbacks side-effect free.

capture(...) is currently called inside setState(prev => ...) updaters. Move telemetry emission outside the updater and compute next state first; this avoids non-idiomatic side effects and duplicate/dev-mode quirks.

Also applies to: 1052-1063

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@app/policies/hooks-client.tsx` around lines 478 - 488, The state updater
passed to setExpandedRow (the setExpandedRow(prev => { ... }) block) performs
side effects by calling capture(...) inside the updater; compute the next state
first (e.g., determine const next = prev === idx ? null : idx outside the
setState callback), call capture(...) after computing next (using items[idx] for
decision/policyName/eventType), and then call setExpandedRow(next) so the
updater remains side-effect free; apply the same refactor to the duplicate block
around the 1052-1063 area.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@app/policies/hooks-client.tsx`:
- Around line 451-466: The telemetry call capture("activity_filter_changed",
...) is firing on initial mount; update the effect surrounding that call in
app/policies/hooks-client.tsx to skip the first render by adding a persistent
flag (e.g., useRef isFirstRender) and returning early on first run, then set the
flag to false so subsequent changes run the existing capture logic; reference
the capture invocation and the effect that reads filterDecision,
filterEventType, filterPolicy, filterSessionId, and filterCli when locating
where to add the isFirstRender guard.

In `@CHANGELOG.md`:
- Line 6: The changelog entry describing the expanded PostHog telemetry (the
long one listing server-side events like cli_install_success/failure,
hook_dispatch_error, policy_evaluation_error, etc., and web-UI events via
usePostHog() in app/policies/hooks-client.tsx) is missing the PR reference
suffix; update that single-line entry to append the PR number in the required
format (for example " (`#376`)") so it becomes one line with the description
followed by the PR reference.

In `@scripts/postinstall.mjs`:
- Around line 176-182: The branch that sends version change telemetry assumes
previousVersion !== currentVersion, making the "reinstall" (cmp === 0) path
unreachable; update the logic around previousVersion/currentVersion so
same-version reinstalls are detectable and reported: compute cmp =
compareSemver(previousVersion, currentVersion) regardless of the outer condition
(or change the condition to allow previousVersion === currentVersion to reach
this block), then call trackInstallEvent("version_changed", { from_version:
previousVersion, to_version: currentVersion, direction: cmp < 0 ? "upgrade" :
cmp > 0 ? "downgrade" : "reinstall", platform: platform() }) so cmp === 0
correctly yields "reinstall" (refer to previousVersion, currentVersion,
compareSemver, trackInstallEvent, platform).
- Around line 141-157: The compareSemver comparator treats longer identifiers as
greater so "1.0.0-beta" > "1.0.0"; flip the tie-break when one side runs out of
identifiers so prerelease identifiers are lower precedence than the plain
release. In the compareSemver function change the undefined-token checks: when
ai === undefined return 1 (the shorter/release is greater) and when bi ===
undefined return -1; keep the existing numeric and lexicographic comparisons
otherwise so prerelease strings like "beta" compare below the corresponding
release.

---

Outside diff comments:
In `@app/policies/hooks-client.tsx`:
- Around line 212-245: Add unit tests under __tests__/hooks/ that verify the new
client-side telemetry capture calls in this file: specifically test the
CopyButton component's handleCopy behavior (import CopyButton and simulate
click) to assert usePostHog.capture is called with "activity_copy_clicked" and
the correct field value (and default "unknown"), covering both
navigator.clipboard.writeText success and the textarea fallback path; similarly
add focused tests for every other UI capture point introduced in this file (the
captures at the ranges noted) by mocking usePostHog, simulating the UI
interaction, and asserting the exact event name and payload are emitted.
- Around line 1142-1153: handleSaveParams currently closes the modal by calling
setConfiguringPolicy(null) after a successful save but does not emit the
"policy_config_modal_closed" telemetry event, causing open/close metrics to be
inconsistent; update handleSaveParams to emit the same telemetry event used for
cancel/close (call the telemetry emitter with "policy_config_modal_closed" and
any required payload) immediately when the modal is closed on save (after
awaiting updatePolicyParamsAction and before/after setConfiguringPolicy as your
convention dictates), and apply the same change to the analogous save handler
around lines 1177-1180 so both save-driven closes and cancel-driven closes emit
the same metric; reference functions: handleSaveParams,
updatePolicyParamsAction, setConfiguringPolicy, reload, and fireActionError to
locate where to add the telemetry call.

---

Nitpick comments:
In `@app/policies/hooks-client.tsx`:
- Around line 478-488: The state updater passed to setExpandedRow (the
setExpandedRow(prev => { ... }) block) performs side effects by calling
capture(...) inside the updater; compute the next state first (e.g., determine
const next = prev === idx ? null : idx outside the setState callback), call
capture(...) after computing next (using items[idx] for
decision/policyName/eventType), and then call setExpandedRow(next) so the
updater remains side-effect free; apply the same refactor to the duplicate block
around the 1052-1063 area.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: e3c33aa6-9851-4189-bedd-0c6e8e01a0e6

📥 Commits

Reviewing files that changed from the base of the PR and between ef306f0 and a43f253.

📒 Files selected for processing (10)

• CHANGELOG.md
• __tests__/hooks/new-telemetry.test.ts
• app/policies/hooks-client.tsx
• bin/failproofai.mjs
• scripts/postinstall.mjs
• src/hooks/custom-hooks-loader.ts
• src/hooks/handler.ts
• src/hooks/install-prompt.ts
• src/hooks/manager.ts
• src/hooks/policy-evaluator.ts