feat: goal tracking in observability — cost per goal progressed#636
feat: goal tracking in observability — cost per goal progressed#636kokevidaurre merged 1 commit intodevelopfrom
Conversation
Every execution now captures: - goals_before: goal name → status before the run - goals_after: goal name → status after the run - goals_changed: list of goals that moved (e.g., not-started → in-progress) Org cycle (squads run --org) shows goal changes summary at the end. obs history shows goal changes per execution record. Also added grade/grade_score fields to ObservabilityRecord for future COO eval integration. This enables measuring: cost per goal progressed — the key metric for self-improving autonomous operation. Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
Code Review
This pull request introduces goal tracking and observability improvements by adding functionality to snapshot and diff goals from 'goals.md' files before and after execution cycles. The changes include new utility functions in 'observability.ts', updates to the execution engine to capture these snapshots, and enhanced reporting in the CLI. I have provided a suggestion to simplify the 'diffGoals' function by using a single loop over a combined set of goal names for better efficiency and readability.
| const changes: GoalChange[] = []; | ||
|
|
||
| for (const [name, afterStatus] of Object.entries(after)) { | ||
| const beforeStatus = before[name] || 'new'; | ||
| if (beforeStatus !== afterStatus) { | ||
| changes.push({ name, before: beforeStatus, after: afterStatus }); | ||
| } | ||
| } | ||
|
|
||
| // Goals that disappeared (moved to achieved/abandoned) | ||
| for (const [name, beforeStatus] of Object.entries(before)) { | ||
| if (!(name in after)) { | ||
| changes.push({ name, before: beforeStatus, after: 'removed' }); | ||
| } | ||
| } | ||
|
|
||
| return changes; |
There was a problem hiding this comment.
The body of this function can be simplified to use a single loop, which is more efficient and easier to read. By creating a combined set of all goal names from both before and after snapshots, you can check for differences in a single pass.
| const changes: GoalChange[] = []; | |
| for (const [name, afterStatus] of Object.entries(after)) { | |
| const beforeStatus = before[name] || 'new'; | |
| if (beforeStatus !== afterStatus) { | |
| changes.push({ name, before: beforeStatus, after: afterStatus }); | |
| } | |
| } | |
| // Goals that disappeared (moved to achieved/abandoned) | |
| for (const [name, beforeStatus] of Object.entries(before)) { | |
| if (!(name in after)) { | |
| changes.push({ name, before: beforeStatus, after: 'removed' }); | |
| } | |
| } | |
| return changes; | |
| const changes: GoalChange[] = []; | |
| const allNames = new Set([...Object.keys(before), ...Object.keys(after)]); | |
| for (const name of allNames) { | |
| const beforeStatus = before[name]; | |
| const afterStatus = after[name]; | |
| if (beforeStatus !== afterStatus) { | |
| changes.push({ | |
| name, | |
| before: beforeStatus ?? 'new', | |
| after: afterStatus ?? 'removed', | |
| }); | |
| } | |
| } | |
| return changes; |
Every run snapshots goals before/after, diffs changes. Org cycle shows goal summary. Enables measuring cost per goal progressed.