Skip to content

[finding] service-automation: FlowRunSummary.failed has no queryable column on sys_automation_run, so "which runs lost rows?" is a summary_json scan #15606

Description

@os-warren

Found while implementing #14456 (the contained-failure visibility contract). Out of scope there and deliberately not widened into — filing it instead.

What is there now

#14456 populates FlowRunSummary.failed (the fold failed = Sigma nodes[].failures) and persists it. It rides inside the summary_json blob on sys_automation_run, including on a summary compacted past the 16 KiB cap, where the totals are kept and the per-node detail is dropped.

The four sibling totals each have a column of their own — selected_count, acted_count, skipped_count, unmeasured_count — and packages/services/service-automation/src/sys-automation-run.object.ts states why, in a comment right above them:

// COLUMNS, not just a blob: `selected_count > 0 AND acted_count = 0` is the
// first FILTER of the broken-sweep detector, and an operator can only alert
// on what is filterable. Buried inside `summary_json` these would be
// readable but not queryable — the difference between a dashboard and an
// alarm.

failed has no such column.

Why that may matter

The premise of the containment work is that a run which caught a per-iteration failure completes GREEN: status = 'completed', and no other stored column moves. So "show me the runs that lost rows this week" is not answerable from the columns at all — it is a LIKE scan over summary_json, or nothing. That is the same "readable but not queryable" gap the comment above was written against, one field over.

It is not obviously the same shape as the broken-sweep filter, which is why this is a finding and not a defect:

  • unmeasured_count exists to QUALIFY acted_count — it is read as part of one filter expression, so it has to be in the same row as its operands.
  • failed would be its own predicate (failed_count > 0), which a caller could equally get by parsing the blob it already fetches.

So the question is a real one and this card does not presume the answer.

Options

  1. Add failed_count, mirroring unmeasured_count: null on rows written before the count existed (never 0 — absent means "not tracked", the convention the spec docblock spells out for both unmeasured and failed), written from record.summary?.failed ?? null beside its four siblings in ObjectStoreSuspendedRunStore.
  2. Leave it in the blob and say so in summary_json's own description, so the next reader does not re-derive the question.

Option 1 is a stored-surface change on an ADR-0103 engine-owned object, which is why it was not taken as a rider on #14456.

Not in scope of this card

Filed from claude/issue-14456-contained-failure-visibility.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions