Skip to content

docs(remediation): correct why two chain claims had no DB row - #65

Merged
lucca65 merged 1 commit into
masterfrom
fix/remediation-backfill-notes
Aug 8, 2026
Merged

docs(remediation): correct why two chain claims had no DB row#65
lucca65 merged 1 commit into
masterfrom
fix/remediation-backfill-notes

Conversation

@lucca65

@lucca65 lucca65 commented Aug 8, 2026

Copy link
Copy Markdown
Member

Follow-up to #61, from running the remediation on prod today.

The cause was misattributed. The generator blamed claimAction's (created_tx, action_id, claimer_id) dedup guard collapsing two claimactions sent in one transaction. Checked against /v1/history/get_actions: both cases (chain 19871 and 19904) are two separate transactions in the same block — same claimer, same action, identical proof_photo, i.e. a double submit. Different created_tx, so that guard never matched them.

The likelier mechanism is the old serial fallback: a truncated chain read made the resolver give up, the serial it fell back to collided with an id already taken, and the insert's .catch(e => logError(...)) swallowed the failure so the row vanished without stopping the block. #61 removes the fallback.

Operational note added. The backfill INSERT leaves created_block/created_tx/created_eos_account NULL, and the GraphQL :claim type marks them non_null — one NULL row nullifies a validator's whole claims list in the Elm app. That fired on prod today; scripts/prod-anomaly-check.sh caught it immediately after the remediation. Both rows were backfilled from the history API and the check is clean again.

Comment-only; no behaviour change.

The generator blamed claimAction's (created_tx, action_id, claimer_id) dedup
guard collapsing two claimactions sent in one transaction. Checked against
/v1/history/get_actions: both cases (chain 19871 and 19904) are two SEPARATE
transactions that landed in the SAME block, same claimer, same action,
identical proof_photo — a double submit. Different created_tx, so that guard
never matched them and is not the cause.

The likelier mechanism is the old serial fallback: a truncated chain read made
the resolver give up, the serial it fell back to collided with an id already
taken, and the insert's `.catch(e => logError(...))` swallowed the failure so
the row disappeared without stopping the block. Dropping the fallback closes
that path.

Also records that the backfilled rows leave created_block/created_tx/
created_eos_account NULL and must be filled from the history API afterwards:
the GraphQL :claim type marks them non_null, so a single NULL row nullifies a
validator's entire claims list in the Elm app. Hit exactly that on prod today
after running the remediation; both rows have since been backfilled from
/v1/history/get_actions and prod-anomaly-check.sh is clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@lucca65
lucca65 merged commit d8eb265 into master Aug 8, 2026
2 checks passed
@lucca65
lucca65 deleted the fix/remediation-backfill-notes branch August 8, 2026 12:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant