Skip to content

fix(weave): call_end_v2 UPDATE on primary key, fix no-op - #7074

Merged
gtarpenning merged 2 commits into
masterfrom
fix-call-end-started-at
Jun 8, 2026
Merged

fix(weave): call_end_v2 UPDATE on primary key, fix no-op#7074
gtarpenning merged 2 commits into
masterfrom
fix-call-end-started-at

Conversation

@gtarpenning

@gtarpenning gtarpenning commented Jun 5, 2026

Copy link
Copy Markdown
Member

Jira: https://coreweave.atlassian.net/browse/WB-35267

Summary

  • call_end_v2 on a calls_complete project runs a lightweight UPDATE keyed on the client started_at; a wrong/omitted value matched 0 rows, leaving the call permanently unfinished while the endpoint returned 200.
  • Fast path keys an existence check + UPDATE on the full PK (project_id, started_at, id); fallback recovers the real started_at by (project_id, id) and raises NotFoundError (404) if the start is truly absent.

Performance

The extra read only runs on the eager call/end -> calls_complete path (_update_call_end_in_calls_complete). Last 7d in prod: that path fired 2,486 times vs ~135k v2 calls/complete writes -> ~1.8% of (calls/complete + call/end) hits, ~0.04% of all ~6.9M call ends. Added cost is one PK point lookup before the existing UPDATE. Local benchmark (1M-row single-project calls_complete clone, CH 26.3):

op p50 p95
added full-PK existence SELECT 3.4ms 5.8ms
existing lightweight UPDATE 5.3ms 7.5ms
added latency (+ms) +3.4ms +5.8ms
added latency (+%) +64% +78%

Testing

clickhouse-backed regression test: mismatched started_at still finishes, unknown id raises NotFoundError.

@wandbot-3000

wandbot-3000 Bot commented Jun 5, 2026

Copy link
Copy Markdown

@codecov

codecov Bot commented Jun 5, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@gtarpenning
gtarpenning force-pushed the fix-call-end-started-at branch from e126df8 to 8eb305e Compare June 5, 2026 23:15
@gtarpenning gtarpenning changed the title fix(weave): match call_end_v2 on (project_id, id), not client started_at fix(weave): read-before-update in call_end_v2 to fix silent no-op Jun 5, 2026
@gtarpenning
gtarpenning force-pushed the fix-call-end-started-at branch from 8eb305e to a4ca9b0 Compare June 5, 2026 23:32
@gtarpenning gtarpenning changed the title fix(weave): read-before-update in call_end_v2 to fix silent no-op fix(weave): key call_end_v2 UPDATE on the full primary key, fix silent no-op Jun 5, 2026
@gtarpenning
gtarpenning force-pushed the fix-call-end-started-at branch from a4ca9b0 to a6dd205 Compare June 8, 2026 15:06
…t no-op

call_end_v2 on a calls_complete project ran a lightweight UPDATE keyed on
the client-supplied started_at. A wrong/omitted started_at matched zero
rows, leaving the call permanently unfinished while the endpoint returned
200.

Fast path: weave_client sends the correct started_at, so confirm the row
with a full primary-key (project_id, started_at, id) point lookup, then
UPDATE on the same key. Only when the client value is missing or wrong do
we fall back to a (project_id, id) lookup to recover the real started_at,
and raise NotFoundError if the start truly doesn't exist.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@gtarpenning
gtarpenning force-pushed the fix-call-end-started-at branch from a6dd205 to 8053abc Compare June 8, 2026 15:28
@gtarpenning gtarpenning changed the title fix(weave): key call_end_v2 UPDATE on the full primary key, fix silent no-op fix(weave): key call_end_v2 UPDATE on primary key, fix silent no-op Jun 8, 2026
@gtarpenning gtarpenning changed the title fix(weave): key call_end_v2 UPDATE on primary key, fix silent no-op fix(weave): call_end_v2 UPDATE on primary key, fix silent no-op Jun 8, 2026
@gtarpenning gtarpenning changed the title fix(weave): call_end_v2 UPDATE on primary key, fix silent no-op fix(weave): call_end_v2 UPDATE on primary key, fix no-op Jun 8, 2026
@gtarpenning
gtarpenning marked this pull request as ready for review June 8, 2026 18:46
@gtarpenning
gtarpenning requested a review from a team as a code owner June 8, 2026 18:46

@jwlee64 jwlee64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why were the started_ats mismatching in the first place, It seems like that would be a fair assumption.

But i guess with the 404 we can surface how often this happens more in our alerting

@gtarpenning

Copy link
Copy Markdown
Member Author

Why were the started_ats mismatching in the first place, It seems like that would be a fair assumption.

But i guess with the 404 we can surface how often this happens more in our alerting

@jwlee64
this was discovered when claude wrote a smoke test that had a bug, was setting started_at on the call end to the call end ended_at for some reason, and nothing was working. I think this is a reasonable tradeoff, doesn't add much latency and always does the right thing.

@gtarpenning
gtarpenning merged commit cbc1868 into master Jun 8, 2026
339 of 343 checks passed
@gtarpenning
gtarpenning deleted the fix-call-end-started-at branch June 8, 2026 22:07
@github-actions github-actions Bot locked and limited conversation to collaborators Jun 8, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants