fix(engine): scheduled workflow idempotency insert conflict

[mrkaye97]

Description

Seeing some of these 2026-07-17T12:30:53.699Z ERR could not run scheduled workflow error="could not claim idempotency key for scheduled workflow: ERROR: duplicate key value violates unique constraint \"v1_idempotency_key_pkey\" (SQLSTATE 23505)" service=ticker errors on cloud

Pretty sure the issue here is that we're attempting two concurrent inserts. I think the root of the problem was using SKIP LOCKED to see which rows to lock, since if a row was locked we'd end up trying to do another insert, which would then fail. Removed that. Then I also updated the schema to change the PK so we don't have overlapping indexes, which I think should also help

Then there was one other issue too, which was that we could spawn duplicate runs if this method was called multiple times since the claimed_by_external_id was set upstream (we were using opts.Id) and instead, we should use a unique id so the run's external id is the claimant, not the external id of the schedule.

Also added a regression test which asserts we only pub to the MQ once

Type of change

• Bug fix (non-breaking change which fixes an issue)

Checklist

Changes have been:

• Tested (unit, integration, or manually with steps specified)
• Linted and formatted
• Documented (where applicable)
• Added to CHANGELOG (where applicable) -- see Keep a Changelog

---

🤖 AI Disclosure

No AI use

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
hatchet-docs	Ready	Preview, Comment	Jul 17, 2026 3:11pm

[Copilot]

Pull request overview

Fixes a concurrency bug in scheduled workflow execution by making idempotency key claiming safe under concurrent callers, and aligning the database schema/queries to avoid insert conflicts.

Changes:

• Update idempotency key schema to use (tenant_id, key) as the primary key and add an index to support expiry cleanup.
• Remove SKIP LOCKED from the idempotency claim query to prevent concurrent “missing row” paths that can lead to duplicate inserts.
• Ensure scheduled workflow runs use a unique run external ID as the claimant, and add a regression test to verify only one MQ publish occurs under concurrency.

Reviewed changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
sql/schema/v1-core.sql	Adjusts v1_idempotency_key primary key and adds an expiry index in the canonical schema.
pkg/repository/sqlcv1/idempotency-keys.sql.go	Regenerates SQLC output to remove SKIP LOCKED from the locking query.
pkg/repository/sqlcv1/idempotency-keys.sql	Updates the source SQL used by SQLC to remove SKIP LOCKED.
pkg/repository/idempotency.go	Adds a constructor used by the new ticker test to build an idempotency repo from a pool.
internal/services/ticker/schedule_workflow_v1.go	Uses a per-run UUID as the claimant/external ID to prevent duplicate runs across repeated calls.
internal/services/ticker/schedule_workflow_v1_test.go	Adds a concurrency regression test asserting only one message is published.
cmd/hatchet-migrate/migrate/migrations/20260717150345_v1_0_131.sql	Migration to shift the PK to (tenant_id, key) and add the expiry index.

Files not reviewed (1)

• pkg/repository/sqlcv1/idempotency-keys.sql.go: Generated file