Fix: Go partition by partition instead of in one shot

[mrkaye97]

Description

Another attempt... looping over the partitions instead of doing it all in one shot so we can make incremental progress

Type of change

• Bug fix (non-breaking change which fixes an issue)

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
hatchet-docs	Ready	Preview, Comment	Apr 27, 2026 8:04pm

[Copilot]

Pull request overview

This PR updates the v1_0_99 OLAP mirror-table migration to backfill data incrementally by iterating partitions and tracking progress, rather than attempting a single large backfill operation.

Changes:

• Remove monolithic backfill SQL statements and replace them with a reusable backfillByPartition helper.
• Add a temporary v1_olap_backfill_progress table to record completed partitions and allow incremental/restartable progress.
• Consolidate post-backfill row-count verification into a single loop after the concurrent backfills complete.

Comments suppressed due to low confidence (1)

cmd/hatchet-migrate/migrate/migrations/20260424190713_v1_0_99.go:498

• v1RunsOlapTable is used here but not defined in this migration file; it currently relies on a constant declared in an older migration (20251226153622_v1_0_66.go). This creates unnecessary coupling between migrations and can lead to compile-time breakage if the older migration file is ever refactored/removed. Define v1RunsOlapTable in this file (or move shared constants/helpers into a non-dated shared file in the migrations package).

	eg.Go(func() error {
		if _, err := db.ExecContext(ctx, `DROP INDEX IF EXISTS ix_v1_runs_olap_tenant_id`); err != nil {
			return fmt.Errorf("drop old index on %s: %w", v1RunsOlapTable, err)
		}

		if _, err := db.ExecContext(ctx, buildCreateMirrorTableSQL(v1RunsOlapTable, v1RunsOlapTable+"_new", v1RunsOlapNewColDefs)); err != nil {
			return fmt.Errorf("create %s_new: %w", v1RunsOlapTable, err)