Skip to content

Add --skip-migrations flag to ETL runner#208

Merged
rickyrombo merged 1 commit intomainfrom
mjp-etl-skip-migrations
Apr 23, 2026
Merged

Add --skip-migrations flag to ETL runner#208
rickyrombo merged 1 commit intomainfrom
mjp-etl-skip-migrations

Conversation

@rickyrombo
Copy link
Copy Markdown
Contributor

Summary

  • Adds skipMigrations field and SetSkipMigrations() method to the ETL Indexer
  • Adds --skip-migrations CLI flag to examples/etl
  • When set, skips db.RunMigrations() on startup — useful when running against a restored production snapshot that already has all tables, avoiding slow CREATE INDEX IF NOT EXISTS on large tables

Test plan

  • Verified locally against a restored DP snapshot (107M-row blocks table) — ETL starts immediately without hanging on migration index creation

🤖 Generated with Claude Code

When running the ETL against a database that already has all tables
(e.g. a restored production snapshot), migrations can be skipped to
avoid long CREATE INDEX IF NOT EXISTS operations on large tables.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an opt-in way for the ETL Indexer (and the examples/etl runner) to skip running DB migrations on startup, which can significantly reduce startup time when pointing at an already-migrated/restored production snapshot.

Changes:

  • Add skipMigrations to etl.Indexer with a SetSkipMigrations() setter.
  • Gate db.RunMigrations() behind the new skipMigrations flag in Indexer.Run().
  • Add --skip-migrations flag to examples/etl and wire it into the indexer.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
pkg/etl/indexer.go Conditionally skips running ETL DB migrations during Run() based on skipMigrations.
pkg/etl/etl.go Adds skipMigrations state and a setter on the Indexer.
examples/etl/main.go Adds and wires --skip-migrations CLI flag for local ETL runner usage.
Comments suppressed due to low confidence (1)

examples/etl/main.go:78

  • --skip-migrations is applied via an if block and its value isn’t included in the startup log fields. For consistency with other boolean setters (and to make it obvious in logs when migrations are being skipped), consider calling indexer.SetSkipMigrations(*skipMigrations) unconditionally and adding a skip_migrations field to the logger.Info("starting ETL local runner", ...) call.
	if *skipMigrations {
		indexer.SetSkipMigrations(true)
	}

	logger.Info("starting ETL local runner",
		zap.String("rpc", *rpcURL),
		zap.String("db", *dbURL),
		zap.Int64("start_block", *startBlock),
	)

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread pkg/etl/indexer.go
@rickyrombo rickyrombo merged commit 478dd6f into main Apr 23, 2026
10 checks passed
@rickyrombo rickyrombo deleted the mjp-etl-skip-migrations branch April 23, 2026 23:34
Copilot AI mentioned this pull request Apr 23, 2026
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants