[V2] Replace GORM with sqlx for database operations by pingsutw · Pull Request #7160 · flyteorg/flyte

pingsutw · 2026-04-06T19:34:46Z

Tracking issue

N/A

Why are the changes needed?

GORM adds ORM abstraction overhead and implicit query generation that makes it harder to reason about database behavior. Replacing it with sqlx gives us:

Explicit SQL queries — no hidden query generation or N+1 surprises
Lighter dependency footprint — sqlx is a thin wrapper over database/sql
Better control over PostgreSQL-specific features (ON CONFLICT, RETURNING, etc.)
Simpler migration system using plain SQL files instead of struct-tag-driven AutoMigrate

What changes were proposed in this pull request?

Complete migration from gorm.io/gorm to github.com/jmoiron/sqlx across the entire V2 codebase:

Database Foundation (`flytestdlib/database/`)

GetDB() / GetReadOnlyDB() now return *sqlx.DB instead of *gorm.DB
PostgreSQL connections use sqlx.Open("pgx", dsn) with the pgx stdlib driver
New custom SQL migration runner (migrate.go) that reads numbered .sql files from embed.FS
Deleted gorm.go (GORM logger adapter)

Models

Replaced all gorm:"..." struct tags with db:"..." tags for sqlx field mapping
Removed all TableName() methods — table names are now explicit in SQL queries

Repository Implementations

All GORM query builder chains replaced with explicit SQL via sqlx.GetContext, sqlx.SelectContext, db.ExecContext
gorm.ErrRecordNotFound → sql.ErrNoRows throughout
gorm.ErrDuplicatedKey → custom ErrAlreadyExists sentinel
clause.OnConflict{DoNothing: true} → INSERT ... ON CONFLICT DO NOTHING
clause.Returning{} → RETURNING SQL clauses
LISTEN/NOTIFY: DSN stored directly instead of extracting from GORM dialector
Renamed filter/sort interfaces: GormQueryExpression → QueryExpression, GetGormOrderExpr → GetOrderExpr

Migrations

Replaced gormigrate with plain SQL migration files under sql/ directories
runs/migrations/sql/001_init_schema.sql — full runs service schema
cache_service/migrations/sql/001_init_schema.sql — cache service schema
Each service embeds its SQL directory and exposes RunMigrations(ctx, db)

App & Service Layer

app/context.go: DB *gorm.DB → DB *sqlx.DB
Setup files use migrations.RunMigrations() instead of gormigrate
Service layer error checks use sql.ErrNoRows
All test files updated

Dependencies

Added: github.com/jmoiron/sqlx, github.com/jackc/pgx/v5/stdlib
Removed: gorm.io/gorm, gorm.io/driver/postgres, github.com/go-gormigrate/gormigrate/v2

How was this patch tested?

go build ./... passes cleanly
go vet ./... passes
Repository unit tests pass (go test ./runs/repository/impl/... — 41 tests)
Zero remaining GORM imports in any .go file
Only gorm.io text remaining: documentation strings in auto-generated flag help text

Labels

changed: Migration from GORM ORM to sqlx for all database operations
removed: GORM, gorm driver, gormigrate dependencies

Check all the applicable boxes

I updated the documentation accordingly.
All new and existing tests passed.
All commits are signed-off.

main
- Flyte 2 #6583
  - [V2] Replace GORM with sqlx for database operations 👈

…ache services Replace GORM-based AutoMigrate with explicit SQL schema files and embed.FS-based migration wrappers that use the new database.Migrate() function from flytestdlib. Signed-off-by: Kevin Su <pingsutw@apache.org>

Remove all gorm:"..." tag portions and TableName() methods from model structs. The db:"..." tags (used by sqlx) are retained. For project.go, add db tags that were previously missing. Table names will now live in SQL queries rather than on the model. Signed-off-by: Kevin Su <pingsutw@apache.org>

Replace gorm.DB with sqlx.DB throughout the database foundation layer. This removes the GORM dependency from the core database package in favor of the lighter-weight sqlx + pgx driver combination. - Rewrite db.go: GetDB/GetReadOnlyDB return *sqlx.DB, remove logConfig param - Rewrite postgres.go: use sqlx.Open("pgx", dsn) instead of gorm.Open - Create migrate.go: custom SQL migration runner using schema_migrations table - Rewrite embedded_postgres_test_helper.go: use sqlx instead of gorm - Delete gorm.go: GORM logger adapter no longer needed - Add github.com/jmoiron/sqlx dependency Signed-off-by: Kevin Su <pingsutw@apache.org>

Replace GORM-based cache service repository implementations with raw sqlx queries. Update error handling to use database/sql.ErrNoRows and a new ErrAlreadyExists sentinel instead of GORM-specific errors. Signed-off-by: Kevin Su <pingsutw@apache.org>

Replace all GORM-based database operations in the runs repository layer with raw SQL via sqlx. This removes the GORM runtime dependency from the runs data access layer while preserving identical behavior. - Rename GormQueryExpr/GormQueryExpression/GetGormOrderExpr to QueryExpr/QueryExpression/GetOrderExpr across interfaces, impls, mocks, and tests - Rewrite project.go, task.go, action.go to use sqlx.DB with $N placeholders and raw SQL (INSERT, SELECT, UPDATE with RETURNING) - Add rewritePlaceholders helper to convert ? to $N for PostgreSQL - Store DSN in actionRepo for pq.Listener instead of extracting from GORM dialector - Add RunSqlxTestMain helper for embedded postgres test infrastructure - Add db tags to Project model for sqlx compatibility - Update all test files to use *sqlx.DB Signed-off-by: Kevin Su <pingsutw@apache.org>

Signed-off-by: Kevin Su <pingsutw@apache.org>

Replace all remaining GORM references with sqlx across the app wiring, service layer, setup files, and test files. Remove gorm, gorm-postgres-driver, and gormigrate dependencies from go.mod. Changes: - app/db.go: InitDB returns *sqlx.DB, calls GetDB without logConfig - app/context.go: SetupContext.DB field is now *sqlx.DB - runs/setup.go: Use migrations.RunMigrations(), simplify ready check - cache_service/setup.go: Same migration and ready check updates - cache_service/service/service.go: NewCacheService takes *sqlx.DB - runs/service/run_logs_service.go: Use sql.ErrNoRows instead of gorm.ErrRecordNotFound - runs/service/run_logs_service_test.go: Match sql.ErrNoRows - All test main_test.go files: Use database.RunTestMain with sqlx callbacks - runs/test/api/setup_test.go: Use sqlx.DB, GetDB without logConfig, RunMigrations - go.mod: Remove gorm.io/gorm, gorm.io/driver/postgres, go-gormigrate/gormigrate Signed-off-by: Kevin Su <pingsutw@apache.org>

PostgreSQL requires `= ANY($1)` with pq.Array() for parameterized IN clauses, not `IN $1`. Also regenerate mocks to match renamed interfaces. Signed-off-by: Kevin Su <pingsutw@apache.org>

Update TestBasicFilter_ValueIn and TestParseStringFilters_ValueInState to expect the PostgreSQL-compatible `= ANY(?)` syntax with pq.Array instead of `IN ?`. Signed-off-by: Kevin Su <pingsutw@apache.org>

1. Add prefix parameter to Migrate/Rollback so each service records versions as "runs/001_init_schema" and "cache_service/001_init_schema" instead of bare "001_init_schema". This prevents collisions when multiple services share the same database (unified manager mode). 2. Change all TIMESTAMP columns to TIMESTAMPTZ in migration SQL files. pgx round-trips Go time.Time values through PostgreSQL, and bare TIMESTAMP (without time zone) silently strips the timezone on write then assumes UTC on read, causing offsets in non-UTC environments. TIMESTAMPTZ preserves the timezone correctly. Signed-off-by: Kevin Su <pingsutw@apache.org>

The previous replace-all of TIMESTAMP→TIMESTAMPTZ accidentally also converted CURRENT_TIMESTAMP→CURRENT_TIMESTAMPTZ which doesn't exist in PostgreSQL. CURRENT_TIMESTAMP works correctly with TIMESTAMPTZ columns. Signed-off-by: Kevin Su <pingsutw@apache.org>

- Accept v2's move of app/ -> flytestdlib/app/ and update imports - Remove unused app/db.go (InitDB was dead code) - Fix database.GetDB calls in cmd/ binaries (remove logger.GetConfig arg) Signed-off-by: Kevin Su <pingsutw@apache.org>

Signed-off-by: Kevin Su <pingsutw@apache.org>

machichima · 2026-04-08T03:02:46Z

Should we add 001_init_schema_down.sql alongside with all our .sql file?
Also, could we use datetime as prefix? e.g. 20260408110000

machichima · 2026-04-08T03:06:04Z

+			return fmt.Errorf("failed to read migration file %s: %w", f, err)
+		}
+
+		tx, err := db.BeginTxx(ctx, nil)


Coud we add defer tx.Rollback() here to ensure the transaction is closed when Commit failed

If BeginTxx returns an error, no transaction was created

machichima · 2026-04-08T03:12:13Z

+
+		var countWhere []string
+		if input.Filter != nil {
+			expr, _ := input.Filter.QueryExpression("")


Should we capture the error here and return it? Like how we did in other function

we skipped the error before, so Claude skipped it too. Update it now

Signed-off-by: Kevin Su <pingsutw@apache.org>

popojk

LGTM now! thanks!

pingsutw added 10 commits April 6, 2026 09:57

Merge Task 2: Models (gorm tags → db tags)

acd8bc4

Merge Task 5: Migration SQL files

9dfb6ff

Merge Task 3: Runs Repository (GORM → sqlx) - resolve conflicts

1ac46c2

Signed-off-by: Kevin Su <pingsutw@apache.org>

Merge Task 4: Cache Repository (GORM → sqlx)

cee190f

Signed-off-by: Kevin Su <pingsutw@apache.org>

github-actions bot mentioned this pull request Apr 6, 2026

Flyte 2 #6583

Draft

3 tasks

pingsutw marked this pull request as draft April 6, 2026 19:43

pingsutw added 2 commits April 6, 2026 12:49

fix: use ANY() for IN filters and regenerate mocks

8fbce5f

PostgreSQL requires `= ANY($1)` with pq.Array() for parameterized IN clauses, not `IN $1`. Also regenerate mocks to match renamed interfaces. Signed-off-by: Kevin Su <pingsutw@apache.org>

fix: update filter tests for ANY() syntax

ce8eeb5

Update TestBasicFilter_ValueIn and TestParseStringFilters_ValueInState to expect the PostgreSQL-compatible `= ANY(?)` syntax with pq.Array instead of `IN ?`. Signed-off-by: Kevin Su <pingsutw@apache.org>

pingsutw marked this pull request as ready for review April 6, 2026 22:09

pingsutw added 3 commits April 6, 2026 15:24

machichima reviewed Apr 8, 2026

View reviewed changes

Comment thread flytestdlib/database/migrate.go

Comment thread runs/repository/models/action.go Outdated

machichima reviewed Apr 8, 2026

View reviewed changes

Comment thread flytestdlib/database/postgres.go

address-comment

3902106

Signed-off-by: Kevin Su <pingsutw@apache.org>

machichima reviewed Apr 8, 2026

View reviewed changes

popojk reviewed Apr 8, 2026

View reviewed changes

Comment thread runs/repository/impl/action.go

popojk reviewed Apr 8, 2026

View reviewed changes

Comment thread runs/repository/impl/action.go Outdated

address comment

26d52d7

Signed-off-by: Kevin Su <pingsutw@apache.org>

popojk reviewed Apr 8, 2026

View reviewed changes

Comment thread runs/repository/impl/filters.go Outdated

AdilFayyaz approved these changes Apr 8, 2026

View reviewed changes

address comment

aa94462

Signed-off-by: Kevin Su <pingsutw@apache.org>

popojk approved these changes Apr 8, 2026

View reviewed changes

pingsutw merged commit c11628a into v2 Apr 8, 2026
17 checks passed

pingsutw deleted the kevin/gorm-to-sqlx branch April 8, 2026 06:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[V2] Replace GORM with sqlx for database operations#7160

[V2] Replace GORM with sqlx for database operations#7160
pingsutw merged 18 commits intov2from
kevin/gorm-to-sqlx

pingsutw commented Apr 6, 2026 •

edited by github-actions bot

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

machichima commented Apr 8, 2026

Uh oh!

machichima Apr 8, 2026

Uh oh!

pingsutw Apr 8, 2026

Uh oh!

machichima Apr 8, 2026

Uh oh!

pingsutw Apr 8, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

popojk left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

pingsutw commented Apr 6, 2026 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tracking issue

Why are the changes needed?

What changes were proposed in this pull request?

Database Foundation (flytestdlib/database/)

Models

Repository Implementations

Migrations

App & Service Layer

Dependencies

How was this patch tested?

Labels

Check all the applicable boxes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

machichima commented Apr 8, 2026

Uh oh!

machichima Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

pingsutw Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

machichima Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

pingsutw Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

popojk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pingsutw commented Apr 6, 2026 •

edited by github-actions bot

Loading

Database Foundation (`flytestdlib/database/`)