chore: add liquibase schema versioning by gracecluvohio · Pull Request #4401 · apache/texera

gracecluvohio · 2026-04-17T04:54:06Z

What changes were proposed in this PR?

This PR adds Liquibase to automate local database migrations. Running docker compose up will now automatically apply any pending SQL migrations under sql/updates and run sbt jooqGenerate, so developers no longer need to manually run SQL scripts after pulling changes.

NB:

The author of the PR with the schema change will need to add a <changeSet> element to changelog.xml so that Liquibase can detect the new update.
\c texera_db is a line that is being used in every sql update file under sql/updates. However, this is a psql command and not SQL, so Liquibase will throw an error. Authors should not add this line to their update scripts, since the database is already set to texera_db for Liquibase in docker-compose.yml.

Any related issues, documentation, discussions?

Closes #4400

How was this PR tested?

Locally

Was this PR authored or co-authored using generative AI tooling?

No

aglinxinyuan

Why is this under file-service folder?

gracecluvohio · 2026-04-17T05:48:32Z

Why is this under file-service folder?

I just wanted to keep it with the docker-compose.yml file. Not sure where else to put it. Would sql be a better folder?

aglinxinyuan · 2026-04-17T06:08:37Z

Why is this under file-service folder?

I just wanted to keep it with the docker-compose.yml file. Not sure where else to put it. Would sql be a better folder?

Yes. Please put it in the SQL folder as a separate docker-compose.yml file so the user can choose whether to use it. Please also fix the Apache header.

mengw15

I agree with Xinyuan's comments. Additionally, do not hard code the Postgres username and password, since its configurations in storage.conf, and devs may use different credentials. Also please double-check host.docker.internal's cross-platform support.

mengw15 · 2026-04-17T06:44:46Z

One question, do we need to use docker compose to do this task? is there other options？

gracecluvohio · 2026-04-17T07:48:25Z

One question, do we need to use docker compose to do this task? is there other options？

It is easiest for developers if it is integrated into docker compose so that developers don’t need to locally install Liquibase.

gracecluvohio · 2026-04-18T07:18:58Z

I agree with Xinyuan's comments. Additionally, do not hard code the Postgres username and password, since its configurations in storage.conf, and devs may use different credentials. Also please double-check host.docker.internal's cross-platform support.

I noticed that host.docker.internal is supported on all platforms except for Linux with Docker Engine/Standalone. Developers using this installation should run docker run --add-host=host.docker.internal:host-gateway liquibase/liquibase:4.29 instead.

gracecluvohio · 2026-04-18T19:21:38Z

I agree with Xinyuan's comments. Additionally, do not hard code the Postgres username and password, since its configurations in storage.conf, and devs may use different credentials. Also please double-check host.docker.internal's cross-platform support.

I don't think there's any straightforward way to avoid hard coding the postgres username and password. Since developers are already manually running the docker-compose.yml in \sql, I think we can leave it up to the developers to replace it with their postgres username and password.

aglinxinyuan

LGTM! Please work with @mengw15 to update the wiki. It's optional for user to use this, but it's required for developers who made DDL changes to add sql link in the xml file.

mengw15

LGTM. Left one minor comment

…e to custom user/password

gracecluvohio · 2026-04-24T04:59:44Z

It'd be nice to avoid confusion. The ultimate goal of this PR is for developers to not have to manually notify everyone that there was a DDL change. Developers can just run the docker compose file. But when they run the docker compose file without running sbt jooqGenerate, there will be compilation errors, which may lead to confusion.

I'm ok with this change.

Which change do you prefer? The jooqGenerate only when there is a schema change, or every time the docker compose file is run?

aglinxinyuan · 2026-04-24T05:01:29Z

It'd be nice to avoid confusion. The ultimate goal of this PR is for developers to not have to manually notify everyone that there was a DDL change. Developers can just run the docker compose file. But when they run the docker compose file without running sbt jooqGenerate, there will be compilation errors, which may lead to confusion.

I'm ok with this change.

Which change do you prefer? The jooqGenerate only when there is a schema change, or every time the docker compose file is run?

You can decide. For our main codebase, we will run jooqGenerate each time we compile.

gracecluvohio · 2026-04-25T07:04:37Z

It'd be nice to avoid confusion. The ultimate goal of this PR is for developers to not have to manually notify everyone that there was a DDL change. Developers can just run the docker compose file. But when they run the docker compose file without running sbt jooqGenerate, there will be compilation errors, which may lead to confusion.

I'm ok with this change.

Which change do you prefer? The jooqGenerate only when there is a schema change, or every time the docker compose file is run?

You can decide. For our main codebase, we will run jooqGenerate each time we compile.

Sounds good, I'll look into running jooqGenerate in the docker file.

chenlica · 2026-04-25T18:51:23Z

@mengw15 @aglinxinyuan Feel free to merge it if you think it's ready.

gracecluvohio · 2026-04-26T19:08:49Z

@mengw15 @aglinxinyuan I've added the sbt jooqGenerate service to sql/docker-compose.yml, please take a look.

…pts to maintain old convention

mengw15

Left some comments, please also test after resolve these comments.

…e name

aglinxinyuan

LGTM!

### What changes were proposed in this PR?  This PR adds Liquibase to automate local database migrations. Running `docker compose up` will now automatically apply any pending SQL migrations under `sql/updates` and run `sbt jooqGenerate`, so developers no longer need to manually run SQL scripts after pulling changes. NB: - The author of the PR with the schema change will need to add a `<changeSet>` element to `changelog.xml` so that Liquibase can detect the new update. - `\c texera_db` is a line that is being used in every sql update file under `sql/updates`. However, this is a psql command and not SQL, so Liquibase will throw an error. Authors should not add this line to their update scripts, since the database is already set to `texera_db` for Liquibase in `docker-compose.yml`. ### Any related issues, documentation, discussions?  Closes apache#4400 ### How was this PR tested?  Locally ### Was this PR authored or co-authored using generative AI tooling?  No --------- Co-authored-by: Chen Li <chenli@gmail.com> Co-authored-by: Xinyuan Lin <xinyual3@uci.edu> Co-authored-by: Meng Wang <125719918+mengw15@users.noreply.github.com> Co-authored-by: Meng Wang <simonmengwang123@gmail.com>

…ordance with PR apache#4401

ci: add liquibase schema versioning

5c73142

github-actions Bot added the service label Apr 17, 2026

Merge branch 'main' into add-liquidbase

b04def9

chenlica requested review from aglinxinyuan and mengw15 April 17, 2026 05:26

aglinxinyuan requested changes Apr 17, 2026

View reviewed changes

Merge branch 'main' into add-liquidbase

48dec76

mengw15 reviewed Apr 17, 2026

View reviewed changes

Merge branch 'main' into add-liquidbase

ed26d6e

chenlica and others added 3 commits April 17, 2026 09:14

Merge branch 'main' into add-liquidbase

21d9973

Merge branch 'main' into add-liquidbase

e248d3d

ci: move changelog and docker compose to sql folder

a6f146c

github-actions Bot removed the service label Apr 18, 2026

aglinxinyuan assigned gracecluvohio Apr 18, 2026

aglinxinyuan added the ddl-change Changes to the TexeraDB DDL label Apr 18, 2026

aglinxinyuan changed the title ~~ci: add liquibase schema versioning~~ chore: add liquibase schema versioning Apr 18, 2026

gracecluvohio requested review from aglinxinyuan and mengw15 April 18, 2026 19:21

Merge branch 'main' into add-liquidbase

9bed408

github-actions Bot removed the ddl-change Changes to the TexeraDB DDL label Apr 18, 2026

aglinxinyuan approved these changes Apr 18, 2026

View reviewed changes

mengw15 approved these changes Apr 18, 2026

View reviewed changes

Comment thread sql/docker-compose.yml Outdated

gracecluvohio added 2 commits April 18, 2026 16:05

ci: change postgres user/password to default and add comment to chang…

5cb15ff

…e to custom user/password

Merge remote-tracking branch 'origin/add-liquidbase' into add-liquidbase

dd37ea8

Merge branch 'main' into add-liquidbase

d822f0c

mengw15 and others added 3 commits April 26, 2026 01:07

Merge branch 'main' into add-liquidbase

6f35679

feat: add sbt jooqGenerate to sql/docker-compose.yml

2f18e00

Merge remote-tracking branch 'origin/add-liquidbase' into add-liquidbase

eaf1765

gracecluvohio requested review from aglinxinyuan and mengw15 April 26, 2026 19:08

mengw15 and others added 5 commits April 27, 2026 14:04

Merge branch 'main' into add-liquidbase

619e90f

feat: strip \c texera_db and SET search_path lines in sql update scri…

7d993e6

…pts to maintain old convention

Merge remote-tracking branch 'origin/add-liquidbase' into add-liquidbase

517522f

Merge branch 'main' into add-liquidbase

a703d8e

Merge branch 'main' into add-liquidbase

7361638

mengw15 reviewed May 1, 2026

View reviewed changes

Comment thread sql/docker-compose.yml Outdated

Comment thread sql/docker-compose.yml Outdated

Comment thread sql/docker-compose.yml Outdated

Comment thread sql/docker-compose.yml Outdated

mengw15 and others added 6 commits May 1, 2026 13:15

Merge branch 'main' into add-liquidbase

96ddc11

refactor: fix volume paths for jooq service in docker compose

390a7eb

Merge remote-tracking branch 'origin/add-liquidbase' into add-liquidbase

4316ec7

refactor: remove clearCheckSums from liquibase service in docker compose

752125f

refactor: make for loop in sql docker compose robust to spaces in fil…

4eb5ff8

…e name

Merge branch 'main' into add-liquidbase

247ddb3

aglinxinyuan approved these changes May 2, 2026

View reviewed changes

Merge branch 'main' into add-liquidbase

b049755

mengw15 enabled auto-merge (squash) May 2, 2026 02:39

mengw15 merged commit fa376d0 into apache:main May 2, 2026
15 checks passed

mengw15 mentioned this pull request May 18, 2026

feat(python-notebook-migration): add database tables for Notebook Migration tool #5055

Open

zyratlo added a commit to zyratlo/texera that referenced this pull request May 18, 2026

removed "\c texera_db" in sql file and added changeSet element in acc…

9c63915

…ordance with PR apache#4401

Conversation

gracecluvohio commented Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this PR?

Any related issues, documentation, discussions?

How was this PR tested?

Was this PR authored or co-authored using generative AI tooling?

Uh oh!

aglinxinyuan left a comment

Choose a reason for hiding this comment

Uh oh!

gracecluvohio commented Apr 17, 2026

Uh oh!

aglinxinyuan commented Apr 17, 2026

Uh oh!

mengw15 left a comment

Choose a reason for hiding this comment

Uh oh!

mengw15 commented Apr 17, 2026

Uh oh!

gracecluvohio commented Apr 17, 2026

Uh oh!

gracecluvohio commented Apr 18, 2026

Uh oh!

gracecluvohio commented Apr 18, 2026

Uh oh!

aglinxinyuan left a comment

Choose a reason for hiding this comment

Uh oh!

mengw15 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gracecluvohio commented Apr 24, 2026

Uh oh!

aglinxinyuan commented Apr 24, 2026

Uh oh!

gracecluvohio commented Apr 25, 2026

Uh oh!

chenlica commented Apr 25, 2026

Uh oh!

gracecluvohio commented Apr 26, 2026

Uh oh!

mengw15 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aglinxinyuan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

gracecluvohio commented Apr 17, 2026 •

edited

Loading