Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flyway / DevServices #19766

Closed
pjgg opened this issue Aug 30, 2021 · 9 comments · Fixed by #40313
Closed

Flyway / DevServices #19766

pjgg opened this issue Aug 30, 2021 · 9 comments · Fixed by #40313

Comments

@pjgg
Copy link
Contributor

pjgg commented Aug 30, 2021

Describe the bug

I am working with reactive Postgresql datasource and Flyway and I couldn't launch my app because I got the following error:

Failed to start application (with profile dev): org.postgresql.util.PSQLException: FATAL: database "store" does not exist

My configuration is the following one:

quarkus.datasource.db-kind=postgresql
quarkus.datasource.devservices.enabled=true
quarkus.datasource.devservices.port=5432
quarkus.datasource.reactive.url=postgresql://localhost:5432/store

## Flyway
quarkus.flyway.migrate-at-start=true
quarkus.flyway.schemas=book
quarkus.datasource.jdbc.url=jdbc:postgresql://localhost:5432/store
quarkus.flyway.locations=db

The problem is that flyway expects that store DB already exists, but doesn't because the container is instantiated by devService, and any other workaround would be an over-engineering solution, which is the opposite of devServices.

Also, flyway requires the DB port, this could be handler by adding quarkus.datasource.devservices.port property but would be great to handler this without fixed ports.

Proposal solution:

We could create a new (maybe optional) devServices property

quarkus.datasource.devservices.db-name that could be used by:

  1. quarkus.datasource.jdbc.url required by flyway

The port issue should be handler internally by the propagation of the port to the different places where is used. In case that end-user provides quarkus.datasource.devservices.db-name and quarkus.datasource.jdbc.url then db-name will be used as a database name in Dev mode.

@pjgg pjgg added the kind/bug Something isn't working label Aug 30, 2021
@quarkus-bot
Copy link

quarkus-bot bot commented Aug 30, 2021

@agreedSkiing
Copy link
Contributor

agreedSkiing commented Feb 4, 2024

Sorry this might be hijacking the enhancement but, another solution could be that if a reactive datasource exists override the connection if it has not been defined and replace the prefix vertx-reactive: with jdbc: so that flyway use the same container.

Based on the #33790 and #36012 which currently causes two devserivces to start, where one is for the reactive datasource and the other for the blocking 😅.

@agreedSkiing
Copy link
Contributor

Redacting my old comment with this one, a property that assigns that has the database devserice port assigned to it would be enought then I could just config quarkus.datasource.blocking.jdbc.url = jdbc:postgresql://localhost:${quarkus.datasource.devservices.assigned.port:5432}/quarkus.

@yrodiere
Copy link
Member

yrodiere commented Apr 22, 2024

Just noticed this issue... Which doesn't appear to be a flyway issue at all?

Regarding the datasource, yes, you need to use a different datasource for reactive and for JDBC. => Wrong, see end of the conversation and pull request.

Regarding the "port" problem.... At first I was puzzled by the fact you were setting reactive.url/jdbc.url, while simultaneously wanting to use dev services (which would set these properties automatically).

Then I got it: you want the two datasources to be backed by the same DB, and this is not what would happen if you were to use devservices on both datasources. So you tried to have devservices launched for one datasource, while having the other datasource point to the DB without launching a devservice. Is that right?

If so, I think what you need is the concept of "shared" dev services, like here for AMQP, but unfortunately it is not implemented for datasources at the moment.

If it was implemented, you could do something like this:

quarkus.datasource.devservices.shared=true
quarkus.datasource.devservices.service-name=store
# This is only necessary in prod, as we use dev services for dev/tests 
%prod.quarkus.datasource.reactive.url=postgresql://localhost:5432/store

## Flyway
quarkus.datasource.blocking.devservices.shared=true
quarkus.datasource.blocking.devservices.service-name=store
# This is only necessary in prod, as we use dev services for dev/tests 
%prod.quarkus.datasource.blocking.jdbc.url=jdbc:postgresql://localhost:5432/store
quarkus.flyway.blocking.migrate-at-start=true
quarkus.flyway.blocking.schemas=book
quarkus.flyway.blocking.locations=db

@geoand can you confirm this could work? Or would I be grossly misusing the concept of "shared" dev services?

Note:

  • quarkus.datasource.db-kind=postgresql is not necessary if you have only the postgresql extension in your classpath
  • quarkus.datasource.devservices.enabled=true is not necessary as this is the default

That being said... Everything would probably be much easier if you could just make a single datasource handle both reactive and JDBC.

@geoand
Copy link
Contributor

geoand commented Apr 23, 2024

@geoand can you confirm this could work? Or would I be grossly misusing the concept of "shared" dev services?

I am not really sure it would work as I anticipate there is no good way for the two different actions that are going to launch the devservice to coordinate.

FWIW, shared was meant to refer to two different application sharing the same resource. That's not to say that this case should not be solved somehow, I am just not sure if the same concept applies.

@yrodiere
Copy link
Member

yrodiere commented Apr 24, 2024

Thanks @geoand , so indeed this would be abusing the feature.

@gsmet told me that using a single datasource for both jdbc and reactive should actually work... I remember differently, but if this used to work at some point, it might be easy to fix. @pjgg could you try something like this, with a single datasource for both jdbc and reactive?

%prod.quarkus.datasource.reactive.url=postgresql://localhost:5432/store
%prod.quarkus.datasource.jdbc.url=jdbc:postgresql://localhost:5432/store

quarkus.flyway.migrate-at-start=true
quarkus.flyway.schemas=book
quarkus.flyway.locations=db

If the above does not work, you can alternatively try this:

quarkus.datasource.jdbc=true
quarkus.datasource.reactive=true
%prod.quarkus.datasource.reactive.url=postgresql://localhost:5432/store
%prod.quarkus.datasource.jdbc.url=jdbc:postgresql://localhost:5432/store

quarkus.flyway.migrate-at-start=true
quarkus.flyway.schemas=book
quarkus.flyway.locations=db

@agreedSkiing
Copy link
Contributor

agreedSkiing commented Apr 26, 2024

Been following this issue so i gave it a try instead :)

quarkus.datasource.jdbc = true
quarkus.datasource.db-kind = postgresql
quarkus.datasource.reactive.url = vertx-reactive:postgresql://localhost:5433/quarkus
quarkus.datasource.reactive.max-size = 50
quarkus.datasource.username = user
quarkus.datasource.password = pass
quarkus.flyway.clean-at-start = true
quarkus.flyway.locations = db/migration,filesystem:target/test-classes/db/migration
quarkus.flyway.migrate-at-start = true
quarkus.flyway.baseline-on-migrate = true

quarkus.datasource.devservices.enabled = true
quarkus.datasource.devservices.image-name = postgres:13
quarkus.datasource.devservices.reuse = false
quarkus.datasource.blocking.devservices.enabled = false

And is worked

> mvn quarkus:dev
[INFO] Scanning for projects...
[INFO] 
[INFO] --------------------------< org.acme:my-app >---------------------------
[INFO] Building my-app 4.0.0-SNAPSHOT
[INFO]   from pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- quarkus:3.8.1:dev (default-cli) @ my-app ---
[INFO] Invoking jacoco:0.8.11:prepare-agent (default) @ my-app
[INFO] argLine set to -javaagent:/home/eriase/.m2/repository/org/jacoco/org.jacoco.agent/0.8.11/org.jacoco.agent-0.8.11-runtime.jar=destfile=/home/eriase/repoes/my-app/target/jacoco-quarkus.exec,append=true,exclclassloader=*QuarkusClassLoader
[INFO] Invoking build-helper:3.5.0:add-source (add-source) @ my-app
[INFO] Source directory: /home/eriase/repoes/my-app/src/generated/java added.
[INFO] Invoking resources:3.3.1:resources (default-resources) @ my-app
[INFO] Copying 29 resources from src/main/resources to target/classes
[INFO] Invoking quarkus:3.8.1:generate-code (default) @ my-app
[INFO] Invoking compiler:3.12.1:compile (default-compile) @ my-app
[INFO] Nothing to compile - all classes are up to date.
[INFO] Invoking resources:3.3.1:testResources (default-testResources) @ my-app
[INFO] Copying 37 resources from src/test/resources to target/test-classes
[INFO] Invoking quarkus:3.8.1:generate-code-tests (default) @ my-app
[INFO] Invoking compiler:3.12.1:testCompile (default-testCompile) @ my-app
[INFO] Nothing to compile - all classes are up to date.
Listening for transport dt_socket at address: 5005
2024-04-26 12:33:08,392+02 INFO  [io.qua.dat.dep.dev.DevServicesDatasourceProcessor] () () (build-23) Dev Services for default datasource (postgresql) started - container ID is 27067be03b9c
2024-04-26 12:33:16,000+02 INFO  [io.qua.ela.res.com.dep.DevServicesElasticsearchProcessor] () () (build-19) Dev Services for Elasticsearch started. Other Quarkus applications in dev mode will find the server automatically. For Quarkus applications in production mode, you can connect to this by configuring your application to use localhost:9200
__  ____  __  _____   ___  __ ____  ______ 
 --/ __ \/ / / / _ | / _ \/ //_/ / / / __/ 
 -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \   
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/   
2024-04-26 12:33:16,572+02 WARN  [io.qua.config] () () (Quarkus Main Thread) Unrecognized configuration key "quarkus.micrometer.export.otlp.default-registry" was provided; it will be ignored; verify that the dependency extension for this configuration is set or that you did 2024-04-26 12:33:16,572+02 WARN  [io.qua.config] () () (Quarkus Main Thread) Unrecognized configuration key "quarkus.micrometer.export.otlp.enabled" was provided; it will be ignored; verify that the dependency extension for this configuration is set or that you did not make a typo
2024-04-26 12:33:18,143+02 INFO  [org.fly.cor.int.lic.VersionPrinter] () () (Quarkus Main Thread) Flyway Community Edition 9.22.3 by Redgate
2024-04-26 12:33:18,145+02 INFO  [org.fly.cor.int.lic.VersionPrinter] () () (Quarkus Main Thread) See release notes here: https://rd.gt/416ObMi
2024-04-26 12:33:18,145+02 INFO  [org.fly.cor.int.lic.VersionPrinter] () () (Quarkus Main Thread) 
2024-04-26 12:33:18,153+02 INFO  [org.fly.cor.FlywayExecutor] () () (Quarkus Main Thread) Database: jdbc:postgresql://localhost:35669/quarkus (PostgreSQL 13.14)
2024-04-26 12:33:18,169+02 INFO  [org.fly.cor.int.sch.JdbcTableSchemaHistory] () () (Quarkus Main Thread) Schema history table "public"."flyway_schema_history" does not exist yet
2024-04-26 12:33:18,171+02 INFO  [org.fly.cor.int.com.cle.CleanExecutor] () () (Quarkus Main Thread) Successfully dropped pre-schema database level objects (execution time 00:00.000s)
2024-04-26 12:33:18,187+02 INFO  [org.fly.cor.int.com.cle.CleanExecutor] () () (Quarkus Main Thread) Successfully cleaned schema "public" (execution time 00:00.014s)
2024-04-26 12:33:18,198+02 INFO  [org.fly.cor.int.com.cle.CleanExecutor] () () (Quarkus Main Thread) Successfully cleaned schema "public" (execution time 00:00.009s)
2024-04-26 12:33:18,199+02 INFO  [org.fly.cor.int.com.cle.CleanExecutor] () () (Quarkus Main Thread) Successfully dropped post-schema database level objects (execution time 00:00.000s)
2024-04-26 12:33:18,277+02 INFO  [org.fly.cor.int.sch.JdbcTableSchemaHistory] () () (Quarkus Main Thread) Schema history table "public"."flyway_schema_history" does not exist yet
2024-04-26 12:33:18,280+02 INFO  [org.fly.cor.int.com.DbValidate] () () (Quarkus Main Thread) Successfully validated 8 migrations (execution time 00:00.062s)
2024-04-26 12:33:18,288+02 INFO  [org.fly.cor.int.sch.JdbcTableSchemaHistory] () () (Quarkus Main Thread) Creating Schema History table "public"."flyway_schema_history" ...
2024-04-26 12:33:18,306+02 INFO  [org.fly.cor.int.com.DbMigrate] () () (Quarkus Main Thread) Current version of schema "public": << Empty Schema >>
2024-04-26 12:33:18,319+02 INFO  [org.fly.cor.int.com.DbMigrate] () () (Quarkus Main Thread) Migrating schema "public" to version "1 - init"
2024-04-26 12:33:18,358+02 INFO  [org.fly.cor.int.sql.DefaultSqlScriptExecutor] () () (Quarkus Main Thread) DB: relation "participant_valid_code_systems" does not exist, skipping
2024-04-26 12:33:18,359+02 INFO  [org.fly.cor.int.sql.DefaultSqlScriptExecutor] () () (Quarkus Main Thread) DB: relation "participant_valid_code_systems" does not exist, skipping
2024-04-26 12:33:18,371+02 INFO  [org.fly.cor.int.com.DbMigrate] () () (Quarkus Main Thread) Migrating schema "public" to version "2 - views"
2024-04-26 12:33:18,386+02 INFO  [org.fly.cor.int.com.DbMigrate] () () (Quarkus Main Thread) Migrating schema "public" with repeatable migration "1 code system"
2024-04-26 12:33:18,474+02 INFO  [org.fly.cor.int.com.DbMigrate] () () (Quarkus Main Thread) Migrating schema "public" with repeatable migration "2 code"
2024-04-26 12:33:18,527+02 INFO  [org.fly.cor.int.com.DbMigrate] () () (Quarkus Main Thread) Migrating schema "public" with repeatable migration "3 organization"
2024-04-26 12:33:18,552+02 INFO  [org.fly.cor.int.com.DbMigrate] () () (Quarkus Main Thread) Migrating schema "public" with repeatable migration "4 participant"
2024-04-26 12:33:18,562+02 INFO  [org.fly.cor.int.com.DbMigrate] () () (Quarkus Main Thread) Migrating schema "public" with repeatable migration "5 search tag"
2024-04-26 12:33:18,583+02 INFO  [org.fly.cor.int.com.DbMigrate] () () (Quarkus Main Thread) Migrating schema "public" with repeatable migration "6 address"
2024-04-26 12:33:18,621+02 INFO  [org.fly.cor.int.com.DbMigrate] () () (Quarkus Main Thread) Successfully applied 8 migrations to schema "public", now at version v2 (execution time 00:00.146s)
2024-04-26 12:33:18,719+02 INFO  [org.hib.rea.pro.imp.ReactiveIntegrator] () () (JPA Startup Thread) HR000001: Hibernate Reactive
2024-04-26 12:33:19,181+02 INFO  [io.quarkus] () () (Quarkus Main Thread) my-app 4.0.0-SNAPSHOT on JVM (powered by Quarkus 3.8.1) started in 14.390s. Listening on: http://0.0.0.0:8090
2024-04-26 12:33:19,181+02 INFO  [io.quarkus] () () (Quarkus Main Thread) Profile dev activated. Live Coding activated.
2024-04-26 12:33:19,182+02 INFO  [io.quarkus] () () (Quarkus Main Thread) Installed features: [agroal, cache, cdi, elasticsearch-rest-client, flyway, hibernate-orm, hibernate-reactive, hibernate-validator, jdbc-postgresql, micrometer, narayana-jta, opentelemetry, reactive-pg-client, resteasy-reactive, resteasy-reactive-jackson, scheduler, smallrye-context-propagation, smallrye-health, smallrye-openapi, swagger-ui, vertx]
2024-04-26 12:33:20,008+02 DEBUG [org.acm.my.app.rep.CodeSystemInMemory] () () (vert.x-worker-thread-1) Running scheduled caching of Codes and Code systems
2024-04-26 12:33:20,432+02 DEBUG [org.acm.my.app.rep.CodeSystemInMemory] (d12ff295287a81e3b9eb4a6006dd2948) (8bee02b946650a74) (vert.x-eventloop-thread-4) Rebuilt code systems cache with 10 systems
2024-04-26 12:33:20,433+02 DEBUG [org.acm.my.app.rep.CodeSystemInMemory] (d12ff295287a81e3b9eb4a6006dd2948) (8bee02b946650a74) (vert.x-eventloop-thread-4) Rebuilt code systems cache with 1009 codes

So the complicated setup with .blocking. isn't needed 👍

P.S. Using quarkus 3.8.1

P.P.S. CodeSystemInMemory uses the reactive datasource to query all data loaded from the

2024-04-26 12:33:18,386+02 INFO [org.fly.cor.int.com.DbMigrate] () () (Quarkus Main Thread) Migrating schema "public" with repeatable migration "1 code system"
2024-04-26 12:33:18,474+02 INFO [org.fly.cor.int.com.DbMigrate] () () (Quarkus Main Thread) Migrating schema "public" with repeatable migration "2 code"

@yrodiere
Copy link
Member

Great, thank you @agreedSkiing!

I'll keep this open to at least add something to the documentation, as I suspect this is a common use case.

@yrodiere
Copy link
Member

The PR updating the documentation is ready, if anyone wants to give their opinion: #40313

Quarkus Documentation automation moved this from To do to Done Apr 26, 2024
@quarkus-bot quarkus-bot bot added this to the 3.11 - main milestone Apr 26, 2024
@gsmet gsmet modified the milestones: 3.11 - main, 3.10.1 May 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment