Async migrations

[iNikem]

Some migrations can effectively be run totally asynchronously relatively to other migrations. E.g. creating of index. It would be very nice to have a possibility to mark some migrations as async and not to block the whole process on them.

[axelfontaine]

I agree this can be valuable. It is not an immediate priority, but let's keep this open and see if it gathers traction.

Cheers
Axel

[anirudha13]

+1 for asynchronous migrations. We are using flyway with phoenix and some of our index creations take quite a long time.

[cpoole]

some of our migrations can take upwards of 24 hours for back-filling massive tables. Would be great if these long migrations didn't block future migrations

[f1xmAn]

Agree, that's valuable feature for us too.

[codersbrew]

Please implement :)

[aayushbhasin]

Would be an extremely useful feature.

[evanye]

+1

[ileshko]

+1!!

[codeyq]

+1!

[kakuism]

+1

[gvdenbro]

+1

[octavian-nita]

+1

Imagine that we might use Flyway's mechanisms to trigger the update of other, non/No-SQL repos such as ElasticSearch or even to send messages to various messaging/streaming infrastructures such as Kafka.

While Java migrations help a great deal since we can implement pretty much whatever we want (including spawning async jobs), native Flyway support would make things a breeze.

And... elaborating on this topic... I'm pretty much starting to see Flyway as a generic update mechanism and not only an SQL-related one. I am not sure whether this idea sprung before but a more decoupled, "update engine"-like mechanism would be great, imho.

[MikielAgutu]

@octavian-nita

And... elaborating on this topic... I'm pretty much starting to see Flyway as a generic update mechanism and not only an SQL-related one. I am not sure whether this idea sprung before but a more decoupled, "update engine"-like mechanism would be great, imho.

In Flyway 7 we shipped a new feature that will help with that: https://flywaydb.org/documentation/migrations#script-migrations

[detec]

+1

[sriram-au]

+1

[mackec1]

+1

[christrotteradhoc]

+1

[silvertonsg]

+1

[malumatos]

+1

[Jojoooo1]

• 1

[dpolivaev]

+1

[rajeshhereforyou]

+1

[spark59]

+1

[kantselovich]

+1

[EarthCitizen]

We ran into an issue in which our containers were failing health checks due to index creations taking too long. Async migration tasks are absolutely essential for embedded Spring Boot start-up migration.

[aprabhat]

@EarthCitizen I ran into same issue of POD getting restarted. Have you found any workaround for this?

[EarthCitizen]

@aprabhat In our case, they were index creations, and we created our Flyway scripts with "if not exists" for the indexes. Therefore, once they were created on the DB, the DB continued to work on them after the connection was lost when the container was restarted, and the scripts were marked as complete on the reboot due to the indexes already existing on the DB.

Basically, we got lucky.

[EarthCitizen]

I agree this can be valuable. It is not an immediate priority, but let's keep this open and see if it gathers traction.

@axelfontaine This issue appears to have traction. There is a clear community need expressed here.

[aprabhat]

@aprabhat In our case, they were index creations, and we created our Flyway scripts with "if not exists" for the indexes. Therefore, once they were created on the DB, the DB continued to work on them after the connection was lost when the container was restarted, and the scripts were marked as complete on the reboot due to the indexes already existing on the DB.

Basically, we got lucky.

@EarthCitizen
I appreciate your response. In my case, I require the addition of a column. To achieve this, I implemented a workaround by creating a stored procedure that checks whether the column already exists. If the POD is terminated before successfully recording the entry in the schema version table, the subsequent POD will execute the same script. If the new column has already been created, the stored procedure will skip the "add column" step, and the new POD will successfully record the entry in the schema version table.

This workaround suffices for now, but it would be highly beneficial if Flyway provided an asynchronous migration feature.

[tomekrojek]

+1

[Siddharth-Srinivas]

+1