Stream Parser for SQL scripts

[bigslack]

Axel,

Have you considered a stream parsing approach to the SQL scripts instead of pulling an array of strings? It may be a more comprehensive way to handle multi-line comments and string literals (as the current partial tokenization method has some parsing issues around determining alternate quotes and regular quotes). It may also handle binary content better too (which I have not had very much luck thus far with flyway doing blobs). Anyway, just curious if you've given much consideration to this approach.

Cheers,
Adam

[CowChris]

+1 Pleaaaaase sort!

Particularly going to affect anyone wanting to use this tool to create a database schema with a good amount of seed data in I expect (like us).

[zacronos]

+1

I am trying to use Flyway to insert some seed data into our db in a couple new tables. The migration script is ~275k lines, and Flyway chokes with the following error: OutOfMemoryError: Java heap space.

[jelmerk]

Was any work on this done ?

In #759 @axelfontaine said that in Flyway 3.1 he wanted to move to a streaming solution but as of today as far as I can tell we still have the same problem

In-fact for me it already chokes on SqlMigrationResolver trying to calculate a checksum

[bjoernhaeuser]

So we can hope to get this with 5.1? :)

[axelfontaine]

@bjoernhaeuser What's your use case for this?

[bjoernhaeuser]

@axelfontaine I have a 20mb sql file which contains all test data which is relevant for our development environment. With flyway 4.2 the process is stuck in tokenizing the SQL file and in the end fails.

[zacronos]

I generally don't suggest alternative packages while projects are still being maintained, but since this has been a known issue for almost 3.5 years, I'm going to go ahead in this case. @axelfontaine, I hope you do not mind.

For those of you who cannot use Flyway because it crashes on large migrations, you might consider https://github.com/realtymaps/dbsync as a fully streaming alternative, at least until Flyway can stream as well. It is a straight SQL-based migration tool like Flyway, and actually was inspired in many ways by Flyway. However it has a much smaller memory footprint, particularly when processing large migrations.

It is not for everyone, of course -- it doesn't support as many databases as Flyway, and it doesn't have Maven or Gradle integration. Instead it is a CLI tool, so it can be readily integrated even in many projects where Flyway is not convenient.

Disclaimer: I'm the developer and maintainer of dbsync.