Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stream Parser for SQL scripts #615

Closed
bigslack opened this issue Sep 24, 2013 · 7 comments
Closed

Stream Parser for SQL scripts #615

bigslack opened this issue Sep 24, 2013 · 7 comments

Comments

@bigslack
Copy link

@bigslack bigslack commented Sep 24, 2013

Axel,

Have you considered a stream parsing approach to the SQL scripts instead of pulling an array of strings? It may be a more comprehensive way to handle multi-line comments and string literals (as the current partial tokenization method has some parsing issues around determining alternate quotes and regular quotes). It may also handle binary content better too (which I have not had very much luck thus far with flyway doing blobs). Anyway, just curious if you've given much consideration to this approach.

Cheers,
Adam

@CowChris
Copy link

@CowChris CowChris commented Jul 18, 2014

+1 Pleaaaaase sort!

Particularly going to affect anyone wanting to use this tool to create a database schema with a good amount of seed data in I expect (like us).

@zacronos
Copy link

@zacronos zacronos commented Aug 22, 2014

+1

I am trying to use Flyway to insert some seed data into our db in a couple new tables. The migration script is ~275k lines, and Flyway chokes with the following error: OutOfMemoryError: Java heap space.

@axelfontaine axelfontaine modified the milestones: Flyway 3.2, Flyway 3.1 Nov 24, 2014
@axelfontaine axelfontaine modified the milestones: Flyway 4.0, Flyway 3.2 Mar 4, 2015
@axelfontaine axelfontaine modified the milestones: Flyway 4.1, Flyway 4.0 Dec 30, 2015
@axelfontaine axelfontaine modified the milestones: Flyway 5.0, Flyway 4.1 Feb 6, 2017
@jelmerk
Copy link

@jelmerk jelmerk commented Jun 25, 2017

Was any work on this done ?

In #759 @axelfontaine said that in Flyway 3.1 he wanted to move to a streaming solution but as of today as far as I can tell we still have the same problem

In-fact for me it already chokes on SqlMigrationResolver trying to calculate a checksum

@bjoernhaeuser
Copy link

@bjoernhaeuser bjoernhaeuser commented Feb 13, 2018

So we can hope to get this with 5.1? :)

@axelfontaine
Copy link
Contributor

@axelfontaine axelfontaine commented Feb 13, 2018

@bjoernhaeuser What's your use case for this?

@bjoernhaeuser
Copy link

@bjoernhaeuser bjoernhaeuser commented Feb 13, 2018

@axelfontaine I have a 20mb sql file which contains all test data which is relevant for our development environment. With flyway 4.2 the process is stuck in tokenizing the SQL file and in the end fails.

@zacronos
Copy link

@zacronos zacronos commented Feb 15, 2018

I generally don't suggest alternative packages while projects are still being maintained, but since this has been a known issue for almost 3.5 years, I'm going to go ahead in this case. @axelfontaine, I hope you do not mind.

For those of you who cannot use Flyway because it crashes on large migrations, you might consider https://github.com/realtymaps/dbsync as a fully streaming alternative, at least until Flyway can stream as well. It is a straight SQL-based migration tool like Flyway, and actually was inspired in many ways by Flyway. However it has a much smaller memory footprint, particularly when processing large migrations.

It is not for everyone, of course -- it doesn't support as many databases as Flyway, and it doesn't have Maven or Gradle integration. Instead it is a CLI tool, so it can be readily integrated even in many projects where Flyway is not convenient.

Disclaimer: I'm the developer and maintainer of dbsync.

axelfontaine added a commit to flyway/flywaydb.org that referenced this issue Mar 14, 2018
axelfontaine added a commit that referenced this issue Apr 8, 2018
axelfontaine added a commit to flyway/flywaydb.org that referenced this issue Apr 8, 2018
dohrayme pushed a commit to dohrayme/flyway that referenced this issue Feb 3, 2020
dohrayme pushed a commit to dohrayme/flyway that referenced this issue Feb 3, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants
You can’t perform that action at this time.