Stream the sql content

[npeters]

• use sql batch mode to speed the exécution
• stream all the read operation on the sql scripts

[axelfontaine]

Thank you very much for this! This is something I've had on the agenda for a long time (see #94) and it's great you tackled it!

I have a couple of things I am unsure about, that would need to be cleaned up before merging:

• Make the pull request mergeable again (it currently conflicts)
• Add a testcase that shows that the correct statements is returned in case it errors out in the middle of the batch
• Test and ensure this works properly with SQL Server (there is already some kind of batch support built-in by using go as a delimiter)
• Clean up the many blank lines in the test dump file.

Also the fact you seem to now generate different checksum values than before (my untested suspicion by reading the code) should really be remedied if possible. I care very strongly about compatibility and new releases should be drop-in replacements as far as possible.

Alternatively, splitting the pull request in 2 between the batch and the streaming support would also make things easier to merge.

Do you see any other things that have to be taken care of?

Keep up the great work!
Axel

[npeters]

I finally split the PR.
This PR is only the streaming support.
I do not think there was a probleme with the checksum: the test check that the old and the new version give the same result.
I hope it will be easier to integrate.

[axelfontaine]

Please check the travis-ci build. It seems to be broken...

[npeters]

I fix it

[npeters]

Fix issue #615

[axelfontaine]

Hi Nicolas,

just wanted to let you know that I haven't forgotten you. As both the streaming and the batch support are rather fundamental changes in the parser and the executor, I want to have some time to test them before going live.

Next week I'll release 2.3. My plan is to merge your work shortly after that and make it part of the Flyway 3.0 release.

Cheers
Axel

[npeters]

Hi Alex

I'm not worry, I follow your work in flyway and I quite impressed !

I'm waiting for the beginning of the 3.0 and I hope I will be able to help you. :)

[rajito]

Hello Axel,

Any idea on when 3.0 would be available with the batch functionality?

For my project, I have made some small changes to the source to use batch updates, which significantly speeds up the migrations. Is it worth submitting a feature/pull request for it at some point in the next couple weeks?

Thank you,
Raj

[axelfontaine]

Yes, pull requests for 3.0 are welcome. Keep the scope of each pull request small (one feature or one bugfix). I expect to release 3.0 before the end of Q1.

[rajito]

Sounds good.

[axelfontaine]

Hi Nicolas,

do you want to submit a small PR for batch support in 3.0? I plan a release next week. If not, no worries, then we'll tackle this for 3.1.

Cheers
Axel

[npeters]

Hello Axel

I'm not plan to submit PR in 3.0 but I will do it for the 3.1

[axelfontaine]

Sounds good. Then we'll have more time to test everything.

[rajito]

Apologies for not getting back to this. I got dragged onto a different project and never got around to making the batch support code into production quality.

[CowChris]

What's the deelio here? Looks like the solution to the memory errors on large sql files got fixed, but it's held up because of a merge conflict.

So close!! :'(

[ghost]

Hi,

What's the state of this? Is it going to be in Flyway 4.0? It looked like there were some problem with merging but the implementation is done. So what's the plan? Thanks!

[axelfontaine]

Closing as this has been implemented differently as part of #615.