fix goose memory consumption on large migrations

[hexdigest]

Hi,

I found out that on large SQL migrations that either have multiple queries (>25k insert queries in my case) or one insert query with 25K lines of values divided by "," goose consumes about 200M of memory. The size of the .sql file is about 5M.

This PR fixes the issue without breaking to much of an existing code.

[VojtechVitek]

Hi @hexdigest. Thanks for your contribution!

1. How does this work in high level?
2. Why do we need sync.Pool at all?
3. Can't we use bytes.Buffer instead?

[hexdigest]

Hi @VojtechVitek

Well, actually this PR fixes 2 problems:

1. Default 64K buffer in bufio.Scanner, so you can't run migrations where the lines are too long, I get "token too long" error when tried to use result of pg_dump as a migration.
2. If you simply use new bigger buffer in endsWithSemicolon func then on large migrations you have a chance to get OOM error before GC cleans everything, in my case container is killed by docker (64Mb limit is set), that's why I used sync.Pool

I can update PR using bytes.Buffer but it's not concurrent safe, so if there'll be parallel tests for endsWithSemicolon/getSQLStatements they might blink. What do you think?

[VojtechVitek]

@hexdigest hey, sorry for late answer. Yeah, if you could use bytes.Buffer, that'd be great. I don't think we need to worry about concurrency, tests might as well use two instances of goose. Goose migrations are supposed to run sequentially anyway.

How did you come up with number for scanBufSize btw, can you explain in the code as a comment?

LGTM, if we get rid of sync.Pool and use bytes.Buffer :)

[VojtechVitek]

LGTM

I found some time today to test this PR against our CI test suite (~100 sample .sql migrations that we ran in production in the past) and against our pre-prod environments. No issues found. Going to merge this.

Thanks again for your contribution!