fix: parser handling comments and empty lines

[mfridman]

Fix #226, fix #198. Preserve #200 and #372

There are two ways we can approach this:

1. Require users to enclose such statements with -- +goose StatementBegin and -- +goose StatementEnd. And we guarantee the statement will have comments preserved.
2. Always preserve leading comments ^-- within a statement before a delimiter (;) is found.

[mfridman]

This fixes the issue reported here: #198 (comment)

Once we're in the middle of a statement, we preserve comments.

Furthermore, we preserve whitespace which was reported here: #200 (comment)

And initially fixed here: #372 (I've captured this test)

[mfridman]

This statement is not wrapped with -- +goose Statement, and we validate that the final line (below) is correctly included in parsed statement.

-----END OPENSSH PRIVATE KEY-----

[mfridman]

This fixes an issue staticcheck found:

SA6002: argument should be pointer-like to avoid allocations (staticcheck)

Nice explanation from the docs.

A sync.Pool is used to avoid unnecessary allocations and reduce the amount of work the garbage collector has to do.

When passing a value that is not a pointer to a function that accepts an interface, the value needs to be placed on the heap, which means an additional allocation. Slices are a common thing to put in sync.Pools, and they’re structs with 3 fields (length, capacity, and a pointer to an array). In order to avoid the extra allocation, one should store a pointer to the slice instead.

[mfridman]

The %q verb prints it out as hex, changing to %d so it's the int value.

- unexpected state '\x00' on line ..
+ unexpected state '0' on line

[mfridman]

In isolation, it looks like we should not be including these comments, but I think it's defensible to say this is a user error and they should avoid putting trailing comments after the statement but before the final -- +goose StatementEnd annotation.

-- +goose StatementBegin
[statement ...];
-- 3 this comment will be preserved
  -- 4 this comment will be preserved


-- +goose StatementEnd

[mfridman]

We could do a second pass over the complex statement and trim any empty lines/comments after the final semicolon. I'll keep this in mind, likely in a follow-up PR.

[mfridman]

Added logic in 98ca499 that does a final cleanup of the statement. Effectively this:

func cleanupStatement(input string) string {
	if n := strings.LastIndex(input, ";"); n > 0 {
		return input[:n+1]
	}
	return input
}

So that this input:

-- +goose StatementBegin




-- 1 this comment will NOT be preserved
  -- 2 this comment will NOT be preserved


CREATE FUNCTION do_something(sql TEXT) RETURNS INTEGER AS $$
BEGIN
  -- initiate technology
  PERFORM something_or_other(sql);

  -- increase technology
  PERFORM some_other_thing(sql);

  -- technology was successful
  RETURN 1;
END;
$$ LANGUAGE plpgsql;

-- 3 this comment will NOT be preserved
  -- 4 this comment will NOT be preserved


-- +goose StatementEnd

Will end up as the following statement:

CREATE FUNCTION do_something(sql TEXT) RETURNS INTEGER AS $$
BEGIN
  -- initiate technology
  PERFORM something_or_other(sql);

  -- increase technology
  PERFORM some_other_thing(sql);

  -- technology was successful
  RETURN 1;
END;
$$ LANGUAGE plpgsql;

[mfridman]

Alright @VojtechVitek , pretty happy with how this ended up. It's a lot easier to test locally and the logic fixes some edge cases with the parser around leading/trailing empty lines and comments, as well as comments within statements that should be preserved.

[VojtechVitek]

LGTM

that was a tricky one! nice work 🎉