feat: split_with_parser, split_with_scanner

[SKalt]

I'm noticing some weirdness in the split_with_scanner example. Malformed tokens seem to be getting dropped:

assert_eq!(
    pg_query::split_with_scanner("select 1; asdf; select 3;").unwrap(),
    vec!["select 1", "asdf", "select 3"],
); // fails: actually produces ["select 1", "select 3"]

Is this expected?

Resolves #5.

[lfittl]

I'm noticing some weirdness in the split_with_scanner example. Malformed tokens seem to be getting dropped:

assert_eq!(
    pg_query::split_with_scanner("select 1; asdf; select 3;").unwrap(),
    vec!["select 1", "asdf", "select 3"],
); // fails: actually produces ["select 1", "select 3"]

Is this expected?

Yup, that's expected, if the statement only contains invalid tokens, we skip it here:

https://github.com/pganalyze/libpg_query/blob/13-latest/src/pg_query_split.c#L113

We could change that behavior, its not something that anyone relies on today, to my knowledge. Would it be helpful for your use case?

[SKalt]

Cool, I'll update my comment on the function.
I don't depend on that function, but I did have a use-case for split_with_scanner: I wanted to split the postgres regression tests into individual statements to build a SQL test corpus (skalt/sql_parser_tests). The problem was using split_with_scanner ended up mixing in psql meta-commands, so I bailed and wrote a nom parser that did what I wanted (skalt/pqsl_splitter).

[lfittl]

Cool, I'll update my comment on the function. I don't depend on that function, but I did have a use-case for split_with_scanner: I wanted to split the postgres regression tests into individual statements to build a SQL test corpus (skalt/sql_parser_tests). The problem was using split_with_scanner ended up mixing in psql meta-commands, so I bailed and wrote a nom parser that did what I wanted (skalt/pqsl_splitter).

Great, and neat idea!

I'll leave the Rust code review to @seanlinsley, since he's been maintaining most of the library, but looks good from a quick glance :)

[seanlinsley]

@lfittl are there cases where these functions can raise unique error messages, or should we be using the existing Parse error type instead?

[lfittl]

Good question. There are no unique Split errors (at least not in the C library), so we could rely on Parse error for split_with_parser.

For split_with_scanner it seems reasonable to have its own error class, since we don't actually call the parser there, so it'd be incorrect to treat errors as "parse errors".

For reference, here is the rather simple code that does the splitting in C: https://github.com/pganalyze/libpg_query/blob/13-latest/src/pg_query_split.c

[seanlinsley]

👍 in general, just not sure if we need the extra error types

[SKalt]

First-time contributors need a maintainer to approve running workflows

@seanlinsley would you be willing to approve workflows on this PR so we're all looking at the same unit test output?

[seanlinsley]

Thanks for submitting this PR @SKalt!