Parsing incomplete SQL statements

[robertbindar]

Hey @tobymao! This is excellent work and I'm pretty sure it will help lots of people in their project.
sqlglot will for sure help us in https://github.com/MariaDB/mariadb_kernel

We are trying to build an autocompletion and introspection feature (as part of a GSoC project) and the ability to get the parsed expression and the type of each token in the expression is exactly what we were looking for (and eventually implementing ourselves if no reasonable solution would be found in the wild).

My question for you is, is it very complicated to make sqlglot able to parse and generate a partial expression tree on incomplete SQL statements?
Now for instance, if you execute parse("select a from t1 where"), you'd get an exception. If you execute parse("select a from"), you would get a partial expression tree, which is fantastic.
Is this a bug in sqlglot and it is designed to deal with incomplete statements? Or it is just by accident that the latter statement above works?

Thanks again for this great project!

[tobymao]

hello @robertbindar.

I don't think it is too complicated to make this work, it was just not my initial target. The reason the latter works is simply by accident at the moment.

The design goal of sqlglot is not to be validator as I expect the actual execution engines to do that. So if it is clean, we can totally add support to deal with incomplete statements. My guess is that we just need to make some of the parsing less strict which should be easy. Feel free to open up some PR's / examples to fix these cases.

[robertbindar]

Hi! I see, sure, validating should be the job of execution engines for sure. I was thinking of adding some kind of option to just not throw some of the exceptions whilst parsing. Incomplete statements are indeed wrong statements, so the default behavior should be to do throw exceptions when wrong statements are passed. Having this functionality under user-specified option sounds like the best idea to me. What do you think?

[tobymao]

as long as the code is clean i think that'd make sense. it should be easy because i have a method in parser#raise_error that can choose to not raise with an option passed

[tobymao]

this has been implemented, let me know if there are other use cases, but you can do

transpile(..., error_level=ErrorLevel.IGNORE)