Interest in partial parsing and sql formatting?

[zmaril]

Hello! I've been loving sqlparser-rs, I've been having a lot of fun using it with a tool I've been working on recently. As part of my work, I would like to do two things with sqlparser-rs that I don't think are supported right now. I'd like to be able to get a partial parse of a sql string back out, for use with code editors and text highlighting. Additionally, I would like to be able to format sql with my particular dialect as well eventually. Would you all be interested in me trying to contribute this back to the library somehow when I get around to it? Did I miss something somewhere hopefully? Thanks again for a great library.

[wpf375516041]

This is a great feature. SQL formatting, removing comments, and highlighting keywords are all very practical capabilities. If necessary, I can also contribute what I can.

[alamb]

I think adding hooks (like offsets / spans into the AST) would fit into sqlparser-rs 's design

I am not quite sure what you mean by "partial parse of a sql string" -- maybe you mean the parser would be able to produce an AST even for a partial SQL query?

Text highlighting I think would be possible if the ast had spans (location in the original parsed query) -- I am not sure highlighting itself fits into the crate

[wpf375516041]

I think we only need to provide some built-in visitors like alibaba/druid to meet the needs, which will also help others learn and use sqlparser @alamb

[alamb]

There is a basic visitor already: https://docs.rs/sqlparser/latest/sqlparser/ast/trait.Visitor.html, perhaps that is enough

[wpf375516041]

Of course, this is a small case for SQL engine developers. I want to provide some SqlUtils to attract more developers. I will try to do something. If there is progress, I will synchronize with everyone.

[Viicos]

I am not quite sure what you mean by "partial parse of a sql string" -- maybe you mean the parser would be able to produce an AST even for a partial SQL query?

The goal would be to be error tolerant yes. Instead of bailing out eagerly when an unexpected token is encountered, an invalid AST node can be produced, and the parsing can continue.

Taking the following example:

SELECT col_name FROM

The parser expects a table factor, but will return an error early.

One option could be to still produce a TableFactor::Table node, with an empty name. Some additional metadata can also be added to indicate that it is an invalid node.

By successfully parsing the given SQL query, editors can more easily provide completions and other LSP features, without having to rely solely on tokens.

For reference, this is how the Ruff's Python parser do it. With the following code:

a = 

It produces a StmtAssign node, with the assigned value being an Name node, with an empty string, and Invalid set as ctx (playground).

---

Introducing such an error tolerant mechanism is a lot of work, although can be done incrementally. However, this may require some breaking changes, in particular in the returned value type of the parsing functions. It currently returns the AST, but would ideally return a parsed result struct, containing the AST and a list of encountered errors. Any feedback on whether there is interest in having this introduced in the parser is welcomed!

[alamb]

cc @iffyio and @yoavcloud as they are the primary maintainers these days

[yoavcloud]

The team I work with at Satori and I have been thinking about this problem for a while. We've had many cases where we could benefit from an incomplete AST for our own purposes, for example, a while ago, the parser only supported the first statement, and it failed on the second:

INSERT INTO tbl VALUES (...)
INSERT OVERWRITE INTO tbl VALUES (...)

For our use case, the two are equivalent and we would have been just fine with an incomplete AST. But as written above, this will likely completely change the error handling strategy of the parser.

I will try to carve out some time in the near future to try and propose a solution.

[alamb]

Maybe we could add something like an Expr that is "IncompleteExpr" or something that could be used as a placeholder when the parser failed / was incomplete

[wpf375516041]

It seems to be just an optional keyword, like AS, INNER JOIN, which can be abbreviated or omitted in many databases @yoavcloud

[wpf375516041]

I am not quite sure what you mean by "partial parse of a sql string" -- maybe you mean the parser would be able to produce an AST even for a partial SQL query?

The goal would be to be error tolerant yes. Instead of bailing out eagerly when an unexpected token is encountered, an invalid AST node can be produced, and the parsing can continue.

Taking the following example:

SELECT col_name FROM
The parser expects a table factor, but will return an error early.

One option could be to still produce a TableFactor::Table node, with an empty name. Some additional metadata can also be added to indicate that it is an invalid node.

By successfully parsing the given SQL query, editors can more easily provide completions and other LSP features, without having to rely solely on tokens.

For reference, this is how the Ruff's Python parser do it. With the following code:

a =
It produces a StmtAssign node, with the assigned value being an Name node, with an empty string, and Invalid set as ctx (playground).

Introducing such an error tolerant mechanism is a lot of work, although can be done incrementally. However, this may require some breaking changes, in particular in the returned value type of the parsing functions. It currently returns the AST, but would ideally return a parsed result struct, containing the AST and a list of encountered errors. Any feedback on whether there is interest in having this introduced in the parser is welcomed!

The question seems to have deviated somewhat. The original goal of this issue was to provide two built-in capabilities:

1. Extracting keywords, such as returning an array of ['select', 'from', ...]
2. SQL formatting

Based on my observations at the time, we only needed to provide an enhanced visitor similar to the one used in the Alibaba/Druid projects.

For SQL engine developers, these two capabilities can be achieved using a basic visitor, but for general developers, it seems a bit difficult.

@Viicos

[Viicos]

For 1., it all depends on what the OP meant by "partial parsing". Is it parsing only a part of a SQL statement (in which case the parser needs to be error tolerant if the partial statement is invalid as is)? Or partial parsing as "only do some steps of the parsing process"?

For the latter, it would only require making the lexer error tolerant. For syntax highlighting, although several alternatives can be used (regex-based lexer, tree-sitter (I believe), TextMate grammars).

[wpf375516041]

From a practical perspective, the requirement to "parse only a portion of a SQL statement" isn't yet clear, and I haven't encountered a similar use case in my existing projects. Implementing this functionality based on the current code structure would require significant changes, so I recommend holding off until there's real demand and prioritizing core scenarios.

Regarding the features mentioned in the initial goal (such as lineage analysis, SQL syntax highlighting, and formatting), there's widespread demand in the data development field. Many companies face SQL scripts that often run into thousands or even tens of thousands of lines, placing significant performance pressure on front-end parsing and processing. Using WASM acceleration is a very promising solution, and Rust + WASM is a currently mainstream technology combination, offering high performance and a strong ecosystem.

Of course, these features are application-layer, and whether the SQL parser should include them remains a decision. My initial thinking was to develop application-side capabilities to attract users to DataFusion.