Update parse method of Simple API to output JSON parse tree

[jpy-git]

Brief summary of the change made

Fixes #2062. Updated parse method of Simple API to output the parse tree in JSON format. This keeps the output of the Simple API simple and decoupled from the internal workings of SQLFluff.

Updated the docs and unit tests accordingly.

Are there any other side effects of this change that we should be aware of?

Changes the output of the parse method of the Simple API. However, we're about to release a new minor release soon anyway so should be fine.

Pull Request checklist

• Please confirm you have completed any of the necessary steps below.

• Included test cases to demonstrate any code changes, which may be one or more of the following:

  • .yml rule test cases in test/fixtures/rules/std_rule_cases.
  • .sql/.yml parser test cases in test/fixtures/dialects (note YML files can be auto generated with python test/generate_parse_fixture_yml.py or by running tox locally).
  • Full autofix test cases in test/fixtures/linter/autofix.
  • Other.

• Added appropriate documentation for the change.

• Created GitHub issues for any relevant followup/future enhancements if appropriate.

[jpy-git]

@tunetheweb @barrywhart pretty happy with how the outputs of the Simple API are looking now 😄

[jpy-git]

Just seen there's some test for the example files that I'll need to update tomorrow, but the rest is good for review

[codecov]

Codecov Report

Merging #2082 (5d323c5) into main (da91d6c) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##              main     #2082   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files          148       148           
  Lines        10576     10575    -1     
=========================================
- Hits         10576     10575    -1     

Impacted Files	Coverage Δ
src/sqlfluff/api/simple.py	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update da91d6c...5d323c5. Read the comment docs.

[jpy-git]

@tunetheweb resolved the coverage issues on this 👍

[barrywhart]

Nice!

[tunetheweb]

I see we lost example 3 (since we can no longer call get_table_references) but I wonder if we need a simple example to show how to equivalent can now be accomplished with the JSON?

Because we've basically gone from this in our docs:

#  -------- PARSING ----------
# NOTE: sqlfluff is still in a relatively early phase of its
# development and so until version 1.0.0 will offer no guarantee
# that the names and structure of the objects returned by these
# parse commands won't change between releases. Use with care
# and keep updated with the changelog for the project for any
# changes in this space.

parsed = sqlfluff.parse(my_bad_query)

# Get the structure of the query
structure = parsed.tree.to_tuple(show_raw=True, code_only=True)
# structure = ('file', (('statement', (('select_statement', (('select_clause', (('keyword', 'SeLEct'), ...

# Extract certain elements
keywords = [keyword.raw for keyword in parsed.tree.recursive_crawl("keyword")]
# keywords = ['SeLEct', 'as', 'from']
tbl_refs = [tbl_ref.raw for tbl_ref in parsed.tree.recursive_crawl("table_reference")]
# tbl_refs == ["myTable"]

to this:

#  -------- PARSING ----------

# Parse the given string and return a JSON representation of the parsed tree.
parse_result = sqlfluff.parse(my_bad_query)
# parse_result = {'file': {'statement': {...}, 'newline': '\n'}}

Appreciate this is more to do with telling someone how to use Python to manipulate JSON so could be argued it's not really in SQLFluff remit to do that, but at same time the original example showed how to crawl the table_reference segment (which IS a SQLFluff concept) and the new, reduced example, doesn't really give that guidance to users.

WDYT?

[tunetheweb]

Ooops took too long to write this (got caught on a call) and now merged. Still let me know what you think.

[barrywhart]

Ohhh, that's a valuable example. We used the SQLFluff API at my last job for this exact purpose -- to determine table references, then use that info to automatically create table lineage diagrams.

[jpy-git]

We can reverse out the change if you want? Do we want to provide an example of how to do this with json or do you want the original method (i.e. full revert)?

[tunetheweb]

Oh no keep the change. JSON example is what I mean.

[jpy-git]

Awesome, let me pull together some examples tonight/tomorrow 😄