Spark3: Add Direct File Query

[R7L208]

Brief summary of the change made

• Adds support to directly query files of supported types.
• Fixes Issue with Databricks absolute path tables #602

Are there any other side effects of this change that we should be aware of?

• Reorganized some grammar for Files to isolate File keywords that support this clause.

• Removed "XML" File format. No test cases and requires additional work before it will parse correctly.

• Linting fails due to special characters in identifiers (L057). These special keywords are required so would be an exception to L057. I've not yet worked on rules so would appreciate any guidance on where to get started on fixing this particular bug.

> sqlfluff lint --dialect spark3 test/fixtures/dialects/spark3/select_from_file.sql
== [test/fixtures/dialects/spark3/select_from_file.sql] FAIL
L:   6 | P:  14 | L057 | Do not use special characters in identifiers.
L:  13 | P:  14 | L057 | Do not use special characters in identifiers.
L:  20 | P:  10 | L057 | Do not use special characters in identifiers.
L:  27 | P:  11 | L057 | Do not use special characters in identifiers.
L:  34 | P:  11 | L057 | Do not use special characters in identifiers.
L:  41 | P:  11 | L057 | Do not use special characters in identifiers.
L:  48 | P:  11 | L057 | Do not use special characters in identifiers.
L:  55 | P:  17 | L057 | Do not use special characters in identifiers.
L:  62 | P:  17 | L057 | Do not use special characters in identifiers.
L:  69 | P:  10 | L057 | Do not use special characters in identifiers.
L:  76 | P:  12 | L057 | Do not use special characters in identifiers.
All Finished 📜 🎉!

[L: 76, P:  6]      |                from_expression:
[L: 76, P:  6]      |                    [META] indent:
[L: 76, P:  6]      |                    from_expression_element:
[L: 76, P:  6]      |                        table_expression:
[L: 76, P:  6]      |                            file_reference:
[L: 76, P:  6]      |                                keyword:                      'DELTA'
[L: 76, P: 11]      |                                dot:                          '.'
[L: 76, P: 12]      |                                identifier:                   '`/mnt/datalake/table`'
[L: 76, P: 33]      |                    [META] dedent:
[L: 76, P: 33]      |    statement_terminator:                                     ';'
[L: 76, P: 34]      |    newline:                                                  '\n'

Pull Request checklist

• Please confirm you have completed any of the necessary steps below.

• Included test cases to demonstrate any code changes, which may be one or more of the following:

  • .yml rule test cases in test/fixtures/rules/std_rule_cases.
  • .sql/.yml parser test cases in test/fixtures/dialects (note YML files can be auto generated with tox -e generate-fixture-yml).
  • Full autofix test cases in test/fixtures/linter/autofix.
  • Other.

• Added appropriate documentation for the change.

• Created GitHub issues for any relevant followup/future enhancements if appropriate.

[codecov]

Codecov Report

Merging #2553 (cdad74f) into main (a8234db) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##              main     #2553   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files          163       163           
  Lines        11828     11838   +10     
=========================================
+ Hits         11828     11838   +10     

Impacted Files	Coverage Δ
src/sqlfluff/dialects/dialect_spark3.py	100.00% <100.00%> (ø)
src/sqlfluff/rules/L057.py	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a8234db...cdad74f. Read the comment docs.

[tunetheweb]

Looks good. One question and one nit.

[R7L208]

@tunetheweb - I'll work on the review comments today. Thank you!

Do you have a suggestion or similar PR you could point me to for adjusting L057 since identifier is required in this case? It shouldn't raise the rule here.

[tunetheweb]

@tunetheweb - I'll work on the review comments today. Thank you!

Do you have a suggestion or similar PR you could point me to for adjusting L057 since identifier is required in this case? It shouldn't raise the rule here.

In L057 there is already an exception for BigQuery:

            # BigQuery table references are quoted in back ticks so allow dots
            #
            # It also allows a star at the end of table_references for wildcards
            # (https://cloud.google.com/bigquery/docs/querying-wildcard-tables)
            #
            # Strip both out before testing the identifier
            if (
                context.dialect.name in ["bigquery"]
                and context.parent_stack
                and context.parent_stack[-1].name == "TableReferenceSegment"
            ):
                if identifier[-1] == "*":
                    identifier = identifier[:-1]
                identifier = identifier.replace(".", "")

You could do something similar for this:

            # Allow slashes in file references
            if (
                context.parent_stack
                and context.parent_stack[-1].type == "file_reference"
            ):
                identifier = identifier.replace("/", "")

[R7L208]

Yep, just added them and running pytest locally then will push

[tunetheweb]

LGTM