Spark3: Support for LATERAL VIEW clause

[R7L208]

Brief summary of the change made

• update sqlfluff/core/parser/grammar/anyof.py to allow exclude named argument to be an iterable
• created LateralViewClauseSegment in sqlfluff/dialects/dialect_spark3.py to match/parse the clause
• removed unneeded comments & TODOs
• Add exclusion for LATERAL keyword in AliasExpressionSegment
• Redefine FromExpressionElementSegment from sqlfluff/dialects/dialect_ansi.py to include LateralViewClauseSegment
• Updated L026 to be default off for Spark3 due to false positives in LATERAL VIEW (and structs are applicable here as well)

Are there any other side effects of this change that we should be aware of?

• I added a try/except in sqlfluff/core/parser/grammar/anyof.py that will use any() to exclude segments then revert to the previously existing code block on a TypeError
• Updated docstring to reflect all dialects in the exclusion list for L026

Pull Request checklist

• Please confirm you have completed any of the necessary steps below.

• Included test cases to demonstrate any code changes, which may be one or more of the following:

  • .yml rule test cases in test/fixtures/rules/std_rule_cases.
  • .sql/.yml parser test cases in test/fixtures/dialects (note YML files can be auto generated with tox -e generate-fixture-yml).
  • Full autofix test cases in test/fixtures/linter/autofix.
  • Other.

• Added appropriate documentation for the change.

• Created GitHub issues for any relevant followup/future enhancements if appropriate.

[codecov]

Codecov Report

Merging #2687 (52d8fc9) into main (0307c76) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##              main     #2687   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files          163       163           
  Lines        12025     12034    +9     
=========================================
+ Hits         12025     12034    +9     

Impacted Files	Coverage Δ
src/sqlfluff/core/parser/grammar/anyof.py	100.00% <100.00%> (ø)
src/sqlfluff/dialects/dialect_spark3.py	100.00% <100.00%> (ø)
src/sqlfluff/rules/L026.py	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0307c76...52d8fc9. Read the comment docs.

[tunetheweb]

A lot cleaner now! But spotted a few more things.

[tunetheweb]

One last question, I promise!

Wouldn't this allow this?:

    LATERAL VIEW OUTER EXPLODE(ARRAY()) AS tbl_name AS c_age;

[R7L208]

Please keep asking 🙂 Updated (and will include CI updates as well) to fix this.

Using the below instead.

        Ref("SingleIdentifierGrammar", optional=True),
        Ref("AliasExpressionSegment", optional=True),

I ran the query with AS tbl_name AS c_age; and it failed. The first AS isn't allowed like you raised.

[R7L208]

Tested the second AS and going to push one more commit then should be good to go

[R7L208]

Also removed two other test cases that were not valid so thank you for catching that initial case.

[tunetheweb]

My read of the above link is this should be:

Suggested change

	Sequence(
	"AS",
	Ref("SingleIdentifierGrammar"),
	optional=True,
	),
	Sequence(
	"AS",
	Delimited(
	Ref("SingleIdentifierGrammar")
	),
	optional=True,
	),

Is that so? And if so should we add a test case for multiple aliases?

[R7L208]

Sorry, not firing on all cylinders today.

        Sequence(
            "AS",
            Delimited(
                Ref("SingleIdentifierGrammar")
            ),
        ),

Should be correct. I forgot to remove the optional clause. I'll add a case for multiple column aliases

[tunetheweb]

Got there in the end!
😁