doesn't support analyzing statement type [set_statement] for SQL: set hive.execution.engine=mr;

[Light-Towers]

Describe the bug

• error: sqllineage.exceptions.UnsupportedStatementException: SQLLineage doesn't support analyzing statement type [set_statement] for SQL:set hive.auto.convert.join=false;

To Reproduce

# -*- coding: utf-8 -*-

from sqllineage.runner import LineageRunner

def test_create_as():

    sql = """
set hive.auto.convert.join=false;

drop table if exists dw.temp_A;
CREATE TABLE dw.temp_A(
  aaa string COMMENT 'aaa值')
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n' STORED AS ORC
 LOCATION 'hdfs://nameservice1/user/hive/warehouse/dw/temp/temp_A';
    """

    # result = LineageRunner(sql)
    result = LineageRunner(sql, 'hive')

    result.print_column_lineage()
    print(result.source_tables)
    print(result.target_tables)


if __name__ == "__main__":
    test_create_as()

Expected behavior
In our Hive SQL file, excute task need change engine (Tez or MR). Ignore set statement during parsing .

Python version (available via python --version)

• 3.10.11

SQLLineage version (available via sqllineage --version):

• 1.4.9

[reata]

This is already supported in master branch via #501

[Light-Towers]

This is already supported in master branch via #501
@reata you mean the next version is ok?

[reata]

Yes, or you can install from master branch if you need it now.

[Light-Towers]

Yes, or you can install from master branch if you need it now.

Hello, I have maked install from master, but it can't support set hive.execution.engine=mr;

# -*- coding: utf-8 -*-
from sqllineage.runner import LineageRunner
def test_create_as():
    sql = """
set hive.execution.engine=mr;
insert into db1.table1 select * from db2.table2;
    """
    # result = LineageRunner(sql)
    result = LineageRunner(sql, 'hive')

    result.print_column_lineage()
    print(result.source_tables)
    print(result.target_tables)

if __name__ == "__main__":
    test_create_as()

[Light-Towers]

use LineageRunner(sql, 'sparksql') can parse

[reata]

Yes, sqlfluff hive dialect is flawed in parsing set_statement. When set value is string, it's only parsable when quoted.

set hive.execution.engine="mr";

We need to fix sqlfluff to get this supported then.