Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[SPARK-33898][SQL][FOLLOWUP] Fix the behavior of
SHOW CREATE TABLE
…
…to output deterministic results ### What changes were proposed in this pull request? This PR fixes a behavior of `SHOW CREATE TABLE` added in `SPARK-33898` (#32931) to output deterministic result. A test `SPARK-33898: SHOW CREATE TABLE` in `DataSourceV2SQLSuite` compares two `CREATE TABLE` statements. One is generated by `SHOW CREATE TABLE` against a created table and the other is expected `CREATE TABLE` statement. The created table has options `from` and `to`, and they are declared in this order. ``` CREATE TABLE $t ( a bigint NOT NULL, b bigint, c bigint, `extra col` ARRAY<INT>, `<another>` STRUCT<x: INT, y: ARRAY<BOOLEAN>> ) USING foo OPTIONS ( from = 0, to = 1) COMMENT 'This is a comment' TBLPROPERTIES ('prop1' = '1') PARTITIONED BY (a) LOCATION '/tmp' ``` And the expected `CREATE TABLE` in the test code is like as follows. ``` "CREATE TABLE testcat.ns1.ns2.tbl (", "`a` BIGINT NOT NULL,", "`b` BIGINT,", "`c` BIGINT,", "`extra col` ARRAY<INT>,", "`<another>` STRUCT<`x`: INT, `y`: ARRAY<BOOLEAN>>)", "USING foo", "OPTIONS(", "'from' = '0',", "'to' = '1')", "PARTITIONED BY (a)", "COMMENT 'This is a comment'", "LOCATION '/tmp'", "TBLPROPERTIES(", "'prop1' = '1')" ``` As you can see, the order of `from` and `to` is expected. But options are implemented as `Map` so the order of key cannot be kept. In fact, this test fails with Scala 2.13. ``` [info] - SPARK-33898: SHOW CREATE TABLE *** FAILED *** (515 milliseconds) [info] Array("CREATE TABLE testcat.ns1.ns2.tbl (", "`a` BIGINT NOT NULL,", "`b` BIGINT,", "`c` BIGINT,", "`extra col` ARRAY<INT>,", "`<another>` STRUCT<`x`: INT, `y`: ARRAY<BOOLEAN>>)", "USING foo", "OPTIONS(", "'to' = '1',", "'from' = '0')", "PARTITIONED BY (a)", "COMMENT 'This is a comment'", "LOCATION '/tmp'", "TBLPROPERTIES(", "'prop1' = '1')") did not equal Array("CREATE TABLE testcat.ns1.ns2.tbl (", "`a` BIGINT NOT NULL,", "`b` BIGINT,", "`c` BIGINT,", "`extra col` ARRAY<INT>,", "`<another>` STRUCT<`x`: INT, `y`: ARRAY<BOOLEAN>>)", "USING foo", "OPTIONS(", "'from' = '0',", "'to' = '1')", "PARTITIONED BY (a)", "COMMENT 'This is a comment'", "LOCATION '/tmp'", "TBLPROPERTIES(", "'prop1' = '1')") (DataSourceV2SQLSuite.scala:1997) ``` In the current master, the test doesn't fail with Scala 2.12 but it's still non-deterministic. ### Why are the changes needed? Bug fix. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? I confirmed that the modified test passed with both Scala 2.12 and Scala 2.13 with this change. Closes #33343 from sarutak/fix-show-create-table-test. Authored-by: Kousuke Saruta <sarutak@oss.nttdata.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
- Loading branch information