[SPARK-45880][SQL] Make the like `pattern` semantics used in all commands consistent #43751

panbingkun · 2023-11-10T09:04:21Z

What changes were proposed in this pull request?

The pr aims to

make the like pattern semantics used in SQL SELECT ... FROM ... WHERE LIKE <pattern> and SHOW TABLE EXTENDED LIKE <pattern> consistent. The SQL for semantic change of like include:
a. SHOW NAMESPACES ... LIKE
b. SHOW TABLES ... LIKE
c. SHOW TABLE EXTENDED ... LIKE
d. SHOW VIEWS ... LIKE
e. SHOW ... FUNCTIONS ... LIKE
f. SHOW CATALOGS LIKE
introduce a new TableCatalog.listTable overload that takes a pattern string for v2 catalog.

Why are the changes needed?

As we discussed in implementing the ShowTablesExtended logic in the V2, we need such an API in TableCatalog to make the code logic more cohesive.
#37588 (comment)

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass GA.
Manually test.

Was this patch authored or co-authored using generative AI tooling?

No.

…at takes a pattern string for v2 catalog

panbingkun · 2023-11-10T09:17:14Z

cc @cloud-fan

github-actions · 2024-02-19T00:18:58Z

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

cloud-fan · 2024-02-19T06:31:31Z

sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableCatalog.java

+   * @param namespace a multi-part namespace
+   * @param pattern the filter pattern, only '*' and '|' are allowed as wildcards, others will
+   *                follow regular expression convention, case-insensitive match and white spaces
+   *                on both ends will be ignored


do we have a doc page for the pattern string semantic? If we do we should reference it here.

I searched the document and the only possible relationship is this one:
https://spark.apache.org/docs/latest/sql-ref-syntax-qry-select-like.html#parameters

Perhaps we should explain it in detail here?
(PS: The first pr that introduces StringUtils.filterPattern is: #12206)

yea if they use the same implementation. The LIKE pattern doc does not even mention the *.

Another option is to document it in the SHOW TABLES doc page.

I have looked at the document https://spark.apache.org/docs/latest/sql-ref-syntax-aux-show-tables.html#parameters(SHOW TABLES doc page) and found that the parameter regex_pattern in it explains the pattern.

Thank you very much for your reminder, Let's refer to it.

cloud-fan · 2024-02-20T06:53:48Z

sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableCatalog.java

+   * If the catalog supports views, this must return identifiers for only tables and not views.
+   *
+   * @param namespace a multi-part namespace
+   * @param pattern the filter pattern, only '*' and '|' are allowed as wildcards, others will


not related to this PR, but the existing doc is a bit vague. | is not a wildcard, right? And | is also a valid syntax in regex. Can we take a look at other databases and see how they document it?

Okay, let me investigate it.

panbingkun · 2024-02-20T07:25:22Z

Just for making investigation records:

Database	Ref Doc	Wildcards
Snowflake	https://docs.snowflake.com/en/sql-reference/sql/show-tables	(% and _)
Clickhouse	https://clickhouse.com/docs/en/sql-reference/statements/show	(% and _)
PostgreSQL
Mysql	https://dev.mysql.com/doc/refman/8.0/en/show-tables.html, https://dev.mysql.com/doc/refman/8.0/en/pattern-matching.html, https://dev.mysql.com/doc/refman/8.0/en/regexp.html	(% and _)
Hive	https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-Show	`Before v4.0.0: (* and \| )` `After v4.0.0: (% and _)`

cloud-fan · 2024-02-20T09:39:54Z

I think it's more natural to follow the same behavior of the LIKE operator here. It seems all databases follow it (except for Hive before 4.0). Spark followed Hive at the beginning and that's probably why Spark has this special and weird behavior for the LIKE pattern in SHOW TABLES.

In fact, this is out of Spark's control, as it's the external catalog that applies the pattern string. We should follow the industry standard for defining the v2 catalog API. We should also update the SHOW TABLES doc page to mention the ideal behavior of the pattern string, as well as the legacy Hive behavior.

panbingkun · 2024-02-20T10:40:02Z

I think it's more natural to follow the same behavior of the LIKE operator here. It seems all databases follow it (except for Hive before 4.0). Spark followed Hive at the beginning and that's probably why Spark has this special and weird behavior for the LIKE pattern in SHOW TABLES.

In fact, this is out of Spark's control, as it's the external catalog that applies the pattern string. We should follow the industry standard for defining the v2 catalog API. We should also update the SHOW TABLES doc page to mention the ideal behavior of the pattern string, as well as the legacy Hive behavior.

Okay, let me handle it in this PR and update the document.
Additionally, do we need to add a legacy configuration (default is new behavior) to determine whether it is using the legacy behavior or the new behavior?
(PS: Yes, from the first PR, we can see that the original author's intention was to respect the legacy hive behavior.

)

cloud-fan · 2024-02-20T11:51:31Z

Yea let's add a legacy config.

panbingkun · 2024-02-22T03:13:00Z

@cloud-fan

The command that supports the 'LIKE ' syntax in spark are as follows:

SQL Synax	Example
SHOW namespaces ((FROM \| IN) multipartIdentifier)? (LIKE? pattern=stringLit)	SHOW NAMESPACES IN `ns` LIKE 'a\|b'
SHOW TABLES ((FROM \| IN) identifierReference)? (LIKE? pattern=stringLit)?	SHOW TABLES IN `DB` LIKE 'a\|b'
SHOW TABLE EXTENDED ((FROM \| IN) ns=identifierReference)? LIKE pattern=stringLit partitionSpec?	SHOW TABLES EXTENDED IN `DB` LIKE 'a\|b'
SHOW VIEWS ((FROM \| IN) identifierReference)?(LIKE? pattern=stringLit)?	SHOW VIEWS IN `DB` LIKE 'a\|b'
SHOW identifier? FUNCTIONS ((FROM \| IN) ns=identifierReference)? (LIKE? (legacy=multipartIdentifier \| pattern=stringLit))?	SHOW FUNCTIONS LIKE 'a\|b'
SHOW CATALOGS (LIKE? pattern=stringLit)?	SHOW CATALOGS LIKE 'a\|b'

If we only change the semantics of the wildcard supported by the pattern in command SHOW TABLES EXTENDED ... LIKE <pattern> from (* and |) to (% and _), it seems to cause semantic inconsistency for users. Therefore, I suggest making similar changes to all the commands above.

In addition, due to the fact that the maximum version of hive supported by Spark currently is 3.1.3, in this version, the widecard supported by the pattern are still: (* and |), and if we want to achieve support for widecard: (% and _), we may need to adopt some workarounds, such as:
In the API for get the databases based on the pattern, the current implementation logic is:

spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala

Lines 905 to 908 in 78f7c30

    
           override def getDatabasesByPattern(hive: Hive, pattern: String): Seq[String] = { 
        
             recordHiveCall() 
        
             hive.getDatabasesByPattern(pattern).asScala.toSeq 
        
           }

When supporting new semantics, the logic may be:
Step 1: use hive.getAllDatabase to retrieve all databases.
Step 2: perform pattern filtering on the Spark end for the above results.

Is this acceptable? This may cause some performance loss, but there seems to be no better way to achieve it.

…ns containing '%' for any character(s), and '_' for a single character

panbingkun · 2024-02-28T01:52:46Z

docs/sql-ref-syntax-aux-show-tables.md

@@ -40,12 +40,18 @@ SHOW TABLES [ { FROM | IN } database_name ] [ LIKE regex_pattern ]

 * **regex_pattern**

-     Specifies the regular expression pattern that is used to filter out unwanted tables. 
+     Specifies the regular expression pattern that is used to filter out unwanted tables.


Before:

After:

panbingkun · 2024-02-28T02:32:14Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/StringUtils.scala

+    }
+  }
+
+  private[util] def likePatternToRegExp(pattern: String): String = {


Follow the implementation logic of hive:
https://github.com/apache/hive/blob/89005659d1d8e167208ba4f9f9aaa2de7703229d/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLike.java#L64-L86

panbingkun · 2024-02-28T02:48:17Z

sql/core/src/test/resources/sql-tests/analyzer-results/show-tables.sql.out

 +- ResolvedNamespace V2SessionCatalog(spark_catalog), [showdb]


 -- !query
-SHOW TABLES LIKE 'show_t1*|show_t2*'


The OR syntax represented by | is no longer supported by default.

panbingkun · 2024-02-28T02:48:55Z

sql/core/src/test/resources/sql-tests/analyzer-results/show-views.sql.out



 -- !query
-SHOW VIEWS LIKE 'view_1*|view_2*'


The OR syntax represented by | is no longer supported by default.

cloud-fan · 2024-02-28T06:03:44Z

This is a hard decision. Technically the behavior of LIKE in many commands (SHOW TABLES LIKE ...) relies on the underlying catalog, which can be HMS of different versions, or a Hive-compatible metastore service. This is out of Spark's control.

From @panbingkun's investigation, the Hive behavior is actually very weird and different from other main-stream SQL systems (they follow the same behavior of the LIKE expression). Hive 4.0 also switches to the more common behavior.

There are some commands that we implement the LIKE filtering by our own, following the Hive behavior. Now we are in a hard position:

If we do nothing, then Spark's behavior of LIKE in commands is non-standard and different from other databases. We may also hit future behavior changes if we upgrade to Hive 4.0.
If we change the LIKE filtering behavior now, it's a breaking change, and also lead to inconsistent behaviors as some commands use Hive to do LIKE filtering.

cc @srielau

panbingkun · 2024-02-28T08:08:18Z

sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowFunctionsSuiteBase.scala

-      QueryTest.checkAnswer(
-        sql(s"SHOW USER FUNCTIONS IN $ns LIKE 'crc32i|date*'"),
-        Seq("crc32i", "date1900", "Date1").map(testFun => Row(qualifiedFunName("ns", testFun))))
+    withSQLConf(


Considering that this UT is testing "|" and "|" is no longer supported in the new mode, we have set the configuration spark.sql.legacy.useVerticalBarAndStarAsWildcardsInLikePattern to true to complete this test. In the future, we can consider removing this UT

panbingkun · 2024-02-28T08:35:21Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala

@@ -626,7 +626,12 @@ private[client] class Shim_v2_0 extends Shim with Logging {

  override def listFunctions(hive: Hive, db: String, pattern: String): Seq[String] = {
    recordHiveCall()
-    hive.getFunctions(db, pattern).asScala.toSeq
+    if (SQLConf.get.legacyUseStarAndVerticalBarAsWildcardsInLikePattern) {


This may cause performance loss, but there seems to be no better way.

panbingkun · 2024-02-28T08:42:27Z

...hive-thriftserver/src/main/java/org/apache/hive/service/cli/operation/MetadataOperation.java

@@ -57,54 +57,6 @@ public void close() throws HiveSQLException {
    cleanupOperationLog();
  }

-  /**


The following method is extracted separately into class MetadataOperationUtils and renamed as legacyXXX

panbingkun · 2024-02-28T08:51:54Z

...erver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkGetFunctionsOperation.scala

@@ -81,7 +87,7 @@ private[hive] class SparkGetFunctionsOperation(

    try {
      matchingDbs.foreach { db =>
-        catalog.listFunctions(db, functionPattern).foreach {
+        catalog.listFunctions(db, functionPattern).sortBy { item => item._1.funcName }.foreach {


In order to make the returned results more stable, as it contains a HashMap data structure.

panbingkun · 2024-02-28T08:53:56Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowTablesExec.scala

@@ -39,7 +39,7 @@ case class ShowTablesExec(

    val tables = catalog.listTables(namespace.toArray)
    tables.map { table =>
-      if (pattern.map(StringUtils.filterPattern(Seq(table.name()), _).nonEmpty).getOrElse(true)) {
+      if (pattern.forall(StringUtils.filterPattern(Seq(table.name()), _).nonEmpty)) {


This is only a correction made based on the syntax prompted by the IDE.

panbingkun · 2024-02-28T08:54:31Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowNamespacesExec.scala

@@ -53,7 +53,7 @@ case class ShowNamespacesExec(

    val rows = new ArrayBuffer[InternalRow]()
    namespaceNames.map { ns =>
-      if (pattern.map(StringUtils.filterPattern(Seq(ns), _).nonEmpty).getOrElse(true)) {
+      if (pattern.forall(StringUtils.filterPattern(Seq(ns), _).nonEmpty)) {


This is only a correction made based on the syntax prompted by the IDE.

panbingkun · 2024-02-28T08:57:28Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/StringUtils.scala

   * @param names the names list to be filtered
   * @param pattern the filter pattern, only '*' and '|' are allowed as wildcards, others will
   *                follow regular expression convention, case insensitive match and white spaces
   *                on both ends will be ignored
   * @return the filtered names list in order
   */
-  def filterPattern(names: Seq[String], pattern: String): Seq[String] = {
+  def filterPatternLegacy(names: Seq[String], pattern: String): Seq[String] = {


Only rename XXX to legacyXXX.

panbingkun · 2024-02-28T10:44:04Z

This is a hard decision. Technically the behavior of LIKE in many commands (SHOW TABLES LIKE ...) relies on the underlying catalog, which can be HMS of different versions, or a Hive-compatible metastore service. This is out of Spark's control.

From @panbingkun's investigation, the Hive behavior is actually very weird and different from other main-stream SQL systems (they follow the same behavior of the LIKE expression). Hive 4.0 also switches to the more common behavior.

There are some commands that we implement the LIKE filtering by our own, following the Hive behavior. Now we are in a hard position:

If we do nothing, then Spark's behavior of LIKE in commands is non-standard and different from other databases. We may also hit future behavior changes if we upgrade to Hive 4.0.

If we change the LIKE filtering behavior now, it's a breaking change, and also lead to inconsistent behaviors as some commands use Hive to do LIKE filtering.

cc @srielau

After efforts, all commands that support syntax Like <pattern> have been changed. By adding a configuration spark.sql.legacy.useVerticalBarAndStarAsWildcardsInLikePattern (default value false), when it is true, the wildcards supported by the pattern of Like are consistent with the semantics supported before Hive version 4 (use '*' for any character(s) and '|' for a choice as wildcards). If it is false, their behavior is consistent with the semantics of SQL Like (use '%' for any character(s) and '_' for a single character as wildcards), and the document has also been updated synchronously.

[SPARK-45880][SQL] Introduce a new TableCatalog.listTable overload th…

e418b3d

…at takes a pattern string for v2 catalog

github-actions bot added the SQL label Nov 10, 2023

panbingkun mentioned this pull request Nov 10, 2023

[SPARK-33393][SQL] Support SHOW TABLE EXTENDED in v2 #37588

Closed

panbingkun marked this pull request as ready for review November 10, 2023 09:17

fix UT

43bfe0f

github-actions bot added the Stale label Feb 19, 2024

cloud-fan reviewed Feb 19, 2024

View reviewed changes

cloud-fan approved these changes Feb 19, 2024

View reviewed changes

cloud-fan removed the Stale label Feb 19, 2024

panbingkun added 5 commits February 19, 2024 19:58

Merge branch 'master' into SPARK-45880

bf18462

update ShowTablesExtendedExec & ShowTablesExec

c51043f

update ref

55fa734

update TableCatalog ref

3fa8e9a

Merge branch 'master' into SPARK-45880

315c8b7

cloud-fan reviewed Feb 20, 2024

View reviewed changes

panbingkun added 2 commits February 21, 2024 15:44

Merge branch 'master' into SPARK-45880

48acc95

Merge branch 'master' into SPARK-45880

c414e8e

[SPARK-45880][SQL] Like pattern accept only SQL type like expressio…

596de52

…ns containing '%' for any character(s), and '_' for a single character

panbingkun changed the title ~~[SPARK-45880][SQL] Introduce a new TableCatalog.listTable overload th…~~ [WIP][SPARK-45880][SQL] Introduce a new TableCatalog.listTable overload th… Feb 26, 2024

panbingkun added 2 commits February 26, 2024 11:26

rebase master

6181d3c

fix some UT

4d382ef

github-actions bot added PYTHON CONNECT labels Feb 26, 2024

panbingkun added 3 commits February 27, 2024 16:18

fix ut

677f2c9

Merge branch 'master' into SPARK-45880

3992a7d

update comment & doc & ut

4b096f2

github-actions bot added the DOCS label Feb 27, 2024

panbingkun commented Feb 28, 2024

View reviewed changes

Merge branch 'master' into SPARK-45880

3a43818

panbingkun commented Feb 28, 2024

View reviewed changes

improve

2aeb620

panbingkun commented Feb 28, 2024

View reviewed changes

panbingkun added 2 commits February 28, 2024 17:23

fix style

7abf241

Merge branch 'master' into SPARK-45880

b276a3d

panbingkun changed the title ~~[WIP][SPARK-45880][SQL] Introduce a new TableCatalog.listTable overload th…~~ [SPARK-45880][SQL] Make the like pattern semantics used in all commands consistent Feb 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-45880][SQL] Make the like `pattern` semantics used in all commands consistent #43751

[SPARK-45880][SQL] Make the like `pattern` semantics used in all commands consistent #43751

panbingkun commented Nov 10, 2023 •

edited

panbingkun commented Nov 10, 2023

github-actions bot commented Feb 19, 2024

cloud-fan Feb 19, 2024

panbingkun Feb 19, 2024

cloud-fan Feb 19, 2024

cloud-fan Feb 19, 2024

panbingkun Feb 20, 2024

cloud-fan Feb 20, 2024

panbingkun Feb 20, 2024

panbingkun commented Feb 20, 2024 •

edited

cloud-fan commented Feb 20, 2024

panbingkun commented Feb 20, 2024

cloud-fan commented Feb 20, 2024

panbingkun commented Feb 22, 2024 •

edited

panbingkun Feb 28, 2024 •

edited

panbingkun Feb 28, 2024

panbingkun Feb 28, 2024

panbingkun Feb 28, 2024

cloud-fan commented Feb 28, 2024

panbingkun Feb 28, 2024

panbingkun Feb 28, 2024

panbingkun Feb 28, 2024

panbingkun Feb 28, 2024

panbingkun Feb 28, 2024

panbingkun Feb 28, 2024

panbingkun Feb 28, 2024

panbingkun commented Feb 28, 2024

[SPARK-45880][SQL] Make the like pattern semantics used in all commands consistent #43751

Are you sure you want to change the base?

[SPARK-45880][SQL] Make the like pattern semantics used in all commands consistent #43751

Conversation

panbingkun commented Nov 10, 2023 • edited

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

panbingkun commented Nov 10, 2023

github-actions bot commented Feb 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

panbingkun commented Feb 20, 2024 • edited

cloud-fan commented Feb 20, 2024

panbingkun commented Feb 20, 2024

cloud-fan commented Feb 20, 2024

panbingkun commented Feb 22, 2024 • edited

panbingkun Feb 28, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cloud-fan commented Feb 28, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

panbingkun commented Feb 28, 2024

[SPARK-45880][SQL] Make the like `pattern` semantics used in all commands consistent #43751

[SPARK-45880][SQL] Make the like `pattern` semantics used in all commands consistent #43751

panbingkun commented Nov 10, 2023 •

edited

panbingkun commented Feb 20, 2024 •

edited

panbingkun commented Feb 22, 2024 •

edited

panbingkun Feb 28, 2024 •

edited