[FLINK-27376][sql] Support current_database function #19218

luoyuxia · 2022-03-24T06:34:00Z

What is the purpose of the change

To support current_database() function in Flink

Brief change log

Add CURRENT_DATABASE definition in BuiltInFunctionDefinitions.java and FlinkSqlOperatorTable.java
Add method currentDatabase to call CURRENT_DATABASE in Expressions.java to make current_database can be used in TableAPI.
Before translation, set a configuration with value current database in PlannerBase. And read the configuration value put to get the current in codegen phase for current_database() function.
Disable current_database in HiveMoudle, so that it'll always use Flink's built-in current_database function.

Verifying this change

Added test in testCurrentDatabase, testCurrentDatabase, testCurrentDatabase, testCurrentDatabase for batch/stream, sql/table.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? Yes
If yes, how is the feature documented? Yes

flinkbot · 2022-03-24T06:40:02Z

CI report:

3c137ae Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run azure re-run the last Azure build

fsk119

Thanks for your contribution. I am not sure whether user with hive module in default dialect still get the same problem.

fsk119 · 2022-06-30T09:21:52Z

...-hive/src/main/java/org/apache/flink/table/planner/delegation/hive/SqlFunctionConverter.java

-                } catch (SemanticException e) {
-                    throw new FlinkHiveException(e);
-                }
+                return convertToLiteral(hiveShim.toHiveTimestamp(currentTS));


It seems we can use

if (convertedOp instanceof SqlCastFunction) { } else if (convertedOp instanceof FlinkSqlTimestampFunction) { } else if (convertedOp.getName().equals("current_database")) { } else { return builder.makeCall(convertedOp, visitList(operands, update)); }

fsk119 · 2022-06-30T09:34:24Z

...-hive/src/main/java/org/apache/flink/table/planner/delegation/hive/SqlFunctionConverter.java

-                    throw new FlinkHiveException(e);
-                }
+                return convertToLiteral(hiveShim.toHiveTimestamp(currentTS));
+            } else if (convertedOp.getName().equals("current_database")) {


I think it's very hacked to fix this problem in this way. Currently, we determine which function is used by comparing the method name, parameters, and return type. But here we only consider the function name.

I think we should introduce a class similar to SqlFunctionCast here and use instanceOf to determine.

fsk119 · 2022-06-30T10:08:17Z

...nk-connector-hive/src/test/java/org/apache/flink/connectors/hive/HiveDialectQueryITCase.java

+                        tableEnv.executeSql("select current_database()").collect());
+        assertThat(result.toString()).isEqualTo("[+I[db1]]");
+        // switch to default database for following test use default database
+        tableEnv.executeSql("use default");


It's better if we can use

@After public void cleanup() { tableEnv.executeSql("use default"); }

I think we don't need to add a cleanup method for currently only one test needs it. It would be better to limit use default in this single test.

luoyuxia · 2022-07-05T04:11:58Z

@fsk119 Thanks for review. Now, I add function current_database for flink itselft. So that both Flink sql and Hive dialect can use it.

fsk119

Thanks for update. It's better we can keep the origin tests in the hive.

fsk119 · 2022-07-05T08:45:22Z

...-table-common/src/main/java/org/apache/flink/table/functions/BuiltInFunctionDefinitions.java

+    // --------------------------------------------------------------------------------------------
+    // Catalog functions
+    // --------------------------------------------------------------------------------------------
+    public static final BuiltInFunctionDefinition CURRENT_DATABASE =


Leave a blank line before the CURRENT_DATABASE. We should align with others.

fsk119 · 2022-07-05T08:52:41Z

...able-planner/src/main/scala/org/apache/flink/table/planner/codegen/calls/StringCallGen.scala

@@ -233,6 +233,10 @@ object StringCallGen {
            isCharacterString(operands(2).resultType) =>
        methodGen(BuiltInMethods.CONVERT_TZ)

+      case CURRENT_DATABASE =>
+        val timestamp = ctx.addReusableQueryLevelCurrentDatabase()


timestamp -> database

fsk119 · 2022-07-05T08:59:29Z

...-table-planner/src/main/java/org/apache/flink/table/planner/utils/InternalConfigOptions.java

@@ -52,6 +52,13 @@ public final class InternalConfigOptions {
                                    + " some temporal functions like LOCAL_TIMESTAMP in batch job to make sure these"
                                    + " temporal functions has query-start semantics.");

+    public static final ConfigOption<String> TABLE_QUERY_CURRENT_DATABASE =
+            key("__table.query-start.current_database__")


Other options use the "-" rather than "_". How about renaming to __table.query-start.current-database__?

fsk119 · 2022-07-05T09:00:40Z

...-table-planner/src/main/java/org/apache/flink/table/planner/utils/InternalConfigOptions.java

+                    .stringType()
+                    .noDefaultValue()
+                    .withDescription(
+                            "The config used to save the current database at query start.");


Also, document who will use the value?

luoyuxia · 2022-07-06T11:54:37Z

@fsk119 Thanks for reviewing. I have updated it.

fsk119

LGTM.

But the CI test fails. Could you take a look?

luoyuxia · 2022-07-11T01:29:10Z

@flinkbot run azure

luoyuxia · 2022-07-11T09:26:26Z

@fsk119 The ci is passed now. Could you please help merge when you're free?

…he#19218)

luoyuxia force-pushed the support-current-database branch from 2a63613 to 9590059 Compare April 2, 2022 11:59

luoyuxia changed the title ~~hive dialect supports select current database~~ [FLINK-27376][hive] Hive dialect supports select current database Apr 25, 2022

luoyuxia changed the title ~~[FLINK-27376][hive] Hive dialect supports select current database~~ [FLINK-27376][hive] Hive dialect supports select current_database() Apr 25, 2022

luoyuxia force-pushed the support-current-database branch from 9590059 to 455edd9 Compare April 25, 2022 04:23

flinkbot added the component=<none> label Apr 25, 2022

luoyuxia force-pushed the support-current-database branch 2 times, most recently from 9fdaa74 to 2ddd919 Compare June 22, 2022 12:05

fsk119 reviewed Jun 30, 2022

View reviewed changes

luoyuxia changed the title ~~[FLINK-27376][hive] Hive dialect supports select current_database()~~ [FLINK-27376][sql] Support current_database() function Jul 5, 2022

luoyuxia force-pushed the support-current-database branch from 2ddd919 to 72c7f36 Compare July 5, 2022 04:01

luoyuxia changed the title ~~[FLINK-27376][sql] Support current_database() function~~ [FLINK-27376][sql] Support current_database as built-in function Jul 5, 2022

luoyuxia changed the title ~~[FLINK-27376][sql] Support current_database as built-in function~~ [FLINK-27376][sql] Support current_database function Jul 5, 2022

luoyuxia force-pushed the support-current-database branch 2 times, most recently from 22d297c to 163bc2d Compare July 5, 2022 04:19

fsk119 reviewed Jul 5, 2022

View reviewed changes

luoyuxia force-pushed the support-current-database branch from f834d97 to b93e796 Compare July 6, 2022 11:49

fsk119 approved these changes Jul 7, 2022

View reviewed changes

luoyuxia force-pushed the support-current-database branch 4 times, most recently from 520d407 to e6f8f44 Compare July 8, 2022 01:51

luoyuxia and others added 3 commits July 8, 2022 13:20

[FLINK-27376][sql] Support select current_database()

e342141

fix comments

3bf368b

fix test failure

3c137ae

luoyuxia force-pushed the support-current-database branch from e6f8f44 to 3c137ae Compare July 8, 2022 05:21

fsk119 merged commit 641fb89 into apache:master Jul 14, 2022

liujiawinds pushed a commit to liujiawinds/flink that referenced this pull request Jul 22, 2022

[FLINK-27376][table] Support current_database built-in function (apac…

c54701b

…he#19218)

huangxiaofeng10047 pushed a commit to huangxiaofeng10047/flink that referenced this pull request Nov 3, 2022

[FLINK-27376][table] Support current_database built-in function (apac…

b6ef116

…he#19218)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FLINK-27376][sql] Support current_database function #19218

[FLINK-27376][sql] Support current_database function #19218

luoyuxia commented Mar 24, 2022 •

edited

flinkbot commented Mar 24, 2022 •

edited

fsk119 left a comment

fsk119 Jun 30, 2022

fsk119 Jun 30, 2022

fsk119 Jun 30, 2022

luoyuxia Jul 6, 2022 •

edited

luoyuxia commented Jul 5, 2022

fsk119 left a comment

fsk119 Jul 5, 2022

fsk119 Jul 5, 2022

fsk119 Jul 5, 2022

fsk119 Jul 5, 2022

luoyuxia commented Jul 6, 2022

fsk119 left a comment

luoyuxia commented Jul 11, 2022

luoyuxia commented Jul 11, 2022

[FLINK-27376][sql] Support current_database function #19218

[FLINK-27376][sql] Support current_database function #19218

Conversation

luoyuxia commented Mar 24, 2022 • edited

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

flinkbot commented Mar 24, 2022 • edited

CI report:

fsk119 left a comment

Choose a reason for hiding this comment

fsk119 Jun 30, 2022

Choose a reason for hiding this comment

fsk119 Jun 30, 2022

Choose a reason for hiding this comment

fsk119 Jun 30, 2022

Choose a reason for hiding this comment

luoyuxia Jul 6, 2022 • edited

Choose a reason for hiding this comment

luoyuxia commented Jul 5, 2022

fsk119 left a comment

Choose a reason for hiding this comment

fsk119 Jul 5, 2022

Choose a reason for hiding this comment

fsk119 Jul 5, 2022

Choose a reason for hiding this comment

fsk119 Jul 5, 2022

Choose a reason for hiding this comment

fsk119 Jul 5, 2022

Choose a reason for hiding this comment

luoyuxia commented Jul 6, 2022

fsk119 left a comment

Choose a reason for hiding this comment

luoyuxia commented Jul 11, 2022

luoyuxia commented Jul 11, 2022

luoyuxia commented Mar 24, 2022 •

edited

flinkbot commented Mar 24, 2022 •

edited

luoyuxia Jul 6, 2022 •

edited