[FLINK-10597][table] Enabled UDFs support in MATCH_RECOGNIZE #7189

dawidwys · 2018-11-28T14:51:17Z

What is the purpose of the change

This PR enables user defined scalar function in MATCH_RECOGNIZE

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
The serializers: (yes / no / don't know)
The runtime per-record code paths (performance sensitive): (yes / no / don't know)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / no / don't know)
The S3 file system connector: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

twalthr

Thank you @dawidwys. I added a couple minor comments.

twalthr · 2018-11-29T08:54:34Z

...libraries/flink-table/src/main/scala/org/apache/flink/table/codegen/MatchCodeGenerator.scala

-      if (clazz == classOf[IterativeCondition[_]]) {
-        val baseClass = classOf[IterativeCondition[_]]
+    val (functionClass, signature, inputStatements) =
+      if (clazz == classOf[IterativeCondition[_]] || clazz == classOf[RichIterativeCondition[_]]) {


Is there a reason to support both interface and rich class? I would keep it simple here and only support the rich variant.

twalthr · 2018-11-29T08:58:51Z

...ink-table/src/main/scala/org/apache/flink/table/runtime/match/IterativeConditionRunner.scala

    LOG.debug("Instantiating IterativeCondition.")
    function = clazz.newInstance()
-    // TODO add logic for opening and closing the function once it can be a RichFunction
+    FunctionUtils.setFunctionRuntimeContext(function, getRuntimeContext)


Is there a reason why a distributed cache is not supported in a CepRuntimeContext? We don't expose many context properties in SQL UDFs but the distributed cache apparently seems to be important.

I don't think there is any particular reason. Just wanted to maximally limit the scope. If it is important though, I will change that.

twalthr · 2018-11-29T09:00:00Z

...ink-table/src/main/scala/org/apache/flink/table/runtime/match/IterativeConditionRunner.scala

-      init()
-    }
+  override def close(): Unit = {
+    super.close()


Remove this line?

twalthr · 2018-11-29T09:02:28Z

...-table/src/main/scala/org/apache/flink/table/runtime/match/PatternSelectFunctionRunner.scala

-      init()
-    }
+  override def close(): Unit = {
+    super.close()


Remove this line?

twalthr · 2018-11-29T09:03:07Z

...-table/src/main/scala/org/apache/flink/table/runtime/match/PatternSelectFunctionRunner.scala

-    }
+  override def close(): Unit = {
+    super.close()
+    function.close()


Use org.apache.flink.api.common.functions.util.FunctionUtils#closeFunction instead?

twalthr · 2018-11-29T09:03:14Z

...ink-table/src/main/scala/org/apache/flink/table/runtime/match/IterativeConditionRunner.scala

-    }
+  override def close(): Unit = {
+    super.close()
+    function.close()


Use org.apache.flink.api.common.functions.util.FunctionUtils#closeFunction instead?

twalthr · 2018-11-29T09:06:02Z

...nk-table/src/test/scala/org/apache/flink/table/runtime/stream/sql/MatchRecognizeITCase.scala

@@ -543,10 +543,79 @@ class MatchRecognizeITCase extends StreamingWithStateTestBase {
    // We do not assert the proctime in the result, cause it is currently


I guess this problem has not been solved with the new rich function?

Unfortunately not, there is a separate issue for this: https://issues.apache.org/jira/browse/FLINK-10596

We just introduced a new rich class. I think it would also be a perfect time to think about this shortcoming.

twalthr · 2018-11-29T09:07:37Z

...nk-table/src/test/scala/org/apache/flink/table/runtime/stream/sql/MatchRecognizeITCase.scala

@@ -543,10 +543,79 @@ class MatchRecognizeITCase extends StreamingWithStateTestBase {
    // We do not assert the proctime in the result, cause it is currently
    // accessed from System.currentTimeMillis(), so there is no graceful way to assert the proctime
  }
+
+  @Test
+  def testRichUdfs(): Unit = {


nit: There are no RichUdfs just UserDefinedFunctions ;-)

twalthr · 2018-11-29T09:11:59Z

...nk-table/src/test/scala/org/apache/flink/table/runtime/stream/sql/MatchRecognizeITCase.scala

 }

 class ToMillis extends ScalarFunction {
  def eval(t: Timestamp): Long = {
    t.toInstant.toEpochMilli + TimeZone.getDefault.getOffset(t.toInstant.toEpochMilli)
  }
 }
+
+private class RichScalarFunc extends ScalarFunction {


Give more meaningful name.

twalthr · 2018-11-29T09:12:38Z

...nk-table/src/test/scala/org/apache/flink/table/runtime/stream/sql/MatchRecognizeITCase.scala

+  }
+
+  override def close(): Unit = {
+    prefix = "ERROR_VALUE"


This line has no meaning so far. Remove the entire method?

dianfu

@dawidwys Thanks a lot for the PR. Have left a few comments.

dianfu · 2018-11-29T09:27:59Z

...libraries/flink-table/src/main/scala/org/apache/flink/table/codegen/MatchCodeGenerator.scala

@@ -131,47 +132,39 @@ class MatchCodeGenerator(
    : GeneratedFunction[F, T] = {


We can already reuse FunctionCodeGenerator.generateFunction() as IterativeCondition and Pattern(Flat)SelectFunction already support rich interfaces.

There is one additional obstacle right now. The reusePatternLists() call, I would remove the comment and leave it separated, as there might be some additional differences in the future. What do you think?

We can add something like reusablePrepareInputStatements in CodeGenerator and place it before reuseInputUnboxingCode when generating the function. Then we can make use of reusablePrepareInputStatements to construct the reusePatternLists. But I'm fine with the current implementation as reusablePrepareInputStatements is currently only useful for CEP.

dianfu · 2018-11-29T09:38:22Z

...-table/src/main/scala/org/apache/flink/table/runtime/match/PatternSelectFunctionRunner.scala

@@ -35,20 +36,26 @@ import org.apache.flink.util.Collector
 class PatternSelectFunctionRunner(
    name: String,
    code: String)
-  extends PatternFlatSelectFunction[Row, CRow]
-  with Compiler[PatternSelectFunction[Row, Row]]
+  extends RichPatternFlatSelectFunction[Row, CRow]


Is there any reason we use a RichPatternFlatSelectFunction which actually wraps a RichPatternSelectFunction?

We need access to the collector to be able to erase timestamp.

OK. Make sense.

@dawidwys I have rethought about this issue and what about always generating a PatternFlatSelectFunction no matter ONE ROW PER MATCH or ALL ROWS PER MATCH is specified?

Agree, had that in mind. I would change that either during introducing the ALL ROWS PER MATCH or as a separate issue, as it is not directly connected with this one and requires more changes in the code generator.

twalthr

Thanks for the update @dawidwys. Feel free to merge this.

dawidwys changed the title ~~[FLINK-10597] Enabled UDFs support in MATCH_RECOGNIZE~~ [FLINK-10597][table] Enabled UDFs support in MATCH_RECOGNIZE Nov 28, 2018

twalthr self-assigned this Nov 28, 2018

dawidwys mentioned this pull request Nov 28, 2018

[FLINK-7599] [table] Support for aggregates in MATCH_RECOGNIZE #7177

Merged

twalthr requested changes Nov 29, 2018

View reviewed changes

dianfu reviewed Nov 29, 2018

View reviewed changes

twalthr approved these changes Nov 30, 2018

View reviewed changes

dawidwys added 2 commits November 30, 2018 14:38

[hotfix][cep] Enabled distributed cache for CEP functions

30befd7

[FLINK-10597] Enabled UDFs support in MATCH_RECOGNIZE

375651e

dawidwys force-pushed the flink10597 branch from 9a747ab to 375651e Compare November 30, 2018 13:39

dawidwys merged commit d985eff into apache:master Nov 30, 2018

dawidwys deleted the flink10597 branch November 30, 2018 15:13

rmetzger added component=API/TableSQL component=Library/CEP labels Mar 18, 2019

flinkbot added component=TableSQL/API and removed component=API/TableSQL labels Mar 17, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FLINK-10597][table] Enabled UDFs support in MATCH_RECOGNIZE #7189

[FLINK-10597][table] Enabled UDFs support in MATCH_RECOGNIZE #7189

dawidwys commented Nov 28, 2018

twalthr left a comment

twalthr Nov 29, 2018

twalthr Nov 29, 2018

dawidwys Nov 29, 2018

twalthr Nov 29, 2018

twalthr Nov 29, 2018

twalthr Nov 29, 2018

twalthr Nov 29, 2018

twalthr Nov 29, 2018

dawidwys Nov 29, 2018

twalthr Nov 29, 2018

twalthr Nov 29, 2018

twalthr Nov 29, 2018

twalthr Nov 29, 2018

dianfu left a comment

dianfu Nov 29, 2018

dawidwys Nov 29, 2018

dianfu Nov 29, 2018

dianfu Nov 29, 2018

dawidwys Nov 29, 2018

dianfu Nov 29, 2018

dianfu Nov 30, 2018

dawidwys Nov 30, 2018

twalthr left a comment

		@@ -543,10 +543,79 @@ class MatchRecognizeITCase extends StreamingWithStateTestBase {
		// We do not assert the proctime in the result, cause it is currently

		@@ -131,47 +132,39 @@ class MatchCodeGenerator(
		: GeneratedFunction[F, T] = {

[FLINK-10597][table] Enabled UDFs support in MATCH_RECOGNIZE #7189

[FLINK-10597][table] Enabled UDFs support in MATCH_RECOGNIZE #7189

Conversation

dawidwys commented Nov 28, 2018

What is the purpose of the change

Does this pull request potentially affect one of the following parts:

Documentation

twalthr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dianfu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

twalthr left a comment

Choose a reason for hiding this comment