[FLINK-8558] [FLINK-8866] [table] Finalize unified table source/sink/format interfaces #6323

twalthr · 2018-07-13T06:51:56Z

What is the purpose of the change

This PR finalizes the efforts done in #6264 and #6201 for having unified interfaces for table sources, table sinks, and table formats. It reduces code duplication and cleans up the code base around factories.

Brief change log

Introduction of org.apache.table.factories.TableFactory a common interface for factories
Introduction of org.apache.table.factories.TableFormatFactory a specific table factory for formats
Specific factories for StreamTableSource, StreamTableSink, BatchTableSource, BatchTableSink, DeserializationSchema, and SerializationSchema
Deprecation of old format-specific table sources (sinks will be deprecated in a follow-up PR)
Possibility to register table source and sink under a common name (table type both in SQL Client YAML)

Verifying this change

Existing tests verify the implementation
Additional ITCases and unit tests have been added
(An end-to-end test will follow in a separate PR)

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): yes
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? yes
If yes, how is the feature documented? not documented

…ts from connectors This PR introduces a format discovery mechanism based on Java Service Providers. The general `TableFormatFactory` is similar to the existing table source discovery mechanism. However, it allows for arbirary format interfaces that might be introduced in the future. At the moment, a connector can request configured instances of `DeserializationSchema` and `SerializationSchema`. In the future we can add interfaces such as a `Writer` or `KeyedSerializationSchema` without breaking backwards compatibility. This PR deprecates the existing strong coupling of connector and format for the Kafa table sources and table source factories. It introduces descriptor-based alternatives.

…tiate TableSinks This closes apache#6201.

pnowojski

Thanks for the update. With this refactor generally speaking it's looking much better :) Please clean up your 6 latests commits:

first commit Rename to TableFactory and move it to factories package is fine
Refactor format factories probably should be squashed with the first one
Unify table factories, Move table type out of descriptors and Make source/sink factories environment-dependent are more or less fine

However the last one Clean up and simplify changes is very strange. Especially parts that are removing code that this PR introduced couple of commits before. Changes there should be split and squashed with appropriate previous commits.

pnowojski · 2018-07-13T07:42:53Z

...r-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/KafkaTableSource.java

-	 * @param schema                Schema of the produced table.
-	 * @param returnType            Type information of the produced physical DataStream.
+	 * @param schema                      Schema of the produced table.
+	 * @param proctimeAttribute           Field name of the processing time attribute, null if no


null

Optional or overload or use builder or disallow null. Same applies to classes that are extending from this one.

pnowojski · 2018-07-13T07:43:03Z

...r-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/KafkaTableSource.java

+	 * @param proctimeAttribute           Field name of the processing time attribute, null if no
+	 *                                    processing time field is defined.
+	 * @param rowtimeAttributeDescriptors Descriptor for a rowtime attribute
+	 * @param fieldMapping                Mapping for the fields of the table schema to


or null

ditto

pnowojski · 2018-07-13T07:47:14Z

...ibraries/flink-table/src/main/scala/org/apache/flink/table/sources/DefinedFieldMapping.scala

@@ -50,7 +50,8 @@ trait DefinedFieldMapping {
    * type. It can also provide a mapping for fields which are not in the [[TableSchema]] to make
    * fields in the physical [[TypeInformation]] accessible for a [[TimestampExtractor]].
    *
-    * @return A mapping from [[TableSchema]] fields to [[TypeInformation]] fields.
+    * @return A mapping from [[TableSchema]] fields to [[TypeInformation]] fields or
+    *         null if no mapping is necessary.
    */
  def getFieldMapping: JMap[String, String]


annotate Nullable or change to Optional.

pnowojski · 2018-07-13T07:51:24Z

...r-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/KafkaTableSource.java

+	 * @param proctimeAttribute           Field name of the processing time attribute, null if no
+	 *                                    processing time field is defined.
+	 * @param rowtimeAttributeDescriptors Descriptor for a rowtime attribute
+	 * @param fieldMapping                Mapping for the fields of the table schema to


How can this field ever be null? SchemaValidator.deriveFieldMapping doesn't allow for that, does it?

Backward compatibility. It could have been null in the past.

pnowojski · 2018-07-13T07:51:51Z

...r-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/KafkaTableSource.java

 			startupMode == that.startupMode &&
 			Objects.equals(specificStartupOffsets, that.specificStartupOffsets);
 	}

 	@Override
 	public int hashCode() {
-		return Objects.hash(schema, topic, properties, returnType,
-			proctimeAttribute, rowtimeAttributeDescriptors, startupMode, specificStartupOffsets);
+		return Objects.hash(schema, proctimeAttribute, rowtimeAttributeDescriptors, fieldMapping,


nit: format one entry per line

pnowojski · 2018-07-13T13:37:51Z

flink-libraries/flink-table/src/main/scala/org/apache/flink/table/api/TableEnvironment.scala

@@ -465,14 +465,14 @@ abstract class TableEnvironment(val config: TableConfig) {
      tableSink: TableSink[_]): Unit

  /**
-    * Registers an external [[TableSink]] which is already configured in this
-    * [[TableEnvironment]]'s catalog.
+    * Registers an external [[TableSink]] with already configured field names and field types in


pnowojski · 2018-07-13T13:42:00Z

...e/src/main/scala/org/apache/flink/table/plan/nodes/logical/FlinkLogicalTableSourceScan.scala

-      }
-      case t => throw TableException(s"Unknown Table type ${t.getClass}.")
+      case t: TableSourceSinkTable[_, _] if t.isStreamSourceTable => true
+      // null


drop //null ?

This information is useful.

pnowojski · 2018-07-13T13:48:11Z

...ies/flink-table/src/main/scala/org/apache/flink/table/plan/schema/TableSourceSinkTable.scala


-class TableSourceSinkTable[T1, T2](val tableSourceTableOpt: Option[TableSourceTable[T1]],
-                                   val tableSinkTableOpt: Option[TableSinkTable[T2]])
+/**


ditto: fixup Shuyi's commit

pnowojski · 2018-07-13T13:51:55Z

...-libraries/flink-table/src/main/scala/org/apache/flink/table/sinks/CsvTableSinkFactory.scala

-
-    csvTableSinkBuilder
-      .build()
+    val path = params.getString(CONNECTOR_PATH)


ditto fixup

pnowojski · 2018-07-13T13:52:53Z

...table/src/test/scala/org/apache/flink/table/runtime/batch/table/TableEnvironmentITCase.scala

@@ -186,7 +186,7 @@ class TableEnvironmentITCase(
  def testInsertIntoMemoryTable(): Unit = {
    val env = ExecutionEnvironment.getExecutionEnvironment
    val tEnv = TableEnvironment.getTableEnvironment(env)
-    MemoryTableSourceSinkUtil.clear
+    MemoryTableSourceSinkUtil.clear()


ditto, and same applies probably to other changes in this commit :(

twalthr · 2018-07-14T06:07:01Z

Thank you @pnowojski. I hope I could address all your comments. I will clean the commit history and improve the commit messages during merging.

pnowojski

Couple minor comments, thanks for your effort! LGTM

pnowojski · 2018-07-14T17:39:51Z

...ibraries/flink-sql-client/src/main/java/org/apache/flink/table/client/config/SourceSink.java

@@ -51,16 +51,10 @@ public void addProperties(DescriptorProperties properties) {
 	}

 	public Source toSource() {
-		final Map<String, String> newProperties = new HashMap<>(properties);
-		newProperties.replace(TableDescriptorValidator.TABLE_TYPE(),
-				TableDescriptorValidator.TABLE_TYPE_VALUE_SOURCE());


I'm just asking because I can not find what code has replaced them. Or there were some dead code?

pnowojski · 2018-07-14T17:43:05Z

...braries/flink-table/src/main/scala/org/apache/flink/table/factories/TableFormatFactory.scala

-    * An empty context means that the factory matches for all requests.
-    */
-  def requiredContext(): util.Map[String, String]
+trait TableFormatFactory[T] extends TableFactory {


Maybe in that case deduplicate comment with @See java doc pointer? Otherwise there is huge risk of comments drifting out of sink.

pnowojski · 2018-07-14T17:45:46Z

...raries/flink-table/src/main/scala/org/apache/flink/table/factories/TableFactoryService.scala

+    * @return the matching factory
+    */
+  def find[T](factoryClass: Class[T], descriptor: Descriptor): T = {
+    Preconditions.checkNotNull(factoryClass)


But this leads to quite some unnecessary code duplication. checkNotNull(factoryClass) appears 5 times here and same applies to other params. Doing the check only in the place where you are touching value would either solve or at least limit this issue.

pnowojski · 2018-07-14T17:50:06Z

...raries/flink-table/src/main/scala/org/apache/flink/table/factories/TableFactoryService.scala

+      val plainContext = mutable.Map[String, String]()
+      plainContext ++= requestedContext
+      // we remove the version for now until we have the first backwards compatibility case
+      // with the version we can provide mappings in case the format changes


Can you create a follow up issue with blocker status for 1.6.0 so that it doesn't slip?

Usually it is very uncommon to define both a batch and streaming source in the same factory. Separating by environment is a concept that can be find throughout the entire flink-table module because both sources and sinks behave quite different per environment. This closes apache#6323.

twalthr · 2018-07-15T08:25:55Z

Thanks @pnowojski. Merging this...

Usually it is very uncommon to define both a batch and streaming source in the same factory. Separating by environment is a concept that can be find throughout the entire flink-table module because both sources and sinks behave quite different per environment. This closes apache#6323.

twalthr and others added 8 commits July 11, 2018 11:45

[FLINK-8866] [table] Create unified interfaces to configure and insta…

42a8a15

…tiate TableSinks This closes apache#6201.

Rename to TableFactory and move it to factories package

311dc62

Refactor format factories

1c581cb

Unify table factories

5c6df75

Move table type out of descriptors

0cd7c44

Make source/sink factories environment-dependent

6b83f2e

Clean up and simplify changes

4f1255f

twalthr mentioned this pull request Jul 13, 2018

[FLINK-8858] [sql-client] Add support for INSERT INTO in SQL Client #6332

Closed

pnowojski requested changes Jul 13, 2018

View reviewed changes

twalthr added 6 commits July 14, 2018 07:57

Replace nulls with Optional

256bad5

Add nullabel annotation

99a3f69

One entry per line

4d8aba0

Improve comment

312eb1b

Improve variable naming

834d553

Move exception

5d4a6ae

Improve TableFactoryService

a4785a9

twalthr force-pushed the UnifiedInterfacesFinal branch from b9e0e59 to a4785a9 Compare July 14, 2018 07:24

Remove unused import

9dc2cad

pnowojski approved these changes Jul 14, 2018

View reviewed changes

Last feedback addressed

72585a1

asfgit closed this in 0e5ac4d Jul 15, 2018

rmetzger added the component=API/TableSQL label Mar 18, 2019

flinkbot added component=TableSQL/API and removed component=API/TableSQL labels Mar 17, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FLINK-8558] [FLINK-8866] [table] Finalize unified table source/sink/format interfaces #6323

[FLINK-8558] [FLINK-8866] [table] Finalize unified table source/sink/format interfaces #6323

twalthr commented Jul 13, 2018

pnowojski left a comment

pnowojski Jul 13, 2018

pnowojski Jul 13, 2018

pnowojski Jul 13, 2018

pnowojski Jul 13, 2018 •

edited

Loading

twalthr Jul 14, 2018

pnowojski Jul 13, 2018 •

edited

Loading

pnowojski Jul 13, 2018

pnowojski Jul 13, 2018

twalthr Jul 14, 2018

pnowojski Jul 13, 2018

pnowojski Jul 13, 2018

pnowojski Jul 13, 2018 •

edited

Loading

twalthr commented Jul 14, 2018

pnowojski left a comment

pnowojski Jul 14, 2018

pnowojski Jul 14, 2018

pnowojski Jul 14, 2018

pnowojski Jul 14, 2018

twalthr commented Jul 15, 2018 •

edited

Loading

[FLINK-8558] [FLINK-8866] [table] Finalize unified table source/sink/format interfaces #6323

[FLINK-8558] [FLINK-8866] [table] Finalize unified table source/sink/format interfaces #6323

Conversation

twalthr commented Jul 13, 2018

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

pnowojski left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pnowojski Jul 13, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pnowojski Jul 13, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pnowojski Jul 13, 2018 • edited Loading

Choose a reason for hiding this comment

twalthr commented Jul 14, 2018

pnowojski left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

twalthr commented Jul 15, 2018 • edited Loading

pnowojski Jul 13, 2018 •

edited

Loading

pnowojski Jul 13, 2018 •

edited

Loading

pnowojski Jul 13, 2018 •

edited

Loading

twalthr commented Jul 15, 2018 •

edited

Loading