-
{%- if this.nav-show_overview %}
-
diff --git a/docs/concepts/flink-architecture.md b/docs/concepts/flink-architecture.md
index 8414943fd167d5..578e809ee88f87 100644
--- a/docs/concepts/flink-architecture.md
+++ b/docs/concepts/flink-architecture.md
@@ -24,50 +24,85 @@ specific language governing permissions and limitations
under the License.
-->
-* This will be replaced by the TOC
-{:toc}
-
-## Flink Applications and Flink Sessions
+Flink is a distributed system and requires effective allocation and management
+of compute resources in order to execute streaming applications. It integrates
+with all common cluster resource managers such as [Hadoop
+YARN](https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/YARN.html),
+[Apache Mesos](https://mesos.apache.org/) and
+[Kubernetes](https://kubernetes.io/), but can also be set up to run as a
+standalone cluster or even as a library.
-`TODO: expand this section`
+This section contains an overview of Flink’s architecture and describes how its
+main components interact to execute applications and recover from failures.
-{% top %}
+* This will be replaced by the TOC
+{:toc}
## Anatomy of a Flink Cluster
-`TODO: expand this section, especially about components of the Flink Master and
-container environments`
-
-The Flink runtime consists of two types of processes:
-
- - The *Flink Master* coordinates the distributed execution. It schedules
- tasks, coordinates checkpoints, coordinates recovery on failures, etc.
+The Flink runtime consists of two types of processes: a _JobManager_ and one or more _TaskManagers_.
- There is always at least one *Flink Master*. A high-availability setup
- might have multiple *Flink Masters*, one of which is always the
- *leader*, and the others are *standby*.
+
- - The *TaskManagers* (also called *workers*) execute the *tasks* (or more
- specifically, the subtasks) of a dataflow, and buffer and exchange the data
- *streams*.
+The *Client* is not part of the runtime and program execution, but is used to
+prepare and send a dataflow to the JobManager. After that, the client can
+disconnect (_detached mode_), or stay connected to receive progress reports
+(_attached mode_). The client runs either as part of the Java/Scala program
+that triggers the execution, or in the command line process `./bin/flink run
+...`.
- There must always be at least one TaskManager.
-
-The Flink Master and TaskManagers can be started in various ways: directly on
+The JobManager and TaskManagers can be started in various ways: directly on
the machines as a [standalone cluster]({% link
ops/deployment/cluster_setup.md %}), in containers, or managed by resource
frameworks like [YARN]({% link ops/deployment/yarn_setup.md
%}) or [Mesos]({% link ops/deployment/mesos.md %}).
-TaskManagers connect to Flink Masters, announcing themselves as available, and
+TaskManagers connect to JobManagers, announcing themselves as available, and
are assigned work.
-The *client* is not part of the runtime and program execution, but is used to
-prepare and send a dataflow to the Flink Master. After that, the client can
-disconnect, or stay connected to receive progress reports. The client runs
-either as part of the Java/Scala program that triggers the execution, or in the
-command line process `./bin/flink run ...`.
+### JobManager
+
+The _JobManager_ has a number of responsibilities related to coordinating the distributed execution of Flink Applications:
+it decides when to schedule the next task (or set of tasks), reacts to finished
+tasks or execution failures, coordinates checkpoints, and coordinates recovery on
+failures, among others. This process consists of three different components:
-
+ * **ResourceManager**
+
+ The _ResourceManager_ is responsible for resource de-/allocation and
+ provisioning in a Flink cluster — it manages **task slots**, which are the
+ unit of resource scheduling in a Flink cluster (see [TaskManagers](#taskmanagers)).
+ Flink implements multiple ResourceManagers for different environments and
+ resource providers such as YARN, Mesos, Kubernetes and standalone
+ deployments. In a standalone setup, the ResourceManager can only distribute
+ the slots of available TaskManagers and cannot start new TaskManagers on
+ its own.
+
+ * **Dispatcher**
+
+ The _Dispatcher_ provides a REST interface to submit Flink applications for
+ execution and starts a new JobManager for each submitted job. It
+ also runs the Flink WebUI to provide information about job executions.
+
+ * **JobMaster**
+
+ A _JobMaster_ is responsible for managing the execution of a single
+ [JobGraph]({% link concepts/glossary.md %}#logical-graph).
+ Multiple jobs can run simultaneously in a Flink cluster, each having its
+ own JobMaster.
+
+There is always at least one JobManager. A high-availability setup might have
+multiple JobManagers, one of which is always the *leader*, and the others are
+*standby* (see [High Availability (HA)]({% link ops/jobmanager_high_availability.md %})).
+
+### TaskManagers
+
+The *TaskManagers* (also called *workers*) execute the tasks of a dataflow, and buffer and exchange the data
+streams.
+
+There must always be at least one TaskManager. The smallest unit of resource scheduling in a TaskManager is a task _slot_. The number of task slots in a
+TaskManager indicates the number of concurrent processing tasks. Note that
+multiple operators may execute in a task slot (see [Tasks and Operator
+Chains](#tasks-and-operator-chains)).
{% top %}
@@ -78,8 +113,7 @@ For distributed execution, Flink *chains* operator subtasks together into
tasks is a useful optimization: it reduces the overhead of thread-to-thread
handover and buffering, and increases overall throughput while decreasing
latency. The chaining behavior can be configured; see the [chaining docs]({%
-link dev/stream/operators/index.md %}#task-chaining-and-resource-groups) for
-details.
+link dev/stream/operators/index.md %}#task-chaining-and-resource-groups) for details.
The sample dataflow in the figure below is executed with five subtasks, and
hence with five parallel threads.
@@ -91,8 +125,8 @@ hence with five parallel threads.
## Task Slots and Resources
Each worker (TaskManager) is a *JVM process*, and may execute one or more
-subtasks in separate threads. To control how many tasks a worker accepts, a
-worker has so called **task slots** (at least one).
+subtasks in separate threads. To control how many tasks a TaskManager accepts, it
+has so called **task slots** (at least one).
Each *task slot* represents a fixed subset of resources of the TaskManager. A
TaskManager with three slots, for example, will dedicate 1/3 of its managed
@@ -129,4 +163,90 @@ two main benefits:
+## Flink Application Execution
+
+A _Flink Application_ is any user program that spawns one or multiple Flink
+jobs from its ``main()`` method. The execution of these jobs can happen in a
+local JVM (``LocalEnvironment``) or on a remote setup of clusters with multiple
+machines (``RemoteEnvironment``). For each program, the
+[``ExecutionEnvironment``]({{ site.baseurl }}/api/java/) provides methods to
+control the job execution (e.g. setting the parallelism) and to interact with
+the outside world (see [Anatomy of a Flink Program]({%
+link dev/datastream_api.md %}#anatomy-of-a-flink-program)).
+
+The jobs of a Flink Application can either be submitted to a long-running
+[Flink Session Cluster]({%
+link concepts/glossary.md %}#flink-session-cluster), a dedicated [Flink Job
+Cluster]({% link concepts/glossary.md %}#flink-job-cluster), or a
+[Flink Application Cluster]({%
+link concepts/glossary.md %}#flink-application-cluster). The difference between
+these options is mainly related to the cluster’s lifecycle and to resource
+isolation guarantees.
+
+### Flink Session Cluster
+
+* **Cluster Lifecycle**: in a Flink Session Cluster, the client connects to a
+ pre-existing, long-running cluster that can accept multiple job submissions.
+ Even after all jobs are finished, the cluster (and the JobManager) will
+ keep running until the session is manually stopped. The lifetime of a Flink
+ Session Cluster is therefore not bound to the lifetime of any Flink Job.
+
+* **Resource Isolation**: TaskManager slots are allocated by the
+ ResourceManager on job submission and released once the job is finished.
+ Because all jobs are sharing the same cluster, there is some competition for
+ cluster resources — like network bandwidth in the submit-job phase. One
+ limitation of this shared setup is that if one TaskManager crashes, then all
+ jobs that have tasks running on this TaskManager will fail; in a similar way, if
+ some fatal error occurs on the JobManager, it will affect all jobs running
+ in the cluster.
+
+* **Other considerations**: having a pre-existing cluster saves a considerable
+ amount of time applying for resources and starting TaskManagers. This is
+ important in scenarios where the execution time of jobs is very short and a
+ high startup time would negatively impact the end-to-end user experience — as
+ is the case with interactive analysis of short queries, where it is desirable
+ that jobs can quickly perform computations using existing resources.
+
+Note: Formerly, a Flink Session Cluster was also known as a Flink Cluster in session mode.+ +### Flink Job Cluster + +* **Cluster Lifecycle**: in a Flink Job Cluster, the available cluster manager + (like YARN or Kubernetes) is used to spin up a cluster for each submitted job + and this cluster is available to that job only. Here, the client first + requests resources from the cluster manager to start the JobManager and + submits the job to the Dispatcher running inside this process. TaskManagers + are then lazily allocated based on the resource requirements of the job. Once + the job is finished, the Flink Job Cluster is torn down. + +* **Resource Isolation**: a fatal error in the JobManager only affects the one job running in that Flink Job Cluster. + +* **Other considerations**: because the ResourceManager has to apply and wait + for external resource management components to start the TaskManager + processes and allocate resources, Flink Job Clusters are more suited to large + jobs that are long-running, have high-stability requirements and are not + sensitive to longer startup times. + +Note: Formerly, a Flink Job Cluster was also known as a Flink Cluster in job (or per-job) mode.+ +### Flink Application Cluster + +* **Cluster Lifecycle**: a Flink Application Cluster is a dedicated Flink + cluster that only executes jobs from one Flink Application and where the + ``main()`` method runs on the cluster rather than the client. The job + submission is a one-step process: you don’t need to start a Flink cluster + first and then submit a job to the existing cluster session; instead, you + package your application logic and dependencies into a executable job JAR and + the cluster entrypoint (``ApplicationClusterEntryPoint``) + is responsible for calling the ``main()`` method to extract the JobGraph. + This allows you to deploy a Flink Application like any other application on + Kubernetes, for example. The lifetime of a Flink Application Cluster is + therefore bound to the lifetime of the Flink Application. + +* **Resource Isolation**: in a Flink Application Cluster, the ResourceManager + and Dispatcher are scoped to a single Flink Application, which provides a + better separation of concerns than the Flink Session Cluster. + +Note: A Flink Job Cluster can be seen as a “run-on-client” alternative to Flink Application Clusters.+ {% top %} diff --git a/docs/concepts/flink-architecture.zh.md b/docs/concepts/flink-architecture.zh.md new file mode 100644 index 00000000000000..1615c148047207 --- /dev/null +++ b/docs/concepts/flink-architecture.zh.md @@ -0,0 +1,252 @@ +--- +title: Flink 架构 +nav-id: flink-architecture +nav-pos: 4 +nav-title: Flink 架构 +nav-parent_id: concepts +--- + + +Flink is a distributed system and requires effective allocation and management +of compute resources in order to execute streaming applications. It integrates +with all common cluster resource managers such as [Hadoop +YARN](https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/YARN.html), +[Apache Mesos](https://mesos.apache.org/) and +[Kubernetes](https://kubernetes.io/), but can also be set up to run as a +standalone cluster or even as a library. + +This section contains an overview of Flink’s architecture and describes how its +main components interact to execute applications and recover from failures. + +* This will be replaced by the TOC +{:toc} + +## Anatomy of a Flink Cluster + +The Flink runtime consists of two types of processes: a _JobManager_ and one or more _TaskManagers_. + +
+
+The *Client* is not part of the runtime and program execution, but is used to
+prepare and send a dataflow to the JobManager. After that, the client can
+disconnect (_detached mode_), or stay connected to receive progress reports
+(_attached mode_). The client runs either as part of the Java/Scala program
+that triggers the execution, or in the command line process `./bin/flink run
+...`.
+
+The JobManager and TaskManagers can be started in various ways: directly on
+the machines as a [standalone cluster]({% link
+ops/deployment/cluster_setup.zh.md %}), in containers, or managed by resource
+frameworks like [YARN]({% link ops/deployment/yarn_setup.zh.md
+%}) or [Mesos]({% link ops/deployment/mesos.zh.md %}).
+TaskManagers connect to JobManagers, announcing themselves as available, and
+are assigned work.
+
+### JobManager
+
+The _JobManager_ has a number of responsibilities related to coordinating the distributed execution of Flink Applications:
+it decides when to schedule the next task (or set of tasks), reacts to finished
+tasks or execution failures, coordinates checkpoints, and coordinates recovery on
+failures, among others. This process consists of three different components:
+
+ * **ResourceManager**
+
+ The _ResourceManager_ is responsible for resource de-/allocation and
+ provisioning in a Flink cluster — it manages **task slots**, which are the
+ unit of resource scheduling in a Flink cluster (see [TaskManagers](#taskmanagers)).
+ Flink implements multiple ResourceManagers for different environments and
+ resource providers such as YARN, Mesos, Kubernetes and standalone
+ deployments. In a standalone setup, the ResourceManager can only distribute
+ the slots of available TaskManagers and cannot start new TaskManagers on
+ its own.
+
+ * **Dispatcher**
+
+ The _Dispatcher_ provides a REST interface to submit Flink applications for
+ execution and starts a new JobManager for each submitted job. It
+ also runs the Flink WebUI to provide information about job executions.
+
+ * **JobMaster**
+
+ A _JobMaster_ is responsible for managing the execution of a single
+ [JobGraph]({% link concepts/glossary.zh.md %}#logical-graph).
+ Multiple jobs can run simultaneously in a Flink cluster, each having its
+ own JobMaster.
+
+There is always at least one JobManager. A high-availability setup might have
+multiple JobManagers, one of which is always the *leader*, and the others are
+*standby* (see [High Availability (HA)]({% link ops/jobmanager_high_availability.zh.md %})).
+
+### TaskManagers
+
+The *TaskManagers* (also called *workers*) execute the tasks of a dataflow, and buffer and exchange the data
+streams.
+
+There must always be at least one TaskManager. The smallest unit of resource scheduling in a TaskManager is a task _slot_. The number of task slots in a
+TaskManager indicates the number of concurrent processing tasks. Note that
+multiple operators may execute in a task slot (see [Tasks and Operator
+Chains](#tasks-and-operator-chains)).
+
+{% top %}
+
+## Tasks and Operator Chains
+
+For distributed execution, Flink *chains* operator subtasks together into
+*tasks*. Each task is executed by one thread. Chaining operators together into
+tasks is a useful optimization: it reduces the overhead of thread-to-thread
+handover and buffering, and increases overall throughput while decreasing
+latency. The chaining behavior can be configured; see the [chaining docs]({%
+link dev/stream/operators/index.zh.md %}#task-chaining-and-resource-groups) for details.
+
+The sample dataflow in the figure below is executed with five subtasks, and
+hence with five parallel threads.
+
+
+
+{% top %}
+
+## Task Slots and Resources
+
+Each worker (TaskManager) is a *JVM process*, and may execute one or more
+subtasks in separate threads. To control how many tasks a TaskManager accepts, it
+has so called **task slots** (at least one).
+
+Each *task slot* represents a fixed subset of resources of the TaskManager. A
+TaskManager with three slots, for example, will dedicate 1/3 of its managed
+memory to each slot. Slotting the resources means that a subtask will not
+compete with subtasks from other jobs for managed memory, but instead has a
+certain amount of reserved managed memory. Note that no CPU isolation happens
+here; currently slots only separate the managed memory of tasks.
+
+By adjusting the number of task slots, users can define how subtasks are
+isolated from each other. Having one slot per TaskManager means that each task
+group runs in a separate JVM (which can be started in a separate container, for
+example). Having multiple slots means more subtasks share the same JVM. Tasks
+in the same JVM share TCP connections (via multiplexing) and heartbeat
+messages. They may also share data sets and data structures, thus reducing the
+per-task overhead.
+
+
+
+By default, Flink allows subtasks to share slots even if they are subtasks of
+different tasks, so long as they are from the same job. The result is that one
+slot may hold an entire pipeline of the job. Allowing this *slot sharing* has
+two main benefits:
+
+ - A Flink cluster needs exactly as many task slots as the highest parallelism
+ used in the job. No need to calculate how many tasks (with varying
+ parallelism) a program contains in total.
+
+ - It is easier to get better resource utilization. Without slot sharing, the
+ non-intensive *source/map()* subtasks would block as many resources as the
+ resource intensive *window* subtasks. With slot sharing, increasing the
+ base parallelism in our example from two to six yields full utilization of
+ the slotted resources, while making sure that the heavy subtasks are fairly
+ distributed among the TaskManagers.
+
+
+
+## Flink Application Execution
+
+A _Flink Application_ is any user program that spawns one or multiple Flink
+jobs from its ``main()`` method. The execution of these jobs can happen in a
+local JVM (``LocalEnvironment``) or on a remote setup of clusters with multiple
+machines (``RemoteEnvironment``). For each program, the
+[``ExecutionEnvironment``]({{ site.baseurl }}/api/java/) provides methods to
+control the job execution (e.g. setting the parallelism) and to interact with
+the outside world (see [Anatomy of a Flink Program]({%
+link dev/datastream_api.zh.md %}#anatomy-of-a-flink-program)).
+
+The jobs of a Flink Application can either be submitted to a long-running
+[Flink Session Cluster]({%
+link concepts/glossary.zh.md %}#flink-session-cluster), a dedicated [Flink Job
+Cluster]({% link concepts/glossary.zh.md %}#flink-job-cluster), or a
+[Flink Application Cluster]({%
+link concepts/glossary.zh.md %}#flink-application-cluster). The difference between
+these options is mainly related to the cluster’s lifecycle and to resource
+isolation guarantees.
+
+### Flink Session Cluster
+
+* **Cluster Lifecycle**: in a Flink Session Cluster, the client connects to a
+ pre-existing, long-running cluster that can accept multiple job submissions.
+ Even after all jobs are finished, the cluster (and the JobManager) will
+ keep running until the session is manually stopped. The lifetime of a Flink
+ Session Cluster is therefore not bound to the lifetime of any Flink Job.
+
+* **Resource Isolation**: TaskManager slots are allocated by the
+ ResourceManager on job submission and released once the job is finished.
+ Because all jobs are sharing the same cluster, there is some competition for
+ cluster resources — like network bandwidth in the submit-job phase. One
+ limitation of this shared setup is that if one TaskManager crashes, then all
+ jobs that have tasks running on this TaskManager will fail; in a similar way, if
+ some fatal error occurs on the JobManager, it will affect all jobs running
+ in the cluster.
+
+* **Other considerations**: having a pre-existing cluster saves a considerable
+ amount of time applying for resources and starting TaskManagers. This is
+ important in scenarios where the execution time of jobs is very short and a
+ high startup time would negatively impact the end-to-end user experience — as
+ is the case with interactive analysis of short queries, where it is desirable
+ that jobs can quickly perform computations using existing resources.
+
+Note: Formerly, a Flink Session Cluster was also known as a Flink Cluster in session mode.+ +### Flink Job Cluster + +* **Cluster Lifecycle**: in a Flink Job Cluster, the available cluster manager + (like YARN or Kubernetes) is used to spin up a cluster for each submitted job + and this cluster is available to that job only. Here, the client first + requests resources from the cluster manager to start the JobManager and + submits the job to the Dispatcher running inside this process. TaskManagers + are then lazily allocated based on the resource requirements of the job. Once + the job is finished, the Flink Job Cluster is torn down. + +* **Resource Isolation**: a fatal error in the JobManager only affects the one job running in that Flink Job Cluster. + +* **Other considerations**: because the ResourceManager has to apply and wait + for external resource management components to start the TaskManager + processes and allocate resources, Flink Job Clusters are more suited to large + jobs that are long-running, have high-stability requirements and are not + sensitive to longer startup times. + +Note: Formerly, a Flink Job Cluster was also known as a Flink Cluster in job (or per-job) mode.+ +### Flink Application Cluster + +* **Cluster Lifecycle**: a Flink Application Cluster is a dedicated Flink + cluster that only executes jobs from one Flink Application and where the + ``main()`` method runs on the cluster rather than the client. The job + submission is a one-step process: you don’t need to start a Flink cluster + first and then submit a job to the existing cluster session; instead, you + package your application logic and dependencies into a executable job JAR and + the cluster entrypoint (``ApplicationClusterEntryPoint``) + is responsible for calling the ``main()`` method to extract the JobGraph. + This allows you to deploy a Flink Application like any other application on + Kubernetes, for example. The lifetime of a Flink Application Cluster is + therefore bound to the lifetime of the Flink Application. + +* **Resource Isolation**: in a Flink Application Cluster, the ResourceManager + and Dispatcher are scoped to a single Flink Application, which provides a + better separation of concerns than the Flink Session Cluster. + +Note: A Flink Job Cluster can be seen as a “run-on-client” alternative to Flink Application Clusters.+ +{% top %} diff --git a/docs/concepts/glossary.md b/docs/concepts/glossary.md index 697b1b044f6cf2..83f0a1de486ce6 100644 --- a/docs/concepts/glossary.md +++ b/docs/concepts/glossary.md @@ -38,7 +38,7 @@ executes a single [Flink Job](#flink-job). The lifetime of the #### Flink Cluster -A distributed system consisting of (typically) one [Flink Master](#flink-master) and one or more +A distributed system consisting of (typically) one [JobManager](#flink-jobmanager) and one or more [Flink TaskManager](#flink-taskmanager) processes. #### Event @@ -88,9 +88,14 @@ see [Logical Graph](#logical-graph) #### Flink JobManager -JobManagers are one of the components running in the [Flink Master](#flink-master). A JobManager is -responsible for supervising the execution of the [Tasks](#task) of a single job. Historically, the -whole [Flink Master](#flink-master) was called JobManager. +The JobManager is the orchestrator of a [Flink Cluster](#flink-cluster). It contains three distinct +components: Flink Resource Manager, Flink Dispatcher and one [Flink JobMaster](#flink-jobmaster) +per running [Flink Job](#flink-job). + +#### Flink JobMaster + +JobMasters are one of the components running in the [JobManager](#flink-jobmanager). A JobMaster is +responsible for supervising the execution of the [Tasks](#task) of a single job. #### Logical Graph @@ -106,12 +111,6 @@ Logical graphs are also often referred to as *dataflow graphs*. Managed State describes application state which has been registered with the framework. For Managed State, Apache Flink will take care about persistence and rescaling among other things. -#### Flink Master - -The Flink Master is the master of a [Flink Cluster](#flink-cluster). It contains three distinct -components: Flink Resource Manager, Flink Dispatcher and one [Flink JobManager](#flink-jobmanager) -per running [Flink Job](#flink-job). - #### Operator Node of a [Logical Graph](#logical-graph). An Operator performs a certain operation, which is @@ -155,7 +154,7 @@ Formerly, a Flink Session Cluster was also known as a Flink Cluster in *session For stream processing programs, the State Backend of a [Flink Job](#flink-job) determines how its [state](#managed-state) is stored on each TaskManager (Java Heap of TaskManager or (embedded) RocksDB) as well as where it is written upon a checkpoint (Java Heap of -[Flink Master](#flink-master) or Filesystem). +[JobManager](#flink-jobmanager) or Filesystem). #### Sub-Task diff --git a/docs/concepts/glossary.zh.md b/docs/concepts/glossary.zh.md index 8efd2f101a7ebd..dc78acaeb0ec81 100644 --- a/docs/concepts/glossary.zh.md +++ b/docs/concepts/glossary.zh.md @@ -25,11 +25,20 @@ under the License. #### Flink Application Cluster -Flink Application Cluster 是一个专用的 [Flink Cluster](#flink-cluster)，它仅用于执行单个 [Flink Job](#flink-job)。[Flink Cluster](#flink-cluster)的生命周期与 [Flink Job](#flink-job)的生命周期绑定在一起。以前，Flink Application Cluster 也称为*job mode*的 Flink Cluster。和 [Flink Session Cluster](#flink-session-cluster) 作对比。 +A Flink Application Cluster is a dedicated [Flink Cluster](#flink-cluster) that +only executes [Flink Jobs](#flink-job) from one [Flink +Application](#flink-application). The lifetime of the [Flink +Cluster](#flink-cluster) is bound to the lifetime of the Flink Application. + +#### Flink Job Cluster + +A Flink Job Cluster is a dedicated [Flink Cluster](#flink-cluster) that only +executes a single [Flink Job](#flink-job). The lifetime of the +[Flink Cluster](#flink-cluster) is bound to the lifetime of the Flink Job. #### Flink Cluster -一般情况下，Flink 集群是由一个 [Flink Master](#flink-master) 和一个或多个 [Flink TaskManager](#flink-taskmanager) 进程组成的分布式系统。 +一般情况下，Flink 集群是由一个 [Flink JobManager](#flink-jobmanager) 和一个或多个 [Flink TaskManager](#flink-taskmanager) 进程组成的分布式系统。 #### Event @@ -47,9 +56,22 @@ Function 是由用户实现的，并封装了 Flink 程序的应用程序逻辑 Instance 常用于描述运行时的特定类型(通常是 [Operator](#operator) 或者 [Function](#function))的一个具体实例。由于 Apache Flink 主要是用 Java 编写的，所以，这与 Java 中的 *Instance* 或 *Object* 的定义相对应。在 Apache Flink 的上下文中，*parallel instance* 也常用于强调同一 [Operator](#operator) 或者 [Function](#function) 的多个 instance 以并行的方式运行。 +#### Flink Application + +A Flink application is a Java Application that submits one or multiple [Flink +Jobs](#flink-job) from the `main()` method (or by some other means). Submitting +jobs is usually done by calling `execute()` on an execution environment. + +The jobs of an application can either be submitted to a long running [Flink +Session Cluster](#flink-session-cluster), to a dedicated [Flink Application +Cluster](#flink-application-cluster), or to a [Flink Job +Cluster](#flink-job-cluster). + #### Flink Job -Flink Job 代表运行时的 Flink 程序。Flink Job 可以提交到长时间运行的 [Flink Session Cluster](#flink-session-cluster)，也可以作为独立的 [Flink Application Cluster](#flink-application-cluster) 启动。 +A Flink Job is the runtime representation of a [logical graph](#logical-graph) +(also often called dataflow graph) that is created and submitted by calling +`execute()` in a [Flink Application](#flink-application). #### JobGraph @@ -57,20 +79,26 @@ Flink Job 代表运行时的 Flink 程序。Flink Job 可以提交到长时间 #### Flink JobManager -JobManager 是在 [Flink Master](#flink-master) 运行中的组件之一。JobManager 负责监督单个作业 [Task](#task) 的执行。以前，整个 [Flink Master](#flink-master) 都叫做 JobManager。 +Flink JobManager 是 [Flink Cluster](#flink-cluster) 的主节点。它包含三个不同的组件：Flink Resource Manager、Flink Dispatcher、运行每个 [Flink Job](#flink-job) 的 [Flink JobMaster](#flink-jobmaster)。 + + +#### Flink JobMaster + +JobMaster 是在 [Flink JobManager](#flink-jobmanager) 运行中的组件之一。JobManager 负责监督单个作业 [Task](#task) 的执行。以前，整个 [Flink JobManager](#flink-jobmanager) 都叫做 JobManager。 #### Logical Graph -Logical Graph 是一种描述流处理程序的高阶逻辑有向图。节点是[Operator](#operator)，边代表输入/输出关系、数据流和数据集中的之一。 +A logical graph is a directed graph where the nodes are [Operators](#operator) +and the edges define input/output-relationships of the operators and correspond +to data streams or data sets. A logical graph is created by submitting jobs +from a [Flink Application](#flink-application). + +Logical graphs are also often referred to as *dataflow graphs*. #### Managed State Managed State 描述了已在框架中注册的应用程序的托管状态。对于托管状态，Apache Flink 会负责持久化和重伸缩等事宜。 -#### Flink Master - -Flink Master 是 [Flink Cluster](#flink-cluster) 的主节点。它包含三个不同的组件：Flink Resource Manager、Flink Dispatcher、运行每个 [Flink Job](#flink-job) 的 [Flink JobManager](#flink-jobmanager)。 - #### Operator [Logical Graph](#logical-graph) 的节点。算子执行某种操作，该操作通常由 [Function](#function) 执行。Source 和 Sink 是数据输入和数据输出的特殊算子。 @@ -97,7 +125,7 @@ Record 是数据集或数据流的组成元素。[Operator](#operator) 和 [Func #### State Backend -对于流处理程序，[Flink Job](#flink-job) 的 State Backend 决定了其 [state](#managed-state) 是如何存储在每个 TaskManager 上的（ TaskManager 的 Java 堆栈或嵌入式 RocksDB），以及它在 checkpoint 时的写入位置（ [Flink Master](#flink-master) 的 Java 堆或者 Filesystem）。 +对于流处理程序，[Flink Job](#flink-job) 的 State Backend 决定了其 [state](#managed-state) 是如何存储在每个 TaskManager 上的（ TaskManager 的 Java 堆栈或嵌入式 RocksDB），以及它在 checkpoint 时的写入位置（ [Flink JobManager](#flink-jobmanager) 的 Java 堆或者 Filesystem）。 #### Sub-Task diff --git a/docs/concepts/index.md b/docs/concepts/index.md index e89ec0c35e5ecd..11beb8e8474817 100644 --- a/docs/concepts/index.md +++ b/docs/concepts/index.md @@ -1,12 +1,11 @@ --- -title: Concepts in Depth +title: Concepts nav-id: concepts nav-pos: 3 -nav-title: ' Concepts in Depth' +nav-title: ' Concepts' nav-parent_id: root nav-show_overview: true permalink: /concepts/index.html -always-expand: true --- -The [Hands-on Training]({% link training/index.md %}) explains the basic concepts +The [Hands-on Training]({% link learn-flink/index.md %}) explains the basic concepts of stateful and timely stream processing that underlie Flink's APIs, and provides examples of how these mechanisms are used in applications. Stateful stream processing is introduced in the context -of [Data Pipelines & ETL]({% link training/etl.md %}#stateful-transformations) -and is further developed in the section on [Fault Tolerance]({% link -training/fault_tolerance.md %}). Timely stream processing is introduced in the section on -[Streaming Analytics]({% link training/streaming_analytics.md %}). +of [Data Pipelines & ETL]({% link learn-flink/etl.md %}#stateful-transformations) +and is further developed in the section on [Fault Tolerance]({% link learn-flink/fault_tolerance.md %}). Timely stream processing is introduced in the section on +[Streaming Analytics]({% link learn-flink/streaming_analytics.md %}). This _Concepts in Depth_ section provides a deeper understanding of how Flink's architecture and runtime implement these concepts. diff --git a/docs/concepts/index.zh.md b/docs/concepts/index.zh.md index a2872dddb7c6f6..4169f6d314bb9b 100644 --- a/docs/concepts/index.zh.md +++ b/docs/concepts/index.zh.md @@ -6,7 +6,6 @@ nav-title: ' 概 nav-parent_id: root nav-show_overview: true permalink: /concepts/index.html -always-expand: true --- -The [Hands-on Training]({% link training/index.zh.md %}) explains the basic concepts -of stateful and timely stream processing that underlie Flink's APIs, and provides examples of how -these mechanisms are used in applications. Stateful stream processing is introduced in the context -of [Data Pipelines & ETL]({% link training/etl.zh.md %}#stateful-transformations) -and is further developed in the section on [Fault Tolerance]({% link -training/fault_tolerance.zh.md %}). Timely stream processing is introduced in the section on -[Streaming Analytics]({% link training/streaming_analytics.zh.md %}). +[实践练习]({% link learn-flink/index.zh.md %})章节介绍了作为 Flink API 根基的有状态实时流处理的基本概念，并且举例说明了如何在 Flink 应用中使用这些机制。其中 [Data Pipelines & ETL]({% link learn-flink/etl.zh.md %}#stateful-transformations) 小节介绍了有状态流处理的概念，并且在 [Fault Tolerance]({% link learn-flink/fault_tolerance.zh.md %}) 小节中进行了深入介绍。[Streaming Analytics]({% link learn-flink/streaming_analytics.zh.md %}) 小节介绍了实时流处理的概念。 -This _Concepts in Depth_ section provides a deeper understanding of how Flink's architecture and runtime -implement these concepts. +本章将深入分析 Flink 分布式运行时架构如何实现这些概念。 -## Flink's APIs +## Flink 中的 API -Flink offers different levels of abstraction for developing streaming/batch applications. +Flink 为流式/批式处理应用程序的开发提供了不同级别的抽象。
- - The lowest level abstraction simply offers **stateful and timely stream processing**. It is
- embedded into the [DataStream API]({{ site.baseurl}}{% link
- dev/datastream_api.zh.md %}) via the [Process Function]({{ site.baseurl }}{%
- link dev/stream/operators/process_function.zh.md %}). It allows users to freely
- process events from one or more streams, and provides consistent, fault tolerant
- *state*. In addition, users can register event time and processing time
- callbacks, allowing programs to realize sophisticated computations.
+ - Flink API 最底层的抽象为**有状态实时流处理**。其抽象实现是 [Process Function]({{ site.baseurl }}{% link dev/stream/operators/process_function.zh.md %})，并且 **Process Function** 被 Flink 框架集成到了 [DataStream API]({{ site.baseurl}}{% link dev/datastream_api.zh.md %}) 中来为我们使用。它允许用户在应用程序中自由地处理来自单流或多流的事件（数据），并提供具有全局一致性和容错保障的*状态*。此外，用户可以在此层抽象中注册事件时间（event time）和处理时间（processing time）回调方法，从而允许程序可以实现复杂计算。
- - In practice, many applications do not need the low-level
- abstractions described above, and can instead program against the **Core APIs**: the
- [DataStream API]({% link dev/datastream_api.zh.md %})
- (bounded/unbounded streams) and the [DataSet API]({% link
- dev/batch/index.zh.md %}) (bounded data sets). These fluent APIs offer the
- common building blocks for data processing, like various forms of
- user-specified transformations, joins, aggregations, windows, state, etc.
- Data types processed in these APIs are represented as classes in the
- respective programming languages.
+ - Flink API 第二层抽象是 **Core APIs**。实际上，许多应用程序不需要使用到上述最底层抽象的 API，而是可以使用 **Core APIs** 进行编程：其中包含 [DataStream API]({% link dev/datastream_api.zh.md %})（应用于有界/无界数据流场景）和 [DataSet API]({% link dev/batch/index.zh.md %})（应用于有界数据集场景）两部分。Core APIs 提供的流式 API（Fluent API）为数据处理提供了通用的模块组件，例如各种形式的用户自定义转换（transformations）、联接（joins）、聚合（aggregations）、窗口（windows）和状态（state）操作等。此层 API 中处理的数据类型在每种编程语言中都有其对应的类。
- The low level *Process Function* integrates with the *DataStream API*,
- making it possible to use the lower-level abstraction on an as-needed basis.
- The *DataSet API* offers additional primitives on bounded data sets,
- like loops/iterations.
+ *Process Function* 这类底层抽象和 *DataStream API* 的相互集成使得用户可以选择使用更底层的抽象 API 来实现自己的需求。*DataSet API* 还额外提供了一些原语，比如循环/迭代（loop/iteration）操作。
- - The **Table API** is a declarative DSL centered around *tables*, which may
- be dynamically changing tables (when representing streams). The [Table
- API]({% link dev/table/index.zh.md %}) follows the
- (extended) relational model: Tables have a schema attached (similar to
- tables in relational databases) and the API offers comparable operations,
- such as select, project, join, group-by, aggregate, etc. Table API
- programs declaratively define *what logical operation should be done*
- rather than specifying exactly *how the code for the operation looks*.
- Though the Table API is extensible by various types of user-defined
- functions, it is less expressive than the *Core APIs*, and more concise to
- use (less code to write). In addition, Table API programs also go through
- an optimizer that applies optimization rules before execution.
+ - Flink API 第三层抽象是 **Table API**。**Table API** 是以表（Table）为中心的声明式编程（DSL）API，例如在流式数据场景下，它可以表示一张正在动态改变的表。[Table API]({% link dev/table/index.zh.md %}) 遵循（扩展）关系模型：即表拥有 schema（类似于关系型数据库中的 schema），并且 Table API 也提供了类似于关系模型中的操作，比如 select、project、join、group-by 和 aggregate 等。Table API 程序是以声明的方式定义*应执行的逻辑操作*，而不是确切地指定程序*应该执行的代码*。尽管 Table API 使用起来很简洁并且可以由各种类型的用户自定义函数扩展功能，但还是比 Core API 的表达能力差。此外，Table API 程序在执行之前还会使用优化器中的优化规则对用户编写的表达式进行优化。
- One can seamlessly convert between tables and *DataStream*/*DataSet*,
- allowing programs to mix the *Table API* with the *DataStream* and
- *DataSet* APIs.
+ 表和 *DataStream*/*DataSet* 可以进行无缝切换，Flink 允许用户在编写应用程序时将 *Table API* 与 *DataStream*/*DataSet* API 混合使用。
- - The highest level abstraction offered by Flink is **SQL**. This abstraction
- is similar to the *Table API* both in semantics and expressiveness, but
- represents programs as SQL query expressions. The [SQL]({{ site.baseurl
- }}{% link dev/table/index.zh.md %}#sql) abstraction closely interacts with the
- Table API, and SQL queries can be executed over tables defined in the
- *Table API*.
+ - Flink API 最顶层抽象是 **SQL**。这层抽象在语义和程序表达式上都类似于 *Table API*，但是其程序实现都是 SQL 查询表达式。[SQL]({{ site.baseurl}}{% link dev/table/index.zh.md %}#sql) 抽象与 Table API 抽象之间的关联是非常紧密的，并且 SQL 查询语句可以在 *Table API* 中定义的表上执行。
diff --git a/docs/concepts/stateful-stream-processing.md b/docs/concepts/stateful-stream-processing.md
index b8cdd634c0492b..45c916b64ba6b9 100644
--- a/docs/concepts/stateful-stream-processing.md
+++ b/docs/concepts/stateful-stream-processing.md
@@ -24,6 +24,11 @@ specific language governing permissions and limitations
under the License.
-->
+* This will be replaced by the TOC
+{:toc}
+
+## What is State?
+
While many operations in a dataflow simply look at one individual *event at a
time* (for example an event parser), some operations remember information
across multiple events (for example window operators). These operations are
@@ -54,19 +59,6 @@ When working with state, it might also be useful to read about [Flink's state
backends]({% link ops/state/state_backends.md %}). Flink
provides different state backends that specify how and where state is stored.
-* This will be replaced by the TOC
-{:toc}
-
-## What is State?
-
-`TODO: expand this section`
-
-{% top %}
-
-## State in Stream & Batch Processing
-
-`TODO: What is this section about? Do we even need it?`
-
{% top %}
## Keyed State
@@ -148,6 +140,8 @@ Keep in mind that everything to do with checkpointing can be done
asynchronously. The checkpoint barriers don't travel in lock step and
operations can asynchronously snapshot their state.
+Since Flink 1.11, checkpoints can be taken with or without alignment. In this
+section, we describe aligned checkpoints first.
#### Barriers
@@ -197,14 +191,16 @@ streams on the snapshot barriers. The figure above illustrates this:
received the barrier *n* from the other inputs as well. Otherwise, it would
mix records that belong to snapshot *n* and with records that belong to
snapshot *n+1*.
- - Streams that report barrier *n* are temporarily set aside. Records that are
- received from these streams are not processed, but put into an input
- buffer.
- Once the last stream has received barrier *n*, the operator emits all
pending outgoing records, and then emits snapshot *n* barriers itself.
- - After that, it resumes processing records from all input streams,
+ - It snapshots the state and resumes processing records from all input streams,
processing records from the input buffers before processing the records
from the streams.
+ - Finally, the operator writes the state asynchronously to the state backend.
+
+Note that the alignment is needed for all operators with multiple inputs and for
+operators after a shuffle when they consume output streams of multiple upstream
+subtasks.
#### Snapshotting Operator State
@@ -214,7 +210,7 @@ snapshots as well.
Operators snapshot their state at the point in time when they have received all
snapshot barriers from their input streams, and before emitting the barriers to
their output streams. At that point, all updates to the state from records
-before the barriers will have been made, and no updates that depend on records
+before the barriers have been made, and no updates that depend on records
from after the barriers have been applied. Because the state of a snapshot may
be large, it is stored in a configurable *[state backend]({{ site.baseurl }}{%
link ops/state/state_backends.md %})*. By default, this is the JobManager's
@@ -250,9 +246,49 @@ updates to that state.
See [Restart Strategies]({% link dev/task_failure_recovery.md
%}#restart-strategies) for more information.
-### State Backends
+### Unaligned Checkpointing
+
+Starting with Flink 1.11, checkpointing can also be performed unaligned.
+The basic idea is that checkpoints can overtake all in-flight data as long as
+the in-flight data becomes part of the operator state.
-`TODO: expand this section`
+Note that this approach is actually closer to the [Chandy-Lamport algorithm
+](http://research.microsoft.com/en-us/um/people/lamport/pubs/chandy.pdf), but
+Flink still inserts the barrier in the sources to avoid overloading the
+checkpoint coordinator.
+
+++ +The figure depicts how an operator handles unaligned checkpoint barriers: + +- The operator reacts on the first barrier that is stored in its input buffers. +- It immediately forwards the barrier to the downstream operator by adding it + to the end of the output buffers. +- The operator marks all overtaken records to be stored asynchronously and + creates a snapshot of its own state. + +Consequently, the operator only briefly stops the processing of input to mark +the buffers, forwards the barrier, and creates the snapshot of the other state. + +Unaligned checkpointing ensures that barriers are arriving at the sink as fast +as possible. It's especially suited for applications with at least one slow +moving data path, where alignment times can reach hours. However, since it's +adding additional I/O pressure, it doesn't help when the I/O to the state +backends is the bottleneck. See the more in-depth discussion in +[ops]({% link ops/state/checkpoints.md %}#unaligned-checkpoints) +for other limitations. + +Note that savepoints will always be aligned. + +#### Unaligned Recovery + +Operators first recover the in-flight data before starting processing any data +from upstream operators in unaligned checkpointing. Aside from that, it +performs the same steps as during [recovery of aligned checkpoints](#recovery). + +### State Backends The exact data structures in which the key/values indexes are stored depends on the chosen [state backend]({% link @@ -270,8 +306,6 @@ logic. ### Savepoints -`TODO: expand this section` - All programs that use checkpointing can resume execution from a **savepoint**. Savepoints allow both updating your programs and your Flink cluster without losing any state. @@ -312,10 +346,6 @@ give *exactly once* guarantees even in *at least once* mode. {% top %} -## End-to-end Exactly-Once Programs - -`TODO: add` - ## State and Fault Tolerance in Batch Programs Flink executes [batch programs](../dev/batch/index.html) as a special case of @@ -335,6 +365,6 @@ programs, with minor exceptions: - The DataSet API introduces special synchronized (superstep-based) iterations, which are only possible on bounded streams. For details, check - out the [iteration docs]({{ site.baseurl }}/dev/batch/iterations.html). + out the [iteration docs]({% link dev/batch/iterations.md %}). {% top %} diff --git a/docs/concepts/stateful-stream-processing.zh.md b/docs/concepts/stateful-stream-processing.zh.md index e7d19ad2347080..29ea1f1a74fa70 100644 --- a/docs/concepts/stateful-stream-processing.zh.md +++ b/docs/concepts/stateful-stream-processing.zh.md @@ -1,8 +1,8 @@ --- -title: Stateful Stream Processing +title: 有状态流处理 nav-id: stateful-stream-processing nav-pos: 2 -nav-title: Stateful Stream Processing +nav-title: 有状态流处理 nav-parent_id: concepts --- +* This will be replaced by the TOC +{:toc} + +## What is State? + While many operations in a dataflow simply look at one individual *event at a time* (for example an event parser), some operations remember information across multiple events (for example window operators). These operations are @@ -54,19 +59,6 @@ When working with state, it might also be useful to read about [Flink's state backends]({% link ops/state/state_backends.zh.md %}). Flink provides different state backends that specify how and where state is stored. -* This will be replaced by the TOC -{:toc} - -## What is State? - -`TODO: expand this section` - -{% top %} - -## State in Stream & Batch Processing - -`TODO: What is this section about? Do we even need it?` - {% top %} ## Keyed State @@ -252,8 +244,6 @@ See [Restart Strategies]({% link dev/task_failure_recovery.zh.md ### State Backends -`TODO: expand this section` - The exact data structures in which the key/values indexes are stored depends on the chosen [state backend]({% link ops/state/state_backends.zh.md %}). One state backend stores data in an in-memory @@ -270,8 +260,6 @@ logic. ### Savepoints -`TODO: expand this section` - All programs that use checkpointing can resume execution from a **savepoint**. Savepoints allow both updating your programs and your Flink cluster without losing any state. @@ -312,10 +300,6 @@ give *exactly once* guarantees even in *at least once* mode. {% top %} -## End-to-end Exactly-Once Programs - -`TODO: add` - ## State and Fault Tolerance in Batch Programs Flink executes [batch programs](../dev/batch/index.html) as a special case of diff --git a/docs/concepts/timely-stream-processing.md b/docs/concepts/timely-stream-processing.md index cb2c1579e8b89a..9b96c9f97a24c8 100644 --- a/docs/concepts/timely-stream-processing.md +++ b/docs/concepts/timely-stream-processing.md @@ -24,16 +24,20 @@ specific language governing permissions and limitations under the License. --> -`TODO: add introduction` - * This will be replaced by the TOC {:toc} -## Latency & Completeness +## Introduction -`TODO: add these two sections` +Timely steam processing is an extension of [stateful stream processing]({% link +concepts/stateful-stream-processing.md %}) in which time plays some role in the +computation. Among other things, this is the case when you do time series +analysis, when doing aggregations based on certain time periods (typically +called windows), or when you do event processing where the time when an event +occured is important. -### Latency vs. Completeness in Batch & Stream Processing +In the following sections we will highlight some of the topics that you should +consider when working with timely Flink Applications. {% top %} diff --git a/docs/concepts/timely-stream-processing.zh.md b/docs/concepts/timely-stream-processing.zh.md index 9f80f72ad3fb6a..8cf20e0954015c 100644 --- a/docs/concepts/timely-stream-processing.zh.md +++ b/docs/concepts/timely-stream-processing.zh.md @@ -1,8 +1,8 @@ --- -title: Timely Stream Processing +title: 及时流处理 nav-id: timely-stream-processing nav-pos: 3 -nav-title: Timely Stream Processing +nav-title: 及时流处理 nav-parent_id: concepts --- -`TODO: add introduction` - * This will be replaced by the TOC {:toc} -## Latency & Completeness +## Introduction -`TODO: add these two sections` +Timely steam processing is an extension of [stateful stream processing]({% link +concepts/stateful-stream-processing.zh.md %}) in which time plays some role in the +computation. Among other things, this is the case when you do time series +analysis, when doing aggregations based on certain time periods (typically +called windows), or when you do event processing where the time when an event +occured is important. -### Latency vs. Completeness in Batch & Stream Processing +In the following sections we will highlight some of the topics that you should +consider when working with timely Flink Applications. {% top %} diff --git a/docs/dev/api_concepts.zh.md b/docs/dev/api_concepts.zh.md deleted file mode 100644 index c71d635a871e4a..00000000000000 --- a/docs/dev/api_concepts.zh.md +++ /dev/null @@ -1,778 +0,0 @@ ---- -title: "基础 API 概念" -nav-parent_id: dev -nav-pos: 1 -nav-show_overview: true -nav-id: api-concepts ---- - - -Flink 程序是实现了分布式集合转换（例如过滤、映射、更新状态、join、分组、定义窗口、聚合）的规范化程序。集合初始创建自 source（例如读取文件、kafka 主题，或本地内存中的集合）。结果通过 sink 返回，例如，它可以将数据写入（分布式）文件，或标准输出（例如命令行终端）。Flink 程序可以在多种环境中运行，独立运行或嵌入到其他程序中。可以在本地 JVM 中执行，也可以在多台机器的集群上执行。 - -针对有界和无界两种数据 source 类型，你可以使用 DataSet API 来编写批处理程序或使用 DataStream API 来编写流处理程序。本篇指南将介绍这两种 API 通用的基本概念，关于使用 API 编写程序的具体信息请查阅 -[流处理指南]({{ site.baseurl }}/zh/dev/datastream_api.html) 和 -[批处理指南]({{ site.baseurl }}/zh/dev/batch/index.html)。 - -**请注意：** 当展示如何使用 API 的实际示例时我们使用 `StreamingExecutionEnvironment` 和 `DataStream API`。对于批处理，将他们替换为 `ExecutionEnvironment` 和 `DataSet API` 即可，概念是完全相同的。 - -* This will be replaced by the TOC -{:toc} - -DataSet 和 DataStream ----------------------- - -Flink 用特有的 `DataSet` 和 `DataStream` 类来表示程序中的数据。你可以将他们视为可能包含重复项的不可变数据集合。对于 `DataSet`，数据是有限的，而对于 `DataStream`，元素的数量可以是无限的。 - -这些集合与标准的 Java 集合有一些关键的区别。首先它们是不可变的，也就是说它们一旦被创建你就不能添加或删除元素了。你也不能简单地检查它们内部的元素。 - -在 Flink 程序中，集合最初通过添加数据 source 来创建，通过使用诸如 `map`、`filter` 等 API 方法对数据 source 进行转换从而派生新的集合。 - -剖析一个 Flink 程序 --------------------------- - -Flink 程序看起来像是转换数据集合的规范化程序。每个程序由一些基本的部分组成： - -1. 获取执行环境， -2. 加载/创建初始数据， -3. 指定对数据的转换操作， -4. 指定计算结果存放的位置， -5. 触发程序执行 - - -
+-- -当设定好整个程序以后你需要调用 `StreamExecutionEnvironment` 的 `execute()` 方法**触发程序执行**。至于在你的本机触发还是提交到集群运行取决于 `ExecutionEnvironment` 的类型。 - -`execute()` 方法返回 `JobExecutionResult`，它包括执行耗时和一个累加器的结果。 - -如果你不需要等待作业的结束，只是想要触发程序执行，你可以调用 `StreamExecutionEnvironment` 的 `executeAsync()` 方法。这个方法将返回一个 `JobClient` 对象，通过 `JobClient` 能够与程序对应的作业进行交互。作为例子，这里介绍通过 `executeAsync()` 实现与 `execute()` 相同行为的方法。 - -{% highlight java %} -final JobClient jobClient = env.executeAsync(); - -final JobExecutionResult jobExecutionResult = jobClient.getJobExecutionResult(userClassloader).get(); -{% endhighlight %} - -有关流数据的 source 和 sink 以及有关 DataStream 支持的转换操作的详细信息请参阅[流处理指南]({{ site.baseurl }}/zh/dev/datastream_api.html)。 - -有关批数据的 source 和 sink 以及有关 DataSet 支持的转换操作的详细信息请参阅[批处理指南]({{ site.baseurl }}/zh/dev/batch/index.html)。 - - -{% top %} - -延迟计算 ---------------- - -无论在本地还是集群执行，所有的 Flink 程序都是延迟执行的：当程序的 main 方法被执行时，并不立即执行数据的加载和转换，而是创建每个操作并将其加入到程序的执行计划中。当执行环境调用 `execute()` 方法显式地触发执行的时候才真正执行各个操作。 - -延迟计算允许你构建复杂的程序，Flink 将其作为整体计划单元来执行。 - -{% top %} - -指定键 ---------------- - -一些转换操作（join, coGroup, keyBy, groupBy）要求在元素集合上定义键。另外一些转换操作 （Reduce, GroupReduce, Aggregate, Windows）允许在应用这些转换之前将数据按键分组。 - -如下对 DataSet 分组 -{% highlight java %} -DataSet<...> input = // [...] -DataSet<...> reduced = input - .groupBy(/*在这里定义键*/) - .reduceGroup(/*一些处理操作*/); -{% endhighlight %} - -如下对 DataStream 指定键 -{% highlight java %} -DataStream<...> input = // [...] -DataStream<...> windowed = input - .keyBy(/*在这里定义键*/) - .window(/*指定窗口*/); -{% endhighlight %} - -Flink 的数据模型不是基于键值对的。因此你不需要将数据集类型物理地打包到键和值中。键都是“虚拟的”：它们的功能是指导分组算子用哪些数据来分组。 - -**请注意：**下面的讨论中我们将以 `DataStream` 和 `keyby` 为例。 -对于 DataSet API 你只需要用 `DataSet` 和 `groupBy` 替换即可。 - -### 为 Tuple 定义键 -{:.no_toc} - -最简单的方式是按照 Tuple 的一个或多个字段进行分组： - -- - -我们现在将概述每个步骤，详细信息请参阅相应章节。请注意所有 Java DataSet API 的核心类可以在这个包 {% gh_link /flink-java/src/main/java/org/apache/flink/api/java "org.apache.flink.api.java" %} 中找到，同时 Java DataStream API 可以在这个包 -{% gh_link /flink-streaming-java/src/main/java/org/apache/flink/streaming/api "org.apache.flink.streaming.api" %} 中找到。 - -`StreamExecutionEnvironment` 是所有 Flink 程序的基础。你可以使用它的这些静态方法获取： - -{% highlight java %} -getExecutionEnvironment() - -createLocalEnvironment() - -createRemoteEnvironment(String host, int port, String... jarFiles) -{% endhighlight %} - -通常你只需要使用 `getExecutionEnvironment()`，因为它会根据上下文环境完成正确的工作：如果你在 IDE 中执行程序或者作为标准的 Java 程序来执行，它会创建你的本机执行环境；如果你将程序封装成 JAR 包，然后通过[命令行]({{ site.baseurl }}/zh/ops/cli.html)调用，Flink 集群管理器会执行你的 main 方法并且 `getExecutionEnvironment()` 会返回在集群上执行程序的执行环境。 - -针对不同的数据 source，执行环境有若干不同读取文件的方法：你可以逐行读取 CSV 文件，或者使用完全自定义的输入格式。要将文本文件作为一系列行读取，你可以使用： - -{% highlight java %} -final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); - -DataStream-text = env.readTextFile("file:///path/to/file"); -{% endhighlight %} - -这样你会得到一个 DataStream 然后对其应用转换操作从而创建新的派生 DataStream。 - -通过调用 DataStream 的转换函数来进行转换。下面是一个映射转换的实例： - -{% highlight java %} -DataStream input = ...; - -DataStream parsed = input.map(new MapFunction - -我们现在将概述每个步骤，详细信息请参阅相应章节。请注意所有 Scala DataSet API 可以在这个包 {% gh_link /flink-scala/src/main/scala/org/apache/flink/api/scala "org.apache.flink.api.scala" %} 中找到，同时，所有 Scala DataStream API 可以在这个包 {% gh_link /flink-streaming-scala/src/main/scala/org/apache/flink/streaming/api/scala "org.apache.flink.streaming.api.scala" %} 中找到。 - -`StreamExecutionEnvironment` 是所有 Flink 程序的基础。你可以使用它的这些静态方法获取： - -{% highlight scala %} -getExecutionEnvironment() - -createLocalEnvironment() - -createRemoteEnvironment(host: String, port: Int, jarFiles: String*) -{% endhighlight %} - -通常你只需要使用 `getExecutionEnvironment()`，因为它会根据上下文环境完成正确的工作，如果你在 IDE 中执行程序或者作为标准的 Java 程序来执行，它会创建你的本机执行环境。如果你将程序封装成 JAR 包，然后通过[命令行]({{ site.baseurl }}/zh/ops/cli.html)调用，Flink 集群管理器会执行你的 main 方法并且 `getExecutionEnvironment()` 会返回在集群上执行程序的执行环境。 - -针对不同的数据 source，执行环境有若干不同的读取文件的方法：你可以逐行读取 CSV 文件，或者使用完全自定义的输入格式。要将文本文件作为一系列行读取，你可以使用： - -{% highlight scala %} -val env = StreamExecutionEnvironment.getExecutionEnvironment() - -val text: DataStream[String] = env.readTextFile("file:///path/to/file") -{% endhighlight %} - -如此你会得到一个 DataStream 然后对其应用转换操作从而创建新的派生 DataStream。 - -通过调用 DataStream 的转换函数来进行转换。下面是一个映射转换的实例： - -{% highlight scala %} -val input: DataSet[String] = ... - -val mapped = input.map { x => x.toInt } -{% endhighlight %} - -这会通过把原始数据集合的每个字符串转换为一个整数创建一个新的 DataStream。 - -一旦你得到了包含最终结果的 DataStream，就可以通过创建 sink 将其写入外部系统。如下是一些创建 sink 的示例： - -{% highlight scala %} -writeAsText(path: String) - -print() -{% endhighlight %} - ---- -按照第一个字段（整型字段）对 Tuple 分组。 - --{% highlight java %} -DataStream--{% highlight scala %} -val input: DataStream[(Int, String, Long)] = // [...] -val keyed = input.keyBy(0) -{% endhighlight %} ---- -这里我们用第一个字段和第二个字段组成的组合键对 Tuple 分组 - -对于嵌套 Tuple 请注意： 如果你的 DataStream 是嵌套 Tuple，例如： - -{% highlight java %} -DataStream-{% highlight java %} -DataStream--{% highlight scala %} -val input: DataSet[(Int, String, Long)] = // [...] -val grouped = input.groupBy(0,1) -{% endhighlight %} ---- -### 使用键选择器函数定义键 -{:.no_toc} - -定义键的另一种方法是“键选择器”函数。键选择器函数将单个元素作为输入并返回元素的键。键可以是任意类型，并且可以由确定性计算得出。 - -下例展示了一个简单返回对象字段的键选择器函数： - -- -下例中，我们有一个包含“word”和“count”两个字段的 POJO：`WC`。要用 `word` 字段分组，我们只需要把它的名字传给 `keyBy()` 函数即可。 -{% highlight java %} -// 普通的 POJO（简单的 Java 对象） -public class WC { - public String word; - public int count; -} -DataStream-words = // [...] -DataStream wordCounts = words.keyBy("word").window(/*指定窗口*/); -{% endhighlight %} - -**字段表达式语法**： - -- 根据字段名称选择 POJO 的字段。例如 `“user”` 就是指 POJO 类型的“user”字段。 - -- 根据字段名称或 0 开始的字段索引选择 Tuple 的字段。例如 `“f0”` 和 `“5”` 分别指 Java Tuple 类型的第一个和第六个字段。 - -- 可以选择 POJO 和 Tuple 的嵌套字段。 例如，一个 POJO 类型有一个“user”字段还是一个 POJO 类型，那么 `“user.zip”` 即指这个“user”字段的“zip”字段。任意嵌套和混合的 POJO 和 Tuple都是支持的，例如 `“f1.user.zip”` 或 `“user.f3.1.zip”`。 - -- 可以使用 `"*"` 通配符表达式选择完整的类型。这也适用于非 Tuple 或 POJO 类型。 - -**字段表达式示例**: - -{% highlight java %} -public static class WC { - public ComplexNestedClass complex; //嵌套的 POJO - private int count; - // 私有字段（count）的 getter 和 setter - public int getCount() { - return count; - } - public void setCount(int c) { - this.count = c; - } -} -public static class ComplexNestedClass { - public Integer someNumber; - public float someFloat; - public Tuple3 - -下例中，我们有一个包含“word”和“count”两个字段的 POJO：`WC`。要用 `word` 字段分组，我们只需要把它的名字传给 `keyBy()` 函数即可。 -{% highlight scala %} -// 普通的 POJO（简单的 Java 对象） -class WC(var word: String, var count: Int) { - def this() { this("", 0L) } -} -val words: DataStream[WC] = // [...] -val wordCounts = words.keyBy("word").window(/*指定窗口*/) - -// 或者，代码少一点的 case class -case class WC(word: String, count: Int) -val words: DataStream[WC] = // [...] -val wordCounts = words.keyBy("word").window(/*指定窗口*/) -{% endhighlight %} - -**字段表达式语法**： - -- 根据字段名称选择 POJO 的字段。例如 `“user”` 就是指 POJO 类型的“user”字段。 - -- 根据 1 开始的字段名称或 0 开始的字段索引选择 Tuple 的字段。例如 `“_1”` 和 `“5”` 分别指 Java Tuple 类型的第一个和第六个字段。 - -- 可以选择 POJO 和 Tuple 的嵌套字段。 例如，一个 POJO 类型有一个“user”字段还是一个 POJO 类型，那么 `“user.zip”` 即指这个“user”字段的“zip”字段。任意嵌套和混合的 POJO 和 Tuple都是支持的，例如 `“_2.user.zip”` 或 `“user._4.1.zip”`。 - -- 可以使用 `"*"` 通配符表达式选择完整的类型。这也适用于非 Tuple 或 POJO 类型。 - -**字段表达式示例**: - -{% highlight scala %} -class WC(var complex: ComplexNestedClass, var count: Int) { - def this() { this(null, 0) } -} - -class ComplexNestedClass( - var someNumber: Int, - someFloat: Float, - word: (Long, Long, String), - hadoopCitizen: IntWritable) { - def this() { this(0, 0, (0, 0, ""), new IntWritable(0)) } -} -{% endhighlight %} - -这些字段表达式对于以上代码示例都是合法的： - -- `"count"`：`WC` 类的 count 字段。 - -- `"complex"`：递归选择 POJO 类型 `ComplexNestedClass` 的 complex 字段的全部字段。 - -- `"complex.word._3"`：选择嵌套 `Tuple3` 类型的最后一个字段。 - -- `"complex.hadoopCitizen"`：选择 hadoop 的 `IntWritable` 类型。 - ---- -{% top %} - -指定转换函数 --------------------------- - -大多数转换操作需要用户定义函数。本节列举了指定它们的不同方法。 - --{% highlight java %} -// 普通的 POJO -public class WC {public String word; public int count;} -DataStream-words = // [...] -KeyedStream keyed = words - .keyBy(new KeySelector -{% highlight scala %} -// 普通的 case class -case class WC(word: String, count: Int) -val words: DataStream[WC] = // [...] -val keyed = words.keyBy( _.word ) -{% endhighlight %} ---- -富函数为用户定义函数（map、reduce 等）额外提供了 4 个方法： `open`、`close`、`getRuntimeContext` 和 `setRuntimeContext`。这些方法有助于向函数传参（请参阅 [向函数传递参数]({{ site.baseurl }}/zh/dev/batch/index.html#passing-parameters-to-functions)）、 -创建和终止本地状态、访问广播变量（请参阅 -[广播变量]({{ site.baseurl }}/zh/dev/batch/index.html#broadcast-variables)）、访问诸如累加器和计数器等运行时信息（请参阅 -[累加器和计数器](#accumulators--counters)）和迭代信息（请参阅 [迭代]({{ site.baseurl }}/zh/dev/batch/iterations.html)）。 - -{% top %} - -支持的数据类型 --------------------- - -Flink 对于 DataSet 或 DataStream 中可以包含的元素类型做了一些限制。这么做是为了使系统能够分析类型以确定有效的执行策略。 - -有七种不同的数据类型： - -1. **Java Tuple** 和 **Scala Case Class** -2. **Java POJO** -3. **基本数据类型** -4. **常规的类** -5. **值** -6. **Hadoop Writable** -7. **特殊类型** - -#### Tuple 和 Case Class - -- -#### 实现接口 - -最基本的方法是实现一个已提供的接口： - -{% highlight java %} -class MyMapFunction implements MapFunction-- - -#### Lambda 函数 - -正如前面的例子中所见，所有操作都接受 lambda 函数来描述操作： -{% highlight scala %} -val data: DataSet[String] = // [...] -data.filter { _.startsWith("http://") } -{% endhighlight %} - -{% highlight scala %} -val data: DataSet[Int] = // [...] -data.reduce { (i1,i2) => i1 + i2 } -// 或者 -data.reduce { _ + _ } -{% endhighlight %} - -#### 富函数 - -所有需要用户定义函数的转换操作都可以将*富*函数作为参数。例如，对于 - -{% highlight scala %} -data.map { x => x.toInt } -{% endhighlight %} - -你可以替换成 - -{% highlight scala %} -class MyMapFunction extends RichMapFunction[String, Int] { - def map(in: String):Int = { in.toInt } -}; -{% endhighlight %} - -并像往常一样将函数传递给 `map` 转换操作： - -{% highlight scala %} -data.map(new MyMapFunction()) -{% endhighlight %} - -富函数也可以被定义为匿名类： -{% highlight scala %} -data.map (new RichMapFunction[String, Int] { - def map(in: String):Int = { in.toInt } -}) -{% endhighlight %} -- --- -#### POJO - -Flink 将满足如下条件的 Java 和 Scala 的类作为特殊的 POJO 数据类型处理： - -- 类必须是公有的。 - -- 它必须有一个公有的无参构造器（默认构造器）。 - -- 所有的字段要么是公有的要么必须可以通过 getter 和 setter 函数访问。例如一个名为 `foo` 的字段，它的 getter 和 setter 方法必须命名为 `getFoo()` 和 `setFoo()`。 - -- 字段的类型必须被已注册的序列化程序所支持。 - -POJO 通常用 `PojoTypeInfo` 表示，并使用 `PojoSerializer`（[Kryo](https://github.com/EsotericSoftware/kryo) 作为可配置的备用序列化器）序列化。 -例外情况是 POJO 是 Avro 类型（Avro 指定的记录）或作为“Avro 反射类型”生成时。 -在这种情况下 POJO 由 `AvroTypeInfo` 表示，并且由 `AvroSerializer` 序列化。 -如果需要，你可以注册自己的序列化器；更多信息请参阅 [序列化](https://ci.apache.org/projects/flink/flink-docs-master/zh/dev/types_serialization.html)。 - -Flink 分析 POJO 类型的结构，也就是说，它会推断出 POJO 的字段。因此，POJO 类型比常规类型更易于使用。此外，Flink 可以比一般类型更高效地处理 POJO。 - -下例展示了一个拥有两个公有字段的简单 POJO。 - -- -Tuple 是复合类型，包含固定数量的各种类型的字段。 -Java API 提供了从 `Tuple1` 到 `Tuple25` 的类。 Tuple 的每一个字段可以是任意 Flink 类型，包括 Tuple，即嵌套的 Tuple。Tuple 的字段可以通过字段名称如 `tuple.f4` 直接访问，也可以使用通常的 getter 方法 `tuple.getField(int position)`。字段索引从 0 开始。请注意，这与 Scala Tuple 形成鲜明对比，但与 Java 的常规索引更为一致。 - -{% highlight java %} -DataStream-- -Scala Case Class（以及作为 Case Class 的特例的 Scala Tuple）是复合类型，包含固定数量的各种类型的字段。Tuple 的字段从 1 开始索引。例如 `_1` 指第一个字段。Case Class 字段用名称索引。 - -{% highlight scala %} -case class WordCount(word: String, count: Int) -val input = env.fromElements( - WordCount("hello", 1), - WordCount("world", 2)) // Case Class 数据集 - -input.keyBy("word")// 以字段表达式“word”为键 - -val input2 = env.fromElements(("hello", 1), ("world", 2)) // Tuple2 数据集 - -input2.keyBy(0, 1) // 以第 0 和第 1 个字段为键 -{% endhighlight %} - ---- -#### 基本数据类型 - -Flink 支持所有 Java 和 Scala 的基本数据类型如 `Integer`、 `String`、和 `Double`。 - -#### 常规的类 - -Flink 支持大部分 Java 和 Scala 的类（API 和自定义）。 -除了包含无法序列化的字段的类，如文件指针，I / O流或其他本地资源。遵循 Java Beans 约定的类通常可以很好地工作。 - -Flink 对于所有未识别为 POJO 类型的类（请参阅上面对于的 POJO 要求）都作为常规类处理。 -Flink 将这些数据类型视为黑盒，并且无法访问其内容（为了诸如高效排序等目的）。常规类使用 [Kryo](https://github.com/EsotericSoftware/kryo) 序列化框架进行序列化和反序列化。 - -#### 值 - -*值* 类型手工描述其序列化和反序列化。它们不是通过通用序列化框架，而是通过实现 `org.apache.flinktypes.Value` 接口的 `read` 和 `write` 方法来为这些操作提供自定义编码。当通用序列化效率非常低时，使用值类型是合理的。例如，用数组实现稀疏向量。已知数组大部分元素为零，就可以对非零元素使用特殊编码，而通用序列化只会简单地将所有数组元素都写入。 - -`org.apache.flinktypes.CopyableValue` 接口以类似的方式支持内部手工克隆逻辑。 - -Flink 有与基本数据类型对应的预定义值类型。（`ByteValue`、 -`ShortValue`、 `IntValue`、`LongValue`、 `FloatValue`、`DoubleValue`、 `StringValue`、`CharValue`、 -`BooleanValue`）。这些值类型充当基本数据类型的可变变体：它们的值可以改变，允许程序员重用对象并减轻垃圾回收器的压力。 - - -#### Hadoop Writable - -可以使用实现了 `org.apache.hadoop.Writable` 接口的类型。它们会使用 `write()` 和 `readFields()` 方法中定义的序列化逻辑。 - -#### 特殊类型 - -可以使用特殊类型，包括 Scala 的 `Either`、`Option` 和 `Try`。 -Java API 有对 `Either` 的自定义实现。 -类似于 Scala 的 `Either`，它表示一个具有 *Left* 或 *Right* 两种可能类型的值。 -`Either` 可用于错误处理或需要输出两种不同类型记录的算子。 - -#### 类型擦除和类型推断 - -*请注意： 本节只与 Java 有关。* - -Java 编译器在编译后抛弃了大量泛型类型信息。这在 Java 中被称作 *类型擦除*。它意味着在运行时，对象的实例已经不知道它的泛型类型了。例如 `DataStream-{% highlight java %} -public class WordWithCount { - - public String word; - public int count; - - public WordWithCount() {} - - public WordWithCount(String word, int count) { - this.word = word; - this.count = count; - } -} - -DataStream-wordCounts = env.fromElements( - new WordWithCount("hello", 1), - new WordWithCount("world", 2)); - -wordCounts.keyBy("word"); // 以字段表达式“word”为键 - -{% endhighlight %} - -{% highlight scala %} -class WordWithCount(var word: String, var count: Int) { - def this() { - this(null, -1) - } -} - -val input = env.fromElements( - new WordWithCount("hello", 1), - new WordWithCount("world", 2)) // Case Class 数据集 - -input.keyBy("word")// 以字段表达式“word”为键 - -{% endhighlight %} --` 和 `DataStream ` 的实例在 JVM 看来是一样的。 - -Flink 在准备程序执行时（程序的 main 方法被调用时）需要类型信息。Flink Java API 尝试重建以各种方式丢弃的类型信息，并将其显式存储在数据集和算子中。你可以通过 `DataStream.getType()` 获取数据类型。此方法返回 `TypeInformation` 的一个实例，这是 Flink 内部表示类型的方式。 - -类型推断有其局限性，在某些情况下需要程序员的“配合”。 -这方面的示例是从集合创建数据集的方法，例如 `ExecutionEnvironment.fromCollection()`，你可以在这里传递一个描述类型的参数。 像 `MapFunction diff --git a/docs/dev/batch/index.zh.md b/docs/dev/batch/index.zh.md index 55a3be0a510546..bbb1173e7db933 100644 --- a/docs/dev/batch/index.zh.md +++ b/docs/dev/batch/index.zh.md @@ -32,11 +32,12 @@ example write the data to (distributed) files, or to standard output (for exampl terminal). Flink programs run in a variety of contexts, standalone, or embedded in other programs. The execution can happen in a local JVM, or on clusters of many machines. -Please see [basic concepts]({{ site.baseurl }}/dev/api_concepts.html) for an introduction -to the basic concepts of the Flink API. +Please refer to the [DataStream API overview]({% link dev/datastream_api.zh.md %}) +for an introduction to the basic concepts of the Flink API. That overview is +for the DataStream API but the basic concepts of the two APIs are the same. In order to create your own Flink DataSet program, we encourage you to start with the -[anatomy of a Flink Program]({{ site.baseurl }}/dev/api_concepts.html#anatomy-of-a-flink-program) +[anatomy of a Flink Program]({% link dev/datastream_api.zh.md %}#anatomy-of-a-flink-program) and gradually add your own [transformations](#dataset-transformations). The remaining sections act as references for additional operations and advanced features. @@ -49,7 +50,7 @@ Example Program The following program is a complete, working example of WordCount. You can copy & paste the code to run it locally. You only have to include the correct Flink's library into your project -(see Section [Linking with Flink]({{ site.baseurl }}/dev/projectsetup/dependencies.html)) and specify the imports. Then you are ready +(see Section [Linking with Flink]({{ site.baseurl }}/dev/project-configuration.html)) and specify the imports. Then you are ready to go!@@ -87,7 +165,7 @@ Apache Flink ships with first class support for a number of common deployment ta@@ -278,7 +279,7 @@ data.distinct(); Joins two data sets by creating all pairs of elements that are equal on their keys. Optionally uses a JoinFunction to turn the pair of elements into a single element, or a FlatJoinFunction to turn the pair of elements into arbitrarily many (including none) - elements. See the keys section to learn how to define join keys. + elements. See the keys section to learn how to define join keys. {% highlight java %} result = input1.join(input2) .where(0) // key of the first input (tuple field 0) @@ -304,7 +305,7 @@ result = input1.join(input2, JoinHint.BROADCAST_HASH_FIRST)OuterJoin - Performs a left, right, or full outer join on two data sets. Outer joins are similar to regular (inner) joins and create all pairs of elements that are equal on their keys. In addition, records of the "outer" side (left, right, or both in case of full) are preserved if no matching key is found in the other side. Matching pairs of elements (or one element and a nullvalue for the other input) are given to a JoinFunction to turn the pair of elements into a single element, or to a FlatJoinFunction to turn the pair of elements into arbitrarily many (including none) elements. See the keys section to learn how to define join keys. + Performs a left, right, or full outer join on two data sets. Outer joins are similar to regular (inner) joins and create all pairs of elements that are equal on their keys. In addition, records of the "outer" side (left, right, or both in case of full) are preserved if no matching key is found in the other side. Matching pairs of elements (or one element and anullvalue for the other input) are given to a JoinFunction to turn the pair of elements into a single element, or to a FlatJoinFunction to turn the pair of elements into arbitrarily many (including none) elements. See the keys section to learn how to define join keys. {% highlight java %} input1.leftOuterJoin(input2) // rightOuterJoin or fullOuterJoin for right or full outer joins .where(0) // key of the first input (tuple field 0) @@ -326,7 +327,7 @@ input1.leftOuterJoin(input2) // rightOuterJoin or fullOuterJoin for right or fulThe two-dimensional variant of the reduce operation. Groups each input on one or more fields and then joins the groups. The transformation function is called per pair of groups. - See the keys section to learn how to define coGroup keys.
+ See the keys section to learn how to define coGroup keys. {% highlight java %} data1.coGroup(data2) .where(0) @@ -600,7 +601,7 @@ data.distinct() Joins two data sets by creating all pairs of elements that are equal on their keys. Optionally uses a JoinFunction to turn the pair of elements into a single element, or a FlatJoinFunction to turn the pair of elements into arbitrarily many (including none) - elements. See the keys section to learn how to define join keys. + elements. See the keys section to learn how to define join keys. {% highlight scala %} // In this case tuple fields are used as keys. "0" is the join field on the first tuple // "1" is the join field on the second tuple. @@ -626,7 +627,7 @@ val result = input1.join(input2, JoinHint.BROADCAST_HASH_FIRST)OuterJoin - Performs a left, right, or full outer join on two data sets. Outer joins are similar to regular (inner) joins and create all pairs of elements that are equal on their keys. In addition, records of the "outer" side (left, right, or both in case of full) are preserved if no matching key is found in the other side. Matching pairs of elements (or one element and a `null` value for the other input) are given to a JoinFunction to turn the pair of elements into a single element, or to a FlatJoinFunction to turn the pair of elements into arbitrarily many (including none) elements. See the keys section to learn how to define join keys. + Performs a left, right, or full outer join on two data sets. Outer joins are similar to regular (inner) joins and create all pairs of elements that are equal on their keys. In addition, records of the "outer" side (left, right, or both in case of full) are preserved if no matching key is found in the other side. Matching pairs of elements (or one element and a `null` value for the other input) are given to a JoinFunction to turn the pair of elements into a single element, or to a FlatJoinFunction to turn the pair of elements into arbitrarily many (including none) elements. See the keys section to learn how to define join keys. {% highlight scala %} val joined = left.leftOuterJoin(right).where(0).equalTo(1) { (left, right) => @@ -642,7 +643,7 @@ val joined = left.leftOuterJoin(right).where(0).equalTo(1) { The two-dimensional variant of the reduce operation. Groups each input on one or more fields and then joins the groups. The transformation function is called per pair of groups. - See the keys section to learn how to define coGroup keys.
+ See the keys section to learn how to define coGroup keys. {% highlight scala %} data1.coGroup(data2).where(0).equalTo(1) {% endhighlight %} @@ -796,6 +797,230 @@ possible for [Data Sources](#data-sources) and [Data Sinks](#data-sinks). {% top %} +Specifying Keys +--------------- + +Some transformations (join, coGroup, groupBy) require that a key be defined on +a collection of elements. Other transformations (Reduce, GroupReduce, +Aggregate) allow data being grouped on a key before they are +applied. + +A DataSet is grouped as +{% highlight java %} +DataSet<...> input = // [...] +DataSet<...> reduced = input + .groupBy(/*define key here*/) + .reduceGroup(/*do something*/); +{% endhighlight %} + +The data model of Flink is not based on key-value pairs. Therefore, +you do not need to physically pack the data set types into keys and +values. Keys are "virtual": they are defined as functions over the +actual data to guide the grouping operator. + +### Define keys for Tuples +{:.no_toc} + +The simplest case is grouping Tuples on one or more +fields of the Tuple: + +++ +The tuples are grouped on the first field (the one of +Integer type). + ++{% highlight java %} +DataSet++{% highlight scala %} +val input: DataSet[(Int, String, Long)] = // [...] +val keyed = input.groupBy(0) +{% endhighlight %} ++++ +Here, we group the tuples on a composite key consisting of the first and the +second field. + +A note on nested Tuples: If you have a DataSet with a nested tuple, such as: + +{% highlight java %} +DataSet+{% highlight java %} +DataSet++{% highlight scala %} +val input: DataSet[(Int, String, Long)] = // [...] +val grouped = input.groupBy(0,1) +{% endhighlight %} ++++ +### Define keys using Key Selector Functions +{:.no_toc} + +An additional way to define keys are "key selector" functions. A key selector function +takes a single element as input and returns the key for the element. The key can be of any type and be derived from deterministic computations. + +The following example shows a key selector function that simply returns the field of an object: + ++ +In the example below, we have a `WC` POJO with two fields "word" and "count". To group by the field `word`, we just pass its name to the `groupBy()` function. +{% highlight java %} +// some ordinary POJO (Plain old Java Object) +public class WC { + public String word; + public int count; +} +DataSet+words = // [...] +DataSet wordCounts = words.groupBy("word") +{% endhighlight %} + +**Field Expression Syntax**: + +- Select POJO fields by their field name. For example `"user"` refers to the "user" field of a POJO type. + +- Select Tuple fields by their field name or 0-offset field index. For example `"f0"` and `"5"` refer to the first and sixth field of a Java Tuple type, respectively. + +- You can select nested fields in POJOs and Tuples. For example `"user.zip"` refers to the "zip" field of a POJO which is stored in the "user" field of a POJO type. Arbitrary nesting and mixing of POJOs and Tuples is supported such as `"f1.user.zip"` or `"user.f3.1.zip"`. + +- You can select the full type using the `"*"` wildcard expressions. This does also work for types which are not Tuple or POJO types. + +**Field Expression Example**: + +{% highlight java %} +public static class WC { + public ComplexNestedClass complex; //nested POJO + private int count; + // getter / setter for private field (count) + public int getCount() { + return count; + } + public void setCount(int c) { + this.count = c; + } +} +public static class ComplexNestedClass { + public Integer someNumber; + public float someFloat; + public Tuple3 + +In the example below, we have a `WC` POJO with two fields "word" and "count". To group by the field `word`, we just pass its name to the `groupBy()` function. +{% highlight scala %} +// some ordinary POJO (Plain old Java Object) +class WC(var word: String, var count: Int) { + def this() { this("", 0L) } +} +val words: DataSet[WC] = // [...] +val wordCounts = words.groupBy("word") + +// or, as a case class, which is less typing +case class WC(word: String, count: Int) +val words: DataSet[WC] = // [...] +val wordCounts = words.groupBy("word") +{% endhighlight %} + +**Field Expression Syntax**: + +- Select POJO fields by their field name. For example `"user"` refers to the "user" field of a POJO type. + +- Select Tuple fields by their 1-offset field name or 0-offset field index. For example `"_1"` and `"5"` refer to the first and sixth field of a Scala Tuple type, respectively. + +- You can select nested fields in POJOs and Tuples. For example `"user.zip"` refers to the "zip" field of a POJO which is stored in the "user" field of a POJO type. Arbitrary nesting and mixing of POJOs and Tuples is supported such as `"_2.user.zip"` or `"user._4.1.zip"`. + +- You can select the full type using the `"_"` wildcard expressions. This does also work for types which are not Tuple or POJO types. + +**Field Expression Example**: + +{% highlight scala %} +class WC(var complex: ComplexNestedClass, var count: Int) { + def this() { this(null, 0) } +} + +class ComplexNestedClass( + var someNumber: Int, + someFloat: Float, + word: (Long, Long, String), + hadoopCitizen: IntWritable) { + def this() { this(0, 0, (0, 0, ""), new IntWritable(0)) } +} +{% endhighlight %} + +These are valid field expressions for the example code above: + +- `"count"`: The count field in the `WC` class. + +- `"complex"`: Recursively selects all fields of the field complex of POJO type `ComplexNestedClass`. + +- `"complex.word._3"`: Selects the last field of the nested `Tuple3`. + +- `"complex.hadoopCitizen"`: Selects the Hadoop `IntWritable` type. + ++++ +{% top %} + Data Sources ------------ @@ -1199,7 +1424,7 @@ myResult.output( #### Locally Sorted Output -The output of a data sink can be locally sorted on specified fields in specified orders using [tuple field positions]({{ site.baseurl }}/dev/api_concepts.html#define-keys-for-tuples) or [field expressions]({{ site.baseurl }}/dev/api_concepts.html#define-keys-using-field-expressions). This works for every output format. +The output of a data sink can be locally sorted on specified fields in specified orders using [tuple field positions](#define-keys-for-tuples) or [field expressions](#define-keys-using-field-expressions). This works for every output format. The following examples show how to use this feature: @@ -1282,7 +1507,7 @@ values map { tuple => tuple._1 + " - " + tuple._2 } #### Locally Sorted Output -The output of a data sink can be locally sorted on specified fields in specified orders using [tuple field positions]({{ site.baseurl }}/dev/api_concepts.html#define-keys-for-tuples) or [field expressions]({{ site.baseurl }}/dev/api_concepts.html#define-keys-using-field-expressions). This works for every output format. +The output of a data sink can be locally sorted on specified fields in specified orders using [tuple field positions](#define-keys-for-tuples) or [field expressions](#define-keys-using-field-expressions). This works for every output format. The following examples show how to use this feature: @@ -1771,7 +1996,7 @@ This information is used by the optimizer to infer whether a data property such partitioning is preserved by a function. For functions that operate on groups of input elements such as `GroupReduce`, `GroupCombine`, `CoGroup`, and `MapPartition`, all fields that are defined as forwarded fields must always be jointly forwarded from the same input element. The forwarded fields of each element that is emitted by a group-wise function may originate from a different element of the function's input group. -Field forward information is specified using [field expressions]({{ site.baseurl }}/dev/api_concepts.html#define-keys-using-field-expressions). +Field forward information is specified using [field expressions](#define-keys-using-field-expressions). Fields that are forwarded to the same position in the output can be specified by their position. The specified position must be valid for the input and output data type and have the same type. For example the String `"f2"` declares that the third field of a Java input tuple is always equal to the third field in the output tuple. @@ -1840,7 +2065,7 @@ Non-forwarded field information for group-wise operators such as `GroupReduce`, **IMPORTANT**: The specification of non-forwarded fields information is optional. However if used, **ALL!** non-forwarded fields must be specified, because all other fields are considered to be forwarded in place. It is safe to declare a forwarded field as non-forwarded. -Non-forwarded fields are specified as a list of [field expressions]({{ site.baseurl }}/dev/api_concepts.html#define-keys-using-field-expressions). The list can be either given as a single String with field expressions separated by semicolons or as multiple Strings. +Non-forwarded fields are specified as a list of [field expressions](#define-keys-using-field-expressions). The list can be either given as a single String with field expressions separated by semicolons or as multiple Strings. For example both `"f1; f3"` and `"f1", "f3"` declare that the second and fourth field of a Java tuple are not preserved in place and all other fields are preserved in place. Non-forwarded field information can only be specified for functions which have identical input and output types. @@ -1891,7 +2116,7 @@ Fields which are only unmodified forwarded to the output without evaluating thei **IMPORTANT**: The specification of read fields information is optional. However if used, **ALL!** read fields must be specified. It is safe to declare a non-read field as read. -Read fields are specified as a list of [field expressions]({{ site.baseurl }}/dev/api_concepts.html#define-keys-using-field-expressions). The list can be either given as a single String with field expressions separated by semicolons or as multiple Strings. +Read fields are specified as a list of [field expressions](#define-keys-using-field-expressions). The list can be either given as a single String with field expressions separated by semicolons or as multiple Strings. For example both `"f1; f3"` and `"f1", "f3"` declare that the second and fourth field of a Java tuple are read and evaluated by the function. Read field information is specified as function class annotations using the following annotations: @@ -2045,7 +2270,7 @@ DataSet+{% highlight java %} +// some ordinary POJO +public class WC {public String word; public int count;} +DataSet+words = // [...] +UnsortedGrouping keyed = words + .groupBy(new KeySelector +{% highlight scala %} +// some ordinary case class +case class WC(word: String, count: Int) +val words: DataSet[WC] = // [...] +val keyed = words.groupBy( _.word ) +{% endhighlight %} ++result = input.map(new MyMapper()); env.execute(); {% endhighlight %} -Access the cached file or directory in a user function (here a `MapFunction`). The function must extend a [RichFunction]({{ site.baseurl }}/dev/api_concepts.html#rich-functions) class because it needs access to the `RuntimeContext`. +Access the cached file or directory in a user function (here a `MapFunction`). The function must extend a [RichFunction]({% link dev/user_defined_functions.zh.md %}#rich-functions) class because it needs access to the `RuntimeContext`. {% highlight java %} @@ -2091,7 +2316,7 @@ val result: DataSet[Integer] = input.map(new MyMapper()) env.execute() {% endhighlight %} -Access the cached file in a user function (here a `MapFunction`). The function must extend a [RichFunction]({{ site.baseurl }}/dev/api_concepts.html#rich-functions) class because it needs access to the `RuntimeContext`. +Access the cached file in a user function (here a `MapFunction`). The function must extend a [RichFunction]({% link dev/user_defined_functions.zh.md %}#rich-functions) class because it needs access to the `RuntimeContext`. {% highlight scala %} @@ -2164,7 +2389,7 @@ class MyFilter(limit: Int) extends FilterFunction[Int] { #### Via `withParameters(Configuration)` -This method takes a Configuration object as an argument, which will be passed to the [rich function]({{ site.baseurl }}/dev/api_concepts.html#rich-functions)'s `open()` +This method takes a Configuration object as an argument, which will be passed to the [rich function]({% link dev/user_defined_functions.zh.md %}#rich-functions)'s `open()` method. The Configuration object is a Map from String keys to different value types. diff --git a/docs/dev/cluster_execution.md b/docs/dev/cluster_execution.md index 2b290549ff76d9..8d5d43701e8023 100644 --- a/docs/dev/cluster_execution.md +++ b/docs/dev/cluster_execution.md @@ -62,7 +62,7 @@ The following illustrates the use of the `RemoteEnvironment`: {% highlight java %} public static void main(String[] args) throws Exception { ExecutionEnvironment env = ExecutionEnvironment - .createRemoteEnvironment("flink-master", 8081, "/home/user/udfs.jar"); + .createRemoteEnvironment("flink-jobmanager", 8081, "/home/user/udfs.jar"); DataSet-When encountering a corrupted message that cannot be deserialized for any reason, there -are two options - either throwing an exception from the `deserialize(...)` method -which will cause the job to fail and be restarted, or returning `null` to allow -the Flink Kafka consumer to silently skip the corrupted message. Note that -due to the consumer's fault tolerance (see below sections for more details), -failing the job on the corrupted message will let the consumer attempt -to deserialize the message again. Therefore, if deserialization still fails, the -consumer will fall into a non-stop restart and fail loop on that corrupted -message. +When encountering a corrupted message that cannot be deserialized for any reason the deserialization schema should return null which will result in the record being skipped. +Due to the consumer's fault tolerance (see below sections for more details), failing the job on the corrupted message will let the consumer attempt to deserialize the message again. +Therefore, if deserialization still fails, the consumer will fall into a non-stop restart and fail loop on that corrupted message. ### Kafka Consumers Start Position Configuration -The Flink Kafka Consumer allows configuring how the start position for Kafka -partitions are determined. - -Example: +The Flink Kafka Consumer allows configuring how the start positions for Kafka partitions are determined.data = env.readTextFile("hdfs://path/to/file"); diff --git a/docs/dev/cluster_execution.zh.md b/docs/dev/cluster_execution.zh.md index 60b1e26909d83f..4613a6a565d8e5 100644 --- a/docs/dev/cluster_execution.zh.md +++ b/docs/dev/cluster_execution.zh.md @@ -62,7 +62,7 @@ The following illustrates the use of the `RemoteEnvironment`: {% highlight java %} public static void main(String[] args) throws Exception { ExecutionEnvironment env = ExecutionEnvironment - .createRemoteEnvironment("flink-master", 8081, "/home/user/udfs.jar"); + .createRemoteEnvironment("flink-jobmanager", 8081, "/home/user/udfs.jar"); DataSet data = env.readTextFile("hdfs://path/to/file"); diff --git a/docs/dev/connectors/cassandra.md b/docs/dev/connectors/cassandra.md index 002d388bcef778..6420799510f9b1 100644 --- a/docs/dev/connectors/cassandra.md +++ b/docs/dev/connectors/cassandra.md @@ -43,7 +43,7 @@ To use this connector, add the following dependency to your project: {% endhighlight %} -Note that the streaming connectors are currently __NOT__ part of the binary distribution. See how to link with them for cluster execution [here]({{ site.baseurl}}/dev/projectsetup/dependencies.html). +Note that the streaming connectors are currently __NOT__ part of the binary distribution. See how to link with them for cluster execution [here]({{ site.baseurl}}/dev/project-configuration.html). ## Installing Apache Cassandra There are multiple ways to bring up a Cassandra instance on local machine: diff --git a/docs/dev/connectors/cassandra.zh.md b/docs/dev/connectors/cassandra.zh.md index 9a51387ea2b7aa..c10ea49820b1c0 100644 --- a/docs/dev/connectors/cassandra.zh.md +++ b/docs/dev/connectors/cassandra.zh.md @@ -43,7 +43,7 @@ To use this connector, add the following dependency to your project: {% endhighlight %} -Note that the streaming connectors are currently __NOT__ part of the binary distribution. See how to link with them for cluster execution [here]({{ site.baseurl}}/dev/projectsetup/dependencies.html). +Note that the streaming connectors are currently __NOT__ part of the binary distribution. See how to link with them for cluster execution [here]({{ site.baseurl}}/dev/project-configuration.html). ## Installing Apache Cassandra There are multiple ways to bring up a Cassandra instance on local machine: @@ -111,7 +111,7 @@ More details on [checkpoints docs]({{ site.baseurl }}/dev/stream/state/checkpoin ## Examples -The Cassandra sinks currently support both Tuple and POJO data types, and Flink automatically detects which type of input is used. For general use case of those streaming data type, please refer to [Supported Data Types]({{ site.baseurl }}/dev/api_concepts.html). We show two implementations based on [SocketWindowWordCount](https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/socket/SocketWindowWordCount.java), for Pojo and Tuple data types respectively. +The Cassandra sinks currently support both Tuple and POJO data types, and Flink automatically detects which type of input is used. For general use case of those streaming data type, please refer to [Supported Data Types]({% link dev/types_serialization.zh.md %}#supported-data-types). We show two implementations based on [SocketWindowWordCount](https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/socket/SocketWindowWordCount.java), for Pojo and Tuple data types respectively. In all these examples, we assumed the associated Keyspace `example` and Table `wordcount` have been created. diff --git a/docs/dev/connectors/elasticsearch.md b/docs/dev/connectors/elasticsearch.md index 5bc1404e3a4a6d..b613d0bf8998ea 100644 --- a/docs/dev/connectors/elasticsearch.md +++ b/docs/dev/connectors/elasticsearch.md @@ -59,7 +59,7 @@ of the Elasticsearch installation: Note that the streaming connectors are currently not part of the binary -distribution. See [here]({{site.baseurl}}/dev/projectsetup/dependencies.html) for information +distribution. See [here]({{site.baseurl}}/dev/project-configuration.html) for information about how to package the program with the libraries for cluster execution. ## Installing Elasticsearch @@ -317,7 +317,7 @@ time of checkpoints. This effectively assures that all requests before the checkpoint was triggered have been successfully acknowledged by Elasticsearch, before proceeding to process more records sent to the sink. -More details on checkpoints and fault tolerance are in the [fault tolerance docs]({{site.baseurl}}/internals/stream_checkpointing.html). +More details on checkpoints and fault tolerance are in the [fault tolerance docs]({{site.baseurl}}/learn-flink/fault_tolerance.html). To use fault tolerant Elasticsearch Sinks, checkpointing of the topology needs to be enabled at the execution environment: @@ -464,7 +464,7 @@ More information about Elasticsearch can be found [here](https://elastic.co). For the execution of your Flink program, it is recommended to build a so-called uber-jar (executable jar) containing all your dependencies -(see [here]({{site.baseurl}}/dev/projectsetup/dependencies.html) for further information). +(see [here]({{site.baseurl}}/dev/project-configuration.html) for further information). Alternatively, you can put the connector's jar file into Flink's `lib/` folder to make it available system-wide, i.e. for all job being run. diff --git a/docs/dev/connectors/elasticsearch.zh.md b/docs/dev/connectors/elasticsearch.zh.md index 59219543d83f47..e29c9208900333 100644 --- a/docs/dev/connectors/elasticsearch.zh.md +++ b/docs/dev/connectors/elasticsearch.zh.md @@ -59,7 +59,7 @@ of the Elasticsearch installation: Note that the streaming connectors are currently not part of the binary -distribution. See [here]({{site.baseurl}}/dev/projectsetup/dependencies.html) for information +distribution. See [here]({{site.baseurl}}/dev/project-configuration.html) for information about how to package the program with the libraries for cluster execution. ## Installing Elasticsearch @@ -317,7 +317,7 @@ time of checkpoints. This effectively assures that all requests before the checkpoint was triggered have been successfully acknowledged by Elasticsearch, before proceeding to process more records sent to the sink. -More details on checkpoints and fault tolerance are in the [fault tolerance docs]({{site.baseurl}}/internals/stream_checkpointing.html). +More details on checkpoints and fault tolerance are in the [fault tolerance docs]({{site.baseurl}}/zh/learn-flink/fault_tolerance.html). To use fault tolerant Elasticsearch Sinks, checkpointing of the topology needs to be enabled at the execution environment: @@ -464,7 +464,7 @@ More information about Elasticsearch can be found [here](https://elastic.co). For the execution of your Flink program, it is recommended to build a so-called uber-jar (executable jar) containing all your dependencies -(see [here]({{site.baseurl}}/dev/projectsetup/dependencies.html) for further information). +(see [here]({{site.baseurl}}/dev/project-configuration.html) for further information). Alternatively, you can put the connector's jar file into Flink's `lib/` folder to make it available system-wide, i.e. for all job being run. diff --git a/docs/dev/connectors/filesystem_sink.md b/docs/dev/connectors/filesystem_sink.md index f24478f3609562..f58ebb36c8f349 100644 --- a/docs/dev/connectors/filesystem_sink.md +++ b/docs/dev/connectors/filesystem_sink.md @@ -42,7 +42,7 @@ following dependency to your project: Note that the streaming connectors are currently not part of the binary distribution. See -[here]({{site.baseurl}}/dev/projectsetup/dependencies.html) +[here]({{site.baseurl}}/dev/project-configuration.html) for information about how to package the program with the libraries for cluster execution. diff --git a/docs/dev/connectors/filesystem_sink.zh.md b/docs/dev/connectors/filesystem_sink.zh.md index 1e570e684123ca..9ae22faa5027d9 100644 --- a/docs/dev/connectors/filesystem_sink.zh.md +++ b/docs/dev/connectors/filesystem_sink.zh.md @@ -39,7 +39,7 @@ under the License. {% endhighlight %} -注意连接器目前还不是二进制发行版的一部分，添加依赖、打包配置以及集群运行信息请参考 [这里]({{site.baseurl}}/zh/dev/projectsetup/dependencies.html)。 +注意连接器目前还不是二进制发行版的一部分，添加依赖、打包配置以及集群运行信息请参考 [这里]({{site.baseurl}}/zh/dev/project-configuration.html)。 #### 分桶文件 Sink diff --git a/docs/dev/connectors/jdbc.md b/docs/dev/connectors/jdbc.md index 927a28fbf9e35c..d3fc286beb84e0 100644 --- a/docs/dev/connectors/jdbc.md +++ b/docs/dev/connectors/jdbc.md @@ -26,36 +26,6 @@ under the License. * This will be replaced by the TOC {:toc} - ---- -title: "JDBC Connector" -nav-title: JDBC -nav-parent_id: connectors -nav-pos: 9 ---- - - -* This will be replaced by the TOC -{:toc} - - This connector provides a sink that writes data to a JDBC database. To use it, add the following dependency to your project (along with your JDBC-driver): @@ -68,7 +38,7 @@ To use it, add the following dependency to your project (along with your JDBC-dr {% endhighlight %} -Note that the streaming connectors are currently __NOT__ part of the binary distribution. See how to link with them for cluster execution [here]({{ site.baseurl}}/dev/projectsetup/dependencies.html). +Note that the streaming connectors are currently __NOT__ part of the binary distribution. See how to link with them for cluster execution [here]({{ site.baseurl}}/dev/project-configuration.html). Created JDBC sink provides at-least-once guarantee. Effectively exactly-once can be achived using upsert statements or idempotent updates. diff --git a/docs/dev/connectors/jdbc.zh.md b/docs/dev/connectors/jdbc.zh.md index 927a28fbf9e35c..d3fc286beb84e0 100644 --- a/docs/dev/connectors/jdbc.zh.md +++ b/docs/dev/connectors/jdbc.zh.md @@ -26,36 +26,6 @@ under the License. * This will be replaced by the TOC {:toc} - ---- -title: "JDBC Connector" -nav-title: JDBC -nav-parent_id: connectors -nav-pos: 9 ---- - - -* This will be replaced by the TOC -{:toc} - - This connector provides a sink that writes data to a JDBC database. To use it, add the following dependency to your project (along with your JDBC-driver): @@ -68,7 +38,7 @@ To use it, add the following dependency to your project (along with your JDBC-dr {% endhighlight %} -Note that the streaming connectors are currently __NOT__ part of the binary distribution. See how to link with them for cluster execution [here]({{ site.baseurl}}/dev/projectsetup/dependencies.html). +Note that the streaming connectors are currently __NOT__ part of the binary distribution. See how to link with them for cluster execution [here]({{ site.baseurl}}/dev/project-configuration.html). Created JDBC sink provides at-least-once guarantee. Effectively exactly-once can be achived using upsert statements or idempotent updates. diff --git a/docs/dev/connectors/kafka.md b/docs/dev/connectors/kafka.md index d084808c01f8da..f2e722f1417142 100644 --- a/docs/dev/connectors/kafka.md +++ b/docs/dev/connectors/kafka.md @@ -23,125 +23,59 @@ specific language governing permissions and limitations under the License. --> +Flink provides an [Apache Kafka](https://kafka.apache.org) connector for reading data from and writing data to Kafka topics with exactly-once guaruntees. + * This will be replaced by the TOC {:toc} -This connector provides access to event streams served by [Apache Kafka](https://kafka.apache.org/). - -Flink provides special Kafka Connectors for reading and writing data from/to Kafka topics. -The Flink Kafka Consumer integrates with Flink's checkpointing mechanism to provide -exactly-once processing semantics. To achieve that, Flink does not purely rely on Kafka's consumer group -offset tracking, but tracks and checkpoints these offsets internally as well. - -Please pick a package (maven artifact id) and class name for your use-case and environment. -For most users, the `FlinkKafkaConsumer010` (part of `flink-connector-kafka`) is appropriate. - - - -
- -Then, import the connector in your maven project: +## Dependency + +Apache Flink ships with multiple Kafka connectors: universal, 0.10, and 0.11. +This universal Kafka connector attempts to track the latest version of the Kafka client. +The version of the client it uses may change between Flink releases. +Modern Kafka clients are backwards compatible with broker versions 0.10.0 or later. +For most users the universal Kafka connector is the most appropriate. +However, for Kafka versions 0.11.x and 0.10.x, we recommend using the dedicated ``0.11`` and ``0.10`` connectors, respectively. +For details on Kafka compatibility, please refer to the official [Kafka documentation](https://kafka.apache.org/protocol.html#protocol_compatibility). +- - - -Maven Dependency -Supported since -Consumer and -
- Producer Class nameKafka version -Notes -- -flink-connector-kafka-0.10{{ site.scala_version_suffix }} -1.2.0 -FlinkKafkaConsumer010 -
- FlinkKafkaProducer0100.10.x -This connector supports Kafka messages with timestamps both for producing and consuming. -- -flink-connector-kafka-0.11{{ site.scala_version_suffix }} -1.4.0 -FlinkKafkaConsumer011 -
- FlinkKafkaProducer0110.11.x -Since 0.11.x Kafka does not support scala 2.10. This connector supports Kafka transactional messaging to provide exactly once semantic for the producer. -- - -flink-connector-kafka{{ site.scala_version_suffix }} -1.7.0 -FlinkKafkaConsumer -
- FlinkKafkaProducer>= 1.0.0 -- This universal Kafka connector attempts to track the latest version of the Kafka client. - The version of the client it uses may change between Flink releases. Starting with Flink 1.9 release, it uses the Kafka 2.2.0 client. - Modern Kafka clients are backwards compatible with broker versions 0.10.0 or later. - However for Kafka 0.11.x and 0.10.x versions, we recommend using dedicated - flink-connector-kafka-0.11{{ site.scala_version_suffix }} and flink-connector-kafka-0.10{{ site.scala_version_suffix }} respectively. - -+-Then instantiate the new source (`FlinkKafkaConsumer`) and sink (`FlinkKafkaProducer`). -The API is backward compatible with the Kafka 0.11 connector, -except of dropping specific Kafka version from the module and class names. +Flink's streaming connectors are not currently part of the binary distribution. +See how to link with them for cluster execution [here]({{ site.baseurl}}/dev/project-configuration.html). ## Kafka Consumer -Flink's Kafka consumer is called `FlinkKafkaConsumer010` (or 011 for Kafka 0.11.0.x versions, etc. -or just `FlinkKafkaConsumer` for Kafka >= 1.0.0 versions). It provides access to one or more Kafka topics. +Flink's Kafka consumer - `FlinkKafkaConsumer` (or `FlinkKafkaConsumer011` for Kafka 0.11.x, +or `FlinkKafkaConsumer010` for Kafka 0.10.x) - provides access to read from one or more Kafka topics. The constructor accepts the following arguments: @@ -152,8 +86,6 @@ The constructor accepts the following arguments: - "bootstrap.servers" (comma separated list of Kafka brokers) - "group.id" the id of the consumer group -Example: -{% highlight xml %}+- +{% endhighlight %} +org.apache.flink -flink-connector-kafka{{ site.scala_version_suffix }} -{{ site.version }} +org.apache.flink +flink-connector-kafka{{ site.scala_version_suffix }} +{{ site.version }} ++{% highlight xml %} +++ {% endhighlight %} - -Note that the streaming connectors are currently not part of the binary distribution. -See how to link with them for cluster execution [here]({{ site.baseurl}}/dev/projectsetup/dependencies.html). - -## Installing Apache Kafka - -* Follow the instructions from [Kafka's quickstart](https://kafka.apache.org/documentation.html#quickstart) to download the code and launch a server (launching a Zookeeper and a Kafka server is required every time before starting the application). -* If the Kafka and Zookeeper servers are running on a remote machine, then the `advertised.host.name` setting in the `config/server.properties` file must be set to the machine's IP address. - -## Kafka 1.0.0+ Connector - -Starting with Flink 1.7, there is a new universal Kafka connector that does not track a specific Kafka major version. -Rather, it tracks the latest version of Kafka at the time of the Flink release. - -If your Kafka broker version is 1.0.0 or newer, you should use this Kafka connector. -If you use an older version of Kafka (0.11 or 0.10), you should use the connector corresponding to the broker version. - -### Compatibility - -The universal Kafka connector is compatible with older and newer Kafka brokers through the compatibility guarantees of the Kafka client API and broker. -It is compatible with broker versions 0.11.0 or newer, depending on the features used. -For details on Kafka compatibility, please refer to the [Kafka documentation](https://kafka.apache.org/protocol.html#protocol_compatibility). - -### Migrating Kafka Connector from 0.11 to universal - -In order to perform the migration, see the [upgrading jobs and Flink versions guide]({{ site.baseurl }}/ops/upgrading.html) -and: -* Use Flink 1.9 or newer for the whole process. -* Do not upgrade the Flink and operators at the same time. -* Make sure that Kafka Consumer and/or Kafka Producer used in your job have assigned unique identifiers (`uid`): -* Use stop with savepoint feature to take the savepoint (for example by using `stop --withSavepoint`)[CLI command]({{ site.baseurl }}/ops/cli.html). - -### Usage - -To use the universal Kafka connector add a dependency to it: - +org.apache.flink +flink-connector-kafka-011{{ site.scala_version_suffix }} +{{ site.version }} {% highlight xml %}+- {% endhighlight %} +Attention The ``0.10`` sink does not support exactly-once writes to Kafka. +org.apache.flink -flink-connector-kafka{{ site.scala_version_suffix }} -{{ site.version }} +org.apache.flink +flink-connector-kafka-010{{ site.scala_version_suffix }} +{{ site.version }} ### The `DeserializationSchema` -The Flink Kafka Consumer needs to know how to turn the binary data in Kafka into Java/Scala objects. The -`DeserializationSchema` allows users to specify such a schema. The `T deserialize(byte[] message)` -method gets called for each Kafka message, passing the value from Kafka. - -It is usually helpful to start from the `AbstractDeserializationSchema`, which takes care of describing the -produced Java/Scala type to Flink's type system. Users that implement a vanilla `DeserializationSchema` need -to implement the `getProducedType(...)` method themselves. +The Flink Kafka Consumer needs to know how to turn the binary data in Kafka into Java/Scala objects. +The `KafkaDeserializationSchema` allows users to specify such a schema. The `T deserialize(ConsumerRecord{% highlight java %} @@ -161,7 +93,7 @@ Properties properties = new Properties(); properties.setProperty("bootstrap.servers", "localhost:9092"); properties.setProperty("group.id", "test"); DataStreamstream = env - .addSource(new FlinkKafkaConsumer010<>("topic", new SimpleStringSchema(), properties)); + .addSource(new FlinkKafkaConsumer<>("topic", new SimpleStringSchema(), properties)); {% endhighlight %} @@ -170,26 +102,17 @@ val properties = new Properties() properties.setProperty("bootstrap.servers", "localhost:9092") properties.setProperty("group.id", "test") stream = env - .addSource(new FlinkKafkaConsumer010[String]("topic", new SimpleStringSchema(), properties)) - .print() + .addSource(new FlinkKafkaConsumer[String]("topic", new SimpleStringSchema(), properties)) {% endhighlight %}{% highlight xml %}- {% endhighlight %}org.apache.flink -flink-avro -{{site.version }} +org.apache.flink +flink-avro +{{site.version }} {% highlight xml %}- {% endhighlight %}org.apache.flink -flink-avro-confluent-registry -{{site.version }} +org.apache.flink +flink-avro-confluent-registry +{{site.version }} +#### Avro格式 + +Flink 也提供了将数据写入 Avro 文件的内置支持。对于创建 AvroWriterFactory 的快捷方法，更多信息可以参考 +[AvroWriters]({{ site.javadocs_baseurl }}/api/java/org/apache/flink/formats/avro/AvroWriters.html). + +使用Avro相关的Writer需要在项目中添加以下依赖： + +{% highlight xml %} +{% highlight java %} final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); -FlinkKafkaConsumer010Then a child implementation to convert the element of type `Person` and set them in the `VectorizedRowBatch` can be like: @@ -631,6 +733,9 @@ Given this, when trying to restore from an old checkpoint/savepoint which assume by subsequent successful checkpoints, Flink will refuse to resume and it will throw an exception as it cannot locate the in-progress file. +Important Note 4: Currently, the `StreamingFileSink` only supports three filesystems: +HDFS, S3, and Local. Flink will throw an exception when using an unsupported filesystem at runtime. + ### S3-specific Important Note 1: For S3, the `StreamingFileSink` diff --git a/docs/dev/connectors/streamfile_sink.zh.md b/docs/dev/connectors/streamfile_sink.zh.md index bd74bef5290533..4e7fe46a1f7c8c 100644 --- a/docs/dev/connectors/streamfile_sink.zh.md +++ b/docs/dev/connectors/streamfile_sink.zh.md @@ -122,11 +122,13 @@ input.addSink(sink) 批量编码 Sink 的创建与行编码 Sink 相似，不过在这里我们不是指定编码器 `Encoder` 而是指定 [BulkWriter.Factory]({{ site.javadocs_baseurl }}/api/java/org/apache/flink/api/common/serialization/BulkWriter.Factory.html) 。 `BulkWriter` 定义了如何添加、刷新元素，以及如何批量编码。 -Flink 有三个内置的 BulkWriter Factory ： +Flink 有四个内置的 BulkWriter Factory ： - [ParquetWriterFactory]({{ site.javadocs_baseurl }}/api/java/org/apache/flink/formats/parquet/ParquetWriterFactory.html) + - [AvroWriterFactory]({{ site.javadocs_baseurl }}/api/java/org/apache/flink/formats/avro/AvroWriterFactory.html) - [SequenceFileWriterFactory]({{ site.javadocs_baseurl }}/api/java/org/apache/flink/formats/sequencefile/SequenceFileWriterFactory.html) - [CompressWriterFactory]({{ site.javadocs_baseurl }}/api/java/org/apache/flink/formats/compress/CompressWriterFactory.html) + - [OrcBulkWriterFactory]({{ site.javadocs_baseurl }}/api/java/org/apache/flink/orc/writer/OrcBulkWriterFactory.html)myConsumer = new FlinkKafkaConsumer010<>(...); +FlinkKafkaConsumer myConsumer = new FlinkKafkaConsumer<>(...); myConsumer.setStartFromEarliest(); // start from the earliest record possible myConsumer.setStartFromLatest(); // start from the latest record myConsumer.setStartFromTimestamp(...); // start from specified epoch timestamp (milliseconds) @@ -269,7 +183,7 @@ DataStream stream = env.addSource(myConsumer); {% highlight scala %} val env = StreamExecutionEnvironment.getExecutionEnvironment() -val myConsumer = new FlinkKafkaConsumer010[String](...) +val myConsumer = new FlinkKafkaConsumer[String](...) myConsumer.setStartFromEarliest() // start from the earliest record possible myConsumer.setStartFromLatest() // start from the latest record myConsumer.setStartFromTimestamp(...) // start from specified epoch timestamp (milliseconds) @@ -339,34 +253,14 @@ fault tolerance for the consumer). ### Kafka Consumers and Fault Tolerance With Flink's checkpointing enabled, the Flink Kafka Consumer will consume records from a topic and periodically checkpoint all -its Kafka offsets, together with the state of other operations, in a consistent manner. In case of a job failure, Flink will restore +its Kafka offsets, together with the state of other operations. In case of a job failure, Flink will restore the streaming program to the state of the latest checkpoint and re-consume the records from Kafka, starting from the offsets that were stored in the checkpoint. The interval of drawing checkpoints therefore defines how much the program may have to go back at most, in case of a failure. +To use fault tolerant Kafka Consumers, checkpointing of the topology needs to be enabled in the [job]({{ site.baseurl }}/ops/config.html#execution-checkpointing-interval). -To use fault tolerant Kafka Consumers, checkpointing of the topology needs to be enabled at the execution environment: - - -- -Also note that Flink can only restart the topology if enough processing slots are available to restart the topology. -So if the topology fails due to loss of a TaskManager, there must still be enough slots available afterwards. -Flink on YARN supports automatic restart of lost YARN containers. - -If checkpointing is not enabled, the Kafka consumer will periodically commit the offsets to Zookeeper. +If checkpointing is disabled, the Kafka consumer will periodically commit the offsets to Zookeeper. ### Kafka Consumers Topic and Partition Discovery @@ -380,15 +274,9 @@ By default, partition discovery is disabled. To enable it, set a non-negative va for `flink.partition-discovery.interval-millis` in the provided properties config, representing the discovery interval in milliseconds. -Limitation When the consumer is restored from a savepoint from Flink versions -prior to Flink 1.3.x, partition discovery cannot be enabled on the restore run. If enabled, the restore would fail -with an exception. In this case, in order to use partition discovery, please first take a savepoint in Flink 1.3.x and -then restore again from that. - #### Topic discovery -At a higher-level, the Flink Kafka Consumer is also capable of discovering topics, based on pattern matching on the -topic names using regular expressions. See the below for an example: +The Kafka Consumer is also capable of discovering topics by matching topic names using regular expressions.-{% highlight java %} -final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); -env.enableCheckpointing(5000); // checkpoint every 5000 msecs -{% endhighlight %} ---{% highlight scala %} -val env = StreamExecutionEnvironment.getExecutionEnvironment() -env.enableCheckpointing(5000) // checkpoint every 5000 msecs -{% endhighlight %} --+@@ -399,7 +287,7 @@ Properties properties = new Properties(); properties.setProperty("bootstrap.servers", "localhost:9092"); properties.setProperty("group.id", "test"); -FlinkKafkaConsumer011+#### Avro format + +Flink also provides built-in support for writing data into Avro files. A list of convenience methods to create +Avro writer factories and their associated documentation can be found in the +[AvroWriters]({{ site.javadocs_baseurl }}/api/java/org/apache/flink/formats/avro/AvroWriters.html) class. + +To use the Avro writers in your application you need to add the following dependency: + +{% highlight xml %} +myConsumer = new FlinkKafkaConsumer011<>( +FlinkKafkaConsumer myConsumer = new FlinkKafkaConsumer<>( java.util.regex.Pattern.compile("test-topic-[0-9]"), new SimpleStringSchema(), properties); @@ -416,7 +304,7 @@ val properties = new Properties() properties.setProperty("bootstrap.servers", "localhost:9092") properties.setProperty("group.id", "test") -val myConsumer = new FlinkKafkaConsumer010[String]( +val myConsumer = new FlinkKafkaConsumer[String]( java.util.regex.Pattern.compile("test-topic-[0-9]"), new SimpleStringSchema, properties) @@ -445,7 +333,7 @@ tolerance guarantees. The committed offsets are only a means to expose the consumer's progress for monitoring purposes. The way to configure offset commit behaviour is different, depending on -whether or not checkpointing is enabled for the job. +whether checkpointing is enabled for the job. - *Checkpointing disabled:* if checkpointing is disabled, the Flink Kafka Consumer relies on the automatic periodic offset committing capability @@ -465,15 +353,14 @@ whether or not checkpointing is enabled for the job. ### Kafka Consumers and Timestamp Extraction/Watermark Emission -In many scenarios, the timestamp of a record is embedded (explicitly or implicitly) in the record itself. -In addition, the user may want to emit watermarks either periodically, or in an irregular fashion, e.g. based on +In many scenarios, the timestamp of a record is embedded in the record itself, or the metadata of the `ConsumerRecord`. +In addition, users may want to emit watermarks either periodically, or irregularly, e.g. based on special records in the Kafka stream that contain the current event-time watermark. For these cases, the Flink Kafka -Consumer allows the specification of an `AssignerWithPeriodicWatermarks` or an `AssignerWithPunctuatedWatermarks`. +Consumer allows the specification of a [watermark strategy]({% link dev/event_time.md %}). -You can specify your custom timestamp extractor/watermark emitter as described -[here]({{ site.baseurl }}/dev/event_timestamps_watermarks.html), or use one from the -[predefined ones]({{ site.baseurl }}/dev/event_timestamp_extractors.html). After doing so, you -can pass it to your consumer in the following way: +You can specify your custom strategy as described +[here]({% link dev/event_timestamps_watermarks.md %}), or use one from the +[predefined ones]({% link dev/event_timestamp_extractors.md %}). -Internally, an instance of the assigner is executed per Kafka partition. -When such an assigner is specified, for each record read from Kafka, the -`extractTimestamp(T element, long previousElementTimestamp)` is called to assign a timestamp to the record and -the `Watermark getCurrentWatermark()` (for periodic) or the -`Watermark checkAndGetNextWatermark(T lastElement, long extractedTimestamp)` (for punctuated) is called to determine -if a new watermark should be emitted and with which timestamp. **Note**: If a watermark assigner depends on records read from Kafka to advance its watermarks (which is commonly the case), all topics and partitions need to have a continuous stream of records. Otherwise, the watermarks of the whole application cannot advance and all time-based operations, such as time windows or functions with timers, cannot make progress. A single idle Kafka partition causes this behavior. -A Flink improvement is planned to prevent this from happening -(see [FLINK-5479: Per-partition watermarks in FlinkKafkaConsumer should consider idle partitions]( -https://issues.apache.org/jira/browse/FLINK-5479)). -In the meanwhile, a possible workaround is to send *heartbeat messages* to all consumed partitions that advance the watermarks of idle partitions. - +Consider setting appropriate [idelness timeouts]({{ site.baseurl }}/dev/event_timestamps_watermarks.html#dealing-with-idle-sources) to mitigate this issue. + ## Kafka Producer -Flink’s Kafka Producer is called `FlinkKafkaProducer011` (or `010` for Kafka 0.10.0.x versions, etc. or just `FlinkKafkaProducer` for Kafka >= 1.0.0 versions). -It allows writing a stream of records to one or more Kafka topics. +Flink’s Kafka Producer - `FlinkKafkaProducer` (or `FlinkKafkaProducer010` for Kafka 0.10.x versions or `FlinkKafkaProducer011` for Kafka 0.11.x versions) - + allows writing a stream of records to one or more Kafka topics. + +The constructor accepts the following arguments: -Example: +1. A default output topic where events should be written +2. A SerializationSchema / KafkaSerializationSchema for serializing data into Kafka +3. Properties for the Kafka client. The following properties are required: + * "bootstrap.servers" (comma separated list of Kafka brokers) +4. A fault-tolerance semantic@@ -482,13 +369,13 @@ Properties properties = new Properties(); properties.setProperty("bootstrap.servers", "localhost:9092"); properties.setProperty("group.id", "test"); -FlinkKafkaConsumer010myConsumer = - new FlinkKafkaConsumer010<>("topic", new SimpleStringSchema(), properties); -myConsumer.assignTimestampsAndWatermarks(new CustomWatermarkEmitter()); +FlinkKafkaConsumer myConsumer = + new FlinkKafkaConsumer<>("topic", new SimpleStringSchema(), properties); +myConsumer.assignTimestampsAndWatermarks( + WatermarkStrategy. + .forBoundedOutOfOrderness(Duration.ofSeconds(20))); -DataStream stream = env - .addSource(myConsumer) - .print(); +DataStream stream = env.addSource(myConsumer); {% endhighlight %} @@ -497,51 +384,50 @@ val properties = new Properties() properties.setProperty("bootstrap.servers", "localhost:9092") properties.setProperty("group.id", "test") -val myConsumer = new FlinkKafkaConsumer010[String]("topic", new SimpleStringSchema(), properties) -myConsumer.assignTimestampsAndWatermarks(new CustomWatermarkEmitter()) -stream = env - .addSource(myConsumer) - .print() +val myConsumer = + new FlinkKafkaConsumer("topic", new SimpleStringSchema(), properties); +myConsumer.assignTimestampsAndWatermarks( + WatermarkStrategy. + .forBoundedOutOfOrderness(Duration.ofSeconds(20))) + +val stream = env.addSource(myConsumer) {% endhighlight %}-The above examples demonstrate the basic usage of creating a Flink Kafka Producer -to write streams to a single Kafka target topic. For more advanced usages, there -are other constructor variants that allow providing the following: - - * *Providing custom properties*: - The producer allows providing a custom properties configuration for the internal `KafkaProducer`. - Please refer to the [Apache Kafka documentation](https://kafka.apache.org/documentation.html) for - details on how to configure Kafka Producers. - * *Custom partitioner*: To assign records to specific - partitions, you can provide an implementation of a `FlinkKafkaPartitioner` to the - constructor. This partitioner will be called for each record in the stream - to determine which exact partition of the target topic the record should be sent to. - Please see [Kafka Producer Partitioning Scheme](#kafka-producer-partitioning-scheme) for more details. - * *Advanced serialization schema*: Similar to the consumer, - the producer also allows using an advanced serialization schema called `KeyedSerializationSchema`, - which allows serializing the key and value separately. It also allows to override the target topic, - so that one producer instance can send data to multiple topics. - -### Kafka Producer Partitioning Scheme - -By default, if a custom partitioner is not specified for the Flink Kafka Producer, the producer will use -a `FlinkFixedPartitioner` that maps each Flink Kafka Producer parallel subtask to a single Kafka partition -(i.e., all records received by a sink subtask will end up in the same Kafka partition). - -A custom partitioner can be implemented by extending the `FlinkKafkaPartitioner` class. All -Kafka versions' constructors allow providing a custom partitioner when instantiating the producer. -Note that the partitioner implementation must be serializable, as they will be transferred across Flink nodes. -Also, keep in mind that any state in the partitioner will be lost on job failures since the partitioner -is not part of the producer's checkpointed state. - -It is also possible to completely avoid using and kind of partitioner, and simply let Kafka partition -the written records by their attached key (as determined for each record using the provided serialization schema). -To do this, provide a `null` custom partitioner when instantiating the producer. It is important -to provide `null` as the custom partitioner; as explained above, if a custom partitioner is not specified -the `FlinkFixedPartitioner` is used instead. - -### Kafka Producers and Fault Tolerance +## The `SerializationSchema` -#### Kafka 0.10 +The Flink Kafka Producer needs to know how to turn Java/Scala objects into binary data. +The `KafkaSerializationSchema` allows users to specify such a schema. +The `ProducerRecord{% highlight java %} -DataStreamstream = ...; +DataStream stream = ... -FlinkKafkaProducer011 myProducer = new FlinkKafkaProducer011 ( - "localhost:9092", // broker list - "my-topic", // target topic - new SimpleStringSchema()); // serialization schema +Properties properties = new Properties(); +properties.setProperty("bootstrap.servers", "localhost:9092"); -// versions 0.10+ allow attaching the records' event timestamp when writing them to Kafka; -// this method is not available for earlier Kafka versions -myProducer.setWriteTimestampToKafka(true); +FlinkKafkaProducer myProducer = new FlinkKafkaProducer<>( + "my-topic", // target topic + new SimpleStringSchema(), // serialization schema + properties, // producer config + FlinkKafkaProducer.Semantic.EXACTLY_ONCE); // fault-tolerance stream.addSink(myProducer); {% endhighlight %} @@ -550,91 +436,36 @@ stream.addSink(myProducer); {% highlight scala %} val stream: DataStream[String] = ... -val myProducer = new FlinkKafkaProducer011[String]( - "localhost:9092", // broker list - "my-topic", // target topic - new SimpleStringSchema) // serialization schema +Properties properties = new Properties +properties.setProperty("bootstrap.servers", "localhost:9092") -// versions 0.10+ allow attaching the records' event timestamp when writing them to Kafka; -// this method is not available for earlier Kafka versions -myProducer.setWriteTimestampToKafka(true) +val myProducer = new FlinkKafkaProducer[String]( + "my-topic", // target topic + new SimpleStringSchema(), // serialization schema + properties, // producer config + FlinkKafkaProducer.Semantic.EXACTLY_ONCE) // fault-tolerance stream.addSink(myProducer) {% endhighlight %} +-## Kafka Connector metrics +## Kafka Connector Metrics Flink's Kafka connectors provide some metrics through Flink's [metrics system]({{ site.baseurl }}/monitoring/metrics.html) to analyze the behavior of the connector. -The producers export Kafka's internal metrics through Flink's metric system for all supported versions. The consumers export -all metrics starting from Kafka version 0.10. The Kafka documentation lists all exported metrics -in its [documentation](http://kafka.apache.org/documentation/#selector_monitoring). +The producers export Kafka's internal metrics through Flink's metric system for all supported versions. +The Kafka documentation lists all exported metrics in its [documentation](http://kafka.apache.org/documentation/#selector_monitoring). In addition to these metrics, all consumers expose the `current-offsets` and `committed-offsets` for each topic partition. The `current-offsets` refers to the current offset in the partition. This refers to the offset of the last element that we retrieved and emitted successfully. The `committed-offsets` is the last committed offset. -The Kafka Consumers in Flink commit the offsets back to the Kafka brokers (Kafka 0.10+). If checkpointing -is disabled, offsets are committed periodically. +The Kafka Consumers in Flink commit the offsets back to the Kafka brokers. +If checkpointing is disabled, offsets are committed periodically. With checkpointing, the commit happens once all operators in the streaming topology have confirmed that they've created a checkpoint of their state. This provides users with at-least-once semantics for the offsets committed to Zookeeper or the broker. For offsets checkpointed to Flink, the system provides exactly once guarantees. @@ -776,6 +605,15 @@ A mismatch in service name between client and server configuration will cause th For more information on Flink configuration for Kerberos security, please see [here]({{ site.baseurl}}/ops/config.html). You can also find [here]({{ site.baseurl}}/ops/security-kerberos.html) further details on how Flink internally setups Kerberos-based security. +## Migrating Kafka Connector from 0.11 to universal + +In order to perform the migration, see the [upgrading jobs and Flink versions guide]({{ site.baseurl }}/ops/upgrading.html) +and: +* Use Flink 1.9 or newer for the whole process. +* Do not upgrade Flink and user operators at the same time. +* Make sure that Kafka Consumer and/or Kafka Producer used in your job have assigned unique identifiers (`uid`): +* Use stop with savepoint feature to take the savepoint (for example by using `stop --withSavepoint`)[CLI command]({{ site.baseurl }}/ops/cli.html). + ## TroubleshootingWith Flink's checkpointing enabled, the `FlinkKafkaProducer011` (`FlinkKafkaProducer` for Kafka >= 1.0.0 versions) can provide exactly-once delivery guarantees. @@ -643,9 +474,8 @@ chosen by passing appropriate `semantic` parameter to the `FlinkKafkaProducer011 * `Semantic.NONE`: Flink will not guarantee anything. Produced records can be lost or they can be duplicated. - * `Semantic.AT_LEAST_ONCE` (default setting): similar to `setFlushOnCheckpoint(true)` in - `FlinkKafkaProducer010`. This guarantees that no records will be lost (although they can be duplicated). - * `Semantic.EXACTLY_ONCE`: uses Kafka transactions to provide exactly-once semantic. Whenever you write + * `Semantic.AT_LEAST_ONCE` (default setting): This guarantees that no records will be lost (although they can be duplicated). + * `Semantic.EXACTLY_ONCE`: Kafka transactions will be used to provide exactly-once semantic. Whenever you write to Kafka using transactions, do not forget about setting desired `isolation.level` (`read_committed` or `read_uncommitted` - the latter one is the default value) for any application consuming records from Kafka. @@ -695,54 +525,53 @@ event of failure of Flink application before first checkpoint, after restarting is no information in the system about previous pool sizes. Thus it is unsafe to scale down Flink application before first checkpoint completes, by factor larger than `FlinkKafkaProducer011.SAFE_SCALE_DOWN_FACTOR`. -## Using Kafka timestamps and Flink event time in Kafka 0.10 - -Since Apache Kafka 0.10+, Kafka's messages can carry -[timestamps](https://cwiki.apache.org/confluence/display/KAFKA/KIP-32+-+Add+timestamps+to+Kafka+message), indicating -the time the event has occurred (see ["event time" in Apache Flink](../event_time.html)) or the time when the message -has been written to the Kafka broker. - -The `FlinkKafkaConsumer010` will emit records with the timestamp attached, if the time characteristic in Flink is -set to `TimeCharacteristic.EventTime` (`StreamExecutionEnvironment.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)`). - -The Kafka consumer does not emit watermarks. To emit watermarks, the same mechanisms as described above in -"Kafka Consumers and Timestamp Extraction/Watermark Emission" using the `assignTimestampsAndWatermarks` method are applicable. - -There is no need to define a timestamp extractor when using the timestamps from Kafka. The `previousElementTimestamp` argument of -the `extractTimestamp()` method contains the timestamp carried by the Kafka message. - -A timestamp extractor for a Kafka consumer would look like this: -{% highlight java %} -public long extractTimestamp(Long element, long previousElementTimestamp) { - return previousElementTimestamp; -} -{% endhighlight %} - +++With Flink's checkpointing enabled, the `FlinkKafkaProducer010` +can provide at-least-once delivery guarantees. +Besides enabling Flink's checkpointing, you should also configure the setter +methods `setLogFailuresOnly(boolean)` and `setFlushOnCheckpoint(boolean)` appropriately. -The `FlinkKafkaProducer010` only emits the record timestamp, if `setWriteTimestampToKafka(true)` is set. + * `setLogFailuresOnly(boolean)`: by default, this is set to `false`. + Enabling this will let the producer only log failures + instead of catching and rethrowing them. This essentially accounts the record + to have succeeded, even if it was never written to the target Kafka topic. This + must be disabled for at-least-once. + * `setFlushOnCheckpoint(boolean)`: by default, this is set to `true`. + With this enabled, Flink's checkpoints will wait for any + on-the-fly records at the time of the checkpoint to be acknowledged by Kafka before + succeeding the checkpoint. This ensures that all records before the checkpoint have + been written to Kafka. This must be enabled for at-least-once. + +In conclusion, the Kafka producer by default has at-least-once guarantees for versions +0.10, with `setLogFailureOnly` set to `false` and `setFlushOnCheckpoint` set +to `true`. -{% highlight java %} -FlinkKafkaProducer010.FlinkKafkaProducer010Configuration config = FlinkKafkaProducer010.writeToKafkaWithTimestamps(streamWithTimestamps, topic, new SimpleStringSchema(), standardProps); -config.setWriteTimestampToKafka(true); -{% endhighlight %} +**Note**: By default, the number of retries is set to "0". This means that when `setLogFailuresOnly` is set to `false`, +the producer fails immediately on errors, including leader changes. The value is set to "0" by default to avoid +duplicate messages in the target topic that are caused by retries. For most production environments with frequent broker changes, +we recommend setting the number of retries to a higher value. +**Note**: There is currently no transactional producer for Kafka, so Flink can not guarantee exactly-once delivery +into a Kafka topic. ++diff --git a/docs/dev/connectors/kafka.zh.md b/docs/dev/connectors/kafka.zh.md index b2cdc54ec56c04..d2686a2b796474 100644 --- a/docs/dev/connectors/kafka.zh.md +++ b/docs/dev/connectors/kafka.zh.md @@ -84,7 +84,7 @@ Flink 提供了专门的 Kafka 连接器，向 Kafka topic 中读取或者写入 {% endhighlight %} 请注意：目前流连接器还不是二进制分发的一部分。 -[在此处]({{ site.baseurl }}/zh/dev/projectsetup/dependencies.html)可以了解到如何链接它们以实现在集群中执行。 +[在此处]({{ site.baseurl }}/zh/dev/project-configuration.html)可以了解到如何链接它们以实现在集群中执行。 ## 安装 Apache Kafka diff --git a/docs/dev/connectors/kinesis.zh.md b/docs/dev/connectors/kinesis.zh.md index 4e9009da5ee22e..9601af298d1585 100644 --- a/docs/dev/connectors/kinesis.zh.md +++ b/docs/dev/connectors/kinesis.zh.md @@ -45,7 +45,25 @@ Due to the licensing issue, the `flink-connector-kinesis{{ site.scala_version_su ## Using the Amazon Kinesis Streams Service Follow the instructions from the [Amazon Kinesis Streams Developer Guide](https://docs.aws.amazon.com/streams/latest/dev/learning-kinesis-module-one-create-stream.html) -to setup Kinesis streams. Make sure to create the appropriate IAM policy and user to read / write to the Kinesis streams. +to setup Kinesis streams. + +## Configuring Access to Kinesis with IAM +Make sure to create the appropriate IAM policy to allow reading / writing to / from the Kinesis streams. See examples [here](https://docs.aws.amazon.com/streams/latest/dev/controlling-access.html). + +Depending on your deployment you would choose a different Credentials Provider to allow access to Kinesis. +By default, the `AUTO` Credentials Provider is used. +If the access key ID and secret key are set in the configuration, the `BASIC` provider is used. + +A specific Credentials Provider can **optionally** be set by using the `AWSConfigConstants.AWS_CREDENTIALS_PROVIDER` setting. + +Supported Credential Providers are: +* `AUTO` - Using the default AWS Credentials Provider chain that searches for credentials in the following order: `ENV_VARS`, `SYS_PROPS`, `WEB_IDENTITY_TOKEN`, `PROFILE` and EC2/ECS credentials provider. +* `BASIC` - Using access key ID and secret key supplied as configuration. +* `ENV_VAR` - Using `AWS_ACCESS_KEY_ID` & `AWS_SECRET_ACCESS_KEY` environment variables. +* `SYS_PROP` - Using Java system properties aws.accessKeyId and aws.secretKey. +* `PROFILE` - Use AWS credentials profile file to create the AWS credentials. +* `ASSUME_ROLE` - Create AWS credentials by assuming a role. The credentials for assuming the role must be supplied. +* `WEB_IDENTITY_TOKEN` - Create AWS credentials by assuming a role using Web Identity Token. ## Kinesis Consumer @@ -91,8 +109,7 @@ The above is a simple example of using the consumer. Configuration for the consu instance, the configuration keys for which can be found in `AWSConfigConstants` (AWS-specific parameters) and `ConsumerConfigConstants` (Kinesis consumer parameters). The example demonstrates consuming a single Kinesis stream in the AWS region "us-east-1". The AWS credentials are supplied using the basic method in which -the AWS access key ID and secret access key are directly supplied in the configuration (other options are setting -`AWSConfigConstants.AWS_CREDENTIALS_PROVIDER` to `ENV_VAR`, `SYS_PROP`, `PROFILE`, `ASSUME_ROLE`, and `AUTO`). Also, data is being consumed +the AWS access key ID and secret access key are directly supplied in the configuration. Also, data is being consumed from the newest position in the Kinesis stream (the other option will be setting `ConsumerConfigConstants.STREAM_INITIAL_POSITION` to `TRIM_HORIZON`, which lets the consumer start reading the Kinesis stream from the earliest record possible). diff --git a/docs/dev/connectors/nifi.md b/docs/dev/connectors/nifi.md index 97fd831b6a6baa..7738a40eb432b4 100644 --- a/docs/dev/connectors/nifi.md +++ b/docs/dev/connectors/nifi.md @@ -37,7 +37,7 @@ following dependency to your project: Note that the streaming connectors are currently not part of the binary distribution. See -[here]({{site.baseurl}}/dev/projectsetup/dependencies.html) +[here]({{site.baseurl}}/dev/project-configuration.html) for information about how to package the program with the libraries for cluster execution. diff --git a/docs/dev/connectors/nifi.zh.md b/docs/dev/connectors/nifi.zh.md index 114092f9a3306b..c34c146750687b 100644 --- a/docs/dev/connectors/nifi.zh.md +++ b/docs/dev/connectors/nifi.zh.md @@ -34,7 +34,7 @@ under the License. {% endhighlight %} -注意这些连接器目前还没有包含在二进制发行版中。添加依赖、打包配置以及集群运行的相关信息请参考 [这里]({{site.baseurl}}/zh/dev/projectsetup/dependencies.html)。 +注意这些连接器目前还没有包含在二进制发行版中。添加依赖、打包配置以及集群运行的相关信息请参考 [这里]({{site.baseurl}}/zh/dev/project-configuration.html)。 #### 安装 Apache NiFi diff --git a/docs/dev/connectors/pubsub.md b/docs/dev/connectors/pubsub.md index 6055d9f67ae90e..c464f48838c5ca 100644 --- a/docs/dev/connectors/pubsub.md +++ b/docs/dev/connectors/pubsub.md @@ -41,7 +41,7 @@ following dependency to your project: Note that the streaming connectors are currently not part of the binary distribution. See -[here]({{ site.baseurl }}/dev/projectsetup/dependencies.html) +[here]({{ site.baseurl }}/dev/project-configuration.html) for information about how to package the program with the libraries for cluster execution. diff --git a/docs/dev/connectors/pubsub.zh.md b/docs/dev/connectors/pubsub.zh.md index eaf5f582c8467c..6fb9bd54f7e3b8 100644 --- a/docs/dev/connectors/pubsub.zh.md +++ b/docs/dev/connectors/pubsub.zh.md @@ -37,7 +37,7 @@ under the License. 注意：此连接器最近才加到 Flink 里，还未接受广泛测试。 -注意连接器目前还不是二进制发行版的一部分，添加依赖、打包配置以及集群运行信息请参考[这里]({{ site.baseurl }}/zh/dev/projectsetup/dependencies.html) +注意连接器目前还不是二进制发行版的一部分，添加依赖、打包配置以及集群运行信息请参考[这里]({{ site.baseurl }}/zh/dev/project-configuration.html) ## Consuming or Producing PubSubMessages diff --git a/docs/dev/connectors/rabbitmq.md b/docs/dev/connectors/rabbitmq.md index a3a56a8349ed87..819979ed9119b6 100644 --- a/docs/dev/connectors/rabbitmq.md +++ b/docs/dev/connectors/rabbitmq.md @@ -47,7 +47,7 @@ This connector provides access to data streams from [RabbitMQ](http://www.rabbit {% endhighlight %} -Note that the streaming connectors are currently not part of the binary distribution. See linking with them for cluster execution [here]({{site.baseurl}}/dev/projectsetup/dependencies.html). +Note that the streaming connectors are currently not part of the binary distribution. See linking with them for cluster execution [here]({{site.baseurl}}/dev/project-configuration.html). ### Installing RabbitMQ Follow the instructions from the [RabbitMQ download page](http://www.rabbitmq.com/download.html). After the installation the server automatically starts, and the application connecting to RabbitMQ can be launched. diff --git a/docs/dev/connectors/rabbitmq.zh.md b/docs/dev/connectors/rabbitmq.zh.md index e213d3f99b19a9..b9889f563a77e5 100644 --- a/docs/dev/connectors/rabbitmq.zh.md +++ b/docs/dev/connectors/rabbitmq.zh.md @@ -43,7 +43,7 @@ Flink 自身既没有复用 "RabbitMQ AMQP Java Client" 的代码，也没有将 {% endhighlight %} -注意连接器现在没有包含在二进制发行版中。集群执行的相关信息请参考 [这里]({{site.baseurl}}/zh/dev/projectsetup/dependencies.html). +注意连接器现在没有包含在二进制发行版中。集群执行的相关信息请参考 [这里]({{site.baseurl}}/zh/dev/project-configuration.html). ### 安装 RabbitMQ 安装 RabbitMQ 请参考 [RabbitMQ 下载页面](http://www.rabbitmq.com/download.html)。安装完成之后，服务会自动拉起，应用程序就可以尝试连接到 RabbitMQ 了。 diff --git a/docs/dev/connectors/streamfile_sink.md b/docs/dev/connectors/streamfile_sink.md index 271019a8480a6b..121a02d6822c2f 100644 --- a/docs/dev/connectors/streamfile_sink.md +++ b/docs/dev/connectors/streamfile_sink.md @@ -139,6 +139,7 @@ are finalized for further encoding purposes. Flink comes with four built-in BulkWriter factories: - [ParquetWriterFactory]({{ site.javadocs_baseurl }}/api/java/org/apache/flink/formats/parquet/ParquetWriterFactory.html) + - [AvroWriterFactory]({{ site.javadocs_baseurl }}/api/java/org/apache/flink/formats/avro/AvroWriterFactory.html) - [SequenceFileWriterFactory]({{ site.javadocs_baseurl }}/api/java/org/apache/flink/formats/sequencefile/SequenceFileWriterFactory.html) - [CompressWriterFactory]({{ site.javadocs_baseurl }}/api/java/org/apache/flink/formats/compress/CompressWriterFactory.html) - [OrcBulkWriterFactory]({{ site.javadocs_baseurl }}/api/java/org/apache/flink/orc/writer/OrcBulkWriterFactory.html) @@ -204,6 +205,106 @@ input.addSink(sink)+ +{% endhighlight %} + +A StreamingFileSink that writes data to Avro files can be created like this: + +org.apache.flink +flink-avro +{{ site.version }} +++ +For creating customized Avro writers, e.g. enabling compression, users need to create the `AvroWriterFactory` +with a custom implementation of the [AvroBuilder]({{ site.javadocs_baseurl }}/api/java/org/apache/flink/formats/avro/AvroBuilder.html) interface: + ++{% highlight java %} +import org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink; +import org.apache.flink.formats.avro.AvroWriters; +import org.apache.avro.Schema; + + +Schema schema = ...; +DataStream+stream = ...; + +final StreamingFileSink sink = StreamingFileSink + .forBulkFormat(outputBasePath, AvroWriters.forGenericRecord(schema)) + .build(); + +input.addSink(sink); + +{% endhighlight %} + +{% highlight scala %} +import org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink +import org.apache.flink.formats.avro.AvroWriters +import org.apache.avro.Schema + +val schema: Schema = ... +val input: DataStream[GenericRecord] = ... + +val sink: StreamingFileSink[GenericRecord] = StreamingFileSink + .forBulkFormat(outputBasePath, AvroWriters.forGenericRecord(schema)) + .build() + +input.addSink(sink) + +{% endhighlight %} ++++ #### ORC Format To enable the data to be bulk encoded in ORC format, Flink offers [OrcBulkWriterFactory]({{ site.javadocs_baseurl }}/api/java/org/apache/flink/formats/orc/writers/OrcBulkWriterFactory.html) @@ -231,6 +332,7 @@ class Person { {% endhighlight %}+{% highlight java %} +AvroWriterFactory factory = new AvroWriterFactory<>((AvroBuilder) out -> { + Schema schema = ReflectData.get().getSchema(Address.class); + DatumWriter datumWriter = new ReflectDatumWriter<>(schema); + + DataFileWriter dataFileWriter = new DataFileWriter<>(datumWriter); + dataFileWriter.setCodec(CodecFactory.snappyCodec()); + dataFileWriter.create(schema, out); + return dataFileWriter; +}); + +DataStream stream = ... +stream.addSink(StreamingFileSink.forBulkFormat( + outputBasePath, + factory).build()); +{% endhighlight %} +++{% highlight scala %} +val factory = new AvroWriterFactory[Address](new AvroBuilder[Address]() { + override def createWriter(out: OutputStream): DataFileWriter[Address] = { + val schema = ReflectData.get.getSchema(classOf[Address]) + val datumWriter = new ReflectDatumWriter[Address](schema) + + val dataFileWriter = new DataFileWriter[Address](datumWriter) + dataFileWriter.setCodec(CodecFactory.snappyCodec) + dataFileWriter.create(schema, out) + dataFileWriter + } +}) + +val stream: DataStream[Address] = ... +stream.addSink(StreamingFileSink.forBulkFormat( + outputBasePath, + factory).build()); +{% endhighlight %} ++重要: 批量编码模式仅支持 OnCheckpointRollingPolicy 策略, 在每次 checkpoint 的时候切割文件。 @@ -188,6 +190,304 @@ input.addSink(sink)+ +{% endhighlight %} + +将数据写入 Avro 文件的 StreamingFileSink 算子可以通过如下方式创建： + +org.apache.flink +flink-avro +{{ site.version }} +++ +如果想要创建自定义的 Avro Writer，例如启用压缩等，用户可以实现 [AvroBuilder]({{ site.javadocs_baseurl }}/api/java/org/apache/flink/formats/avro/AvroBuilder.html) +接口并自行创建一个 `AvroWriterFactory` 实例： + ++{% highlight java %} +import org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink; +import org.apache.flink.formats.avro.AvroWriters; +import org.apache.avro.Schema; + + +Schema schema = ...; +DataStream+stream = ...; + +final StreamingFileSink sink = StreamingFileSink + .forBulkFormat(outputBasePath, AvroWriters.forGenericRecord(schema)) + .build(); + +input.addSink(sink); + +{% endhighlight %} + +{% highlight scala %} +import org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink +import org.apache.flink.formats.avro.AvroWriters +import org.apache.avro.Schema + +val schema: Schema = ... +val input: DataStream[GenericRecord] = ... + +val sink: StreamingFileSink[GenericRecord] = StreamingFileSink + .forBulkFormat(outputBasePath, AvroWriters.forGenericRecord(schema)) + .build() + +input.addSink(sink) + +{% endhighlight %} ++++ +#### ORC Format + +To enable the data to be bulk encoded in ORC format, Flink offers [OrcBulkWriterFactory]({{ site.javadocs_baseurl }}/api/java/org/apache/flink/formats/orc/writers/OrcBulkWriterFactory.html) +which takes a concrete implementation of [Vectorizer]({{ site.javadocs_baseurl }}/api/java/org/apache/flink/orc/vector/Vectorizer.html). + +Like any other columnar format that encodes data in bulk fashion, Flink's `OrcBulkWriter` writes the input elements in batches. It uses +ORC's `VectorizedRowBatch` to achieve this. + +Since the input element has to be transformed to a `VectorizedRowBatch`, users have to extend the abstract `Vectorizer` +class and override the `vectorize(T element, VectorizedRowBatch batch)` method. As you can see, the method provides an +instance of `VectorizedRowBatch` to be used directly by the users so users just have to write the logic to transform the +input `element` to `ColumnVectors` and set them in the provided `VectorizedRowBatch` instance. + +For example, if the input element is of type `Person` which looks like: + ++{% highlight java %} +AvroWriterFactory factory = new AvroWriterFactory<>((AvroBuilder) out -> { + Schema schema = ReflectData.get().getSchema(Address.class); + DatumWriter datumWriter = new ReflectDatumWriter<>(schema); + + DataFileWriter dataFileWriter = new DataFileWriter<>(datumWriter); + dataFileWriter.setCodec(CodecFactory.snappyCodec()); + dataFileWriter.create(schema, out); + return dataFileWriter; +}); + +DataStream stream = ... +stream.addSink(StreamingFileSink.forBulkFormat( + outputBasePath, + factory).build()); +{% endhighlight %} +++{% highlight scala %} +val factory = new AvroWriterFactory[Address](new AvroBuilder[Address]() { + override def createWriter(out: OutputStream): DataFileWriter[Address] = { + val schema = ReflectData.get.getSchema(classOf[Address]) + val datumWriter = new ReflectDatumWriter[Address](schema) + + val dataFileWriter = new DataFileWriter[Address](datumWriter) + dataFileWriter.setCodec(CodecFactory.snappyCodec) + dataFileWriter.create(schema, out) + dataFileWriter + } +}) + +val stream: DataStream[Address] = ... +stream.addSink(StreamingFileSink.forBulkFormat( + outputBasePath, + factory).build()); +{% endhighlight %} ++++ +Then a child implementation to convert the element of type `Person` and set them in the `VectorizedRowBatch` can be like: + ++{% highlight java %} + +class Person { + private final String name; + private final int age; + ... +} + +{% endhighlight %} ++++ +To use the ORC bulk encoder in an application, users need to add the following dependency: + +{% highlight xml %} ++{% highlight java %} +import org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector; +import org.apache.hadoop.hive.ql.exec.vector.LongColumnVector; + +import java.io.IOException; +import java.io.Serializable; +import java.nio.charset.StandardCharsets; + +public class PersonVectorizer extends Vectorizer+implements Serializable { + public PersonVectorizer(String schema) { + super(schema); + } + @Override + public void vectorize(Person element, VectorizedRowBatch batch) throws IOException { + BytesColumnVector nameColVector = (BytesColumnVector) batch.cols[0]; + LongColumnVector ageColVector = (LongColumnVector) batch.cols[1]; + int row = batch.size++; + nameColVector.setVal(row, element.getName().getBytes(StandardCharsets.UTF_8)); + ageColVector.vector[row] = element.getAge(); + } +} + +{% endhighlight %} + +{% highlight scala %} +import java.nio.charset.StandardCharsets +import org.apache.hadoop.hive.ql.exec.vector.{BytesColumnVector, LongColumnVector} + +class PersonVectorizer(schema: String) extends Vectorizer[Person](schema) { + + override def vectorize(element: Person, batch: VectorizedRowBatch): Unit = { + val nameColVector = batch.cols(0).asInstanceOf[BytesColumnVector] + val ageColVector = batch.cols(1).asInstanceOf[LongColumnVector] + nameColVector.setVal(batch.size + 1, element.getName.getBytes(StandardCharsets.UTF_8)) + ageColVector.vector(batch.size + 1) = element.getAge + } + +} + +{% endhighlight %} +++ +{% endhighlight %} + +And then a `StreamingFileSink` that writes data in ORC format can be created like this: + +org.apache.flink +flink-orc{{ site.scala_version_suffix }} +{{ site.version }} +++ +OrcBulkWriterFactory can also take Hadoop `Configuration` and `Properties` so that a custom Hadoop configuration and ORC +writer properties can be provided. + ++{% highlight java %} +import org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink; +import org.apache.flink.orc.writer.OrcBulkWriterFactory; + +String schema = "struct<_col0:string,_col1:int>"; +DataStream+stream = ...; + +final OrcBulkWriterFactory writerFactory = new OrcBulkWriterFactory<>(new PersonVectorizer(schema)); + +final StreamingFileSink sink = StreamingFileSink + .forBulkFormat(outputBasePath, writerFactory) + .build(); + +input.addSink(sink); + +{% endhighlight %} + +{% highlight scala %} +import org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink +import org.apache.flink.orc.writer.OrcBulkWriterFactory + +val schema: String = "struct<_col0:string,_col1:int>" +val input: DataStream[Person] = ... +val writerFactory = new OrcBulkWriterFactory(new PersonVectorizer(schema)); + +val sink: StreamingFileSink[Person] = StreamingFileSink + .forBulkFormat(outputBasePath, writerFactory) + .build() + +input.addSink(sink) + +{% endhighlight %} ++++ +The complete list of ORC writer properties can be found [here](https://orc.apache.org/docs/hive-config.html). + +Users who want to add user metadata to the ORC files can do so by calling `addUserMetadata(...)` inside the overriding +`vectorize(...)` method. + ++{% highlight java %} +String schema = ...; +Configuration conf = ...; +Properties writerProperties = new Properties(); + +writerProps.setProperty("orc.compress", "LZ4"); +// Other ORC supported properties can also be set similarly. + +final OrcBulkWriterFactory+writerFactory = new OrcBulkWriterFactory<>( + new PersonVectorizer(schema), writerProperties, conf); + +{% endhighlight %} + +{% highlight scala %} +val schema: String = ... +val conf: Configuration = ... +val writerProperties: Properties = new Properties() + +writerProps.setProperty("orc.compress", "LZ4") +// Other ORC supported properties can also be set similarly. + +val writerFactory = new OrcBulkWriterFactory( + new PersonVectorizer(schema), writerProperties, conf) +{% endhighlight %} ++++ #### Hadoop SequenceFile 格式 在应用中使用 SequenceFile 批量编码器，你需要添加以下依赖： @@ -405,6 +705,8 @@ Hadoop 2.7 之前的版本不支持这个方法，因此 Flink 会报异常。 重要提示 3: Flink 以及 `StreamingFileSink` 不会覆盖已经提交的数据。因此如果尝试从一个包含 in-progress 文件的旧 checkpoint/savepoint 恢复， 且这些 in-progress 文件会被接下来的成功 checkpoint 提交，Flink 会因为无法找到 in-progress 文件而抛异常，从而恢复失败。 +重要提示 4: 目前 `StreamingFileSink` 只支持三种文件系统: HDFS、S3和Local。如果配置了不支持的文件系统，在执行的时候 Flink 会抛出异常。 + ### S3 特有的注意事项 重要提示 1: 对于 S3，`StreamingFileSink` 只支持基于 [Hadoop](https://hadoop.apache.org/) diff --git a/docs/dev/connectors/twitter.md b/docs/dev/connectors/twitter.md index 9ca394b69d0c1a..d2e76d8209de9e 100644 --- a/docs/dev/connectors/twitter.md +++ b/docs/dev/connectors/twitter.md @@ -36,7 +36,7 @@ To use this connector, add the following dependency to your project: {% endhighlight %} Note that the streaming connectors are currently not part of the binary distribution. -See linking with them for cluster execution [here]({{site.baseurl}}/dev/projectsetup/dependencies.html). +See linking with them for cluster execution [here]({{site.baseurl}}/dev/project-configuration.html). #### Authentication In order to connect to the Twitter stream the user has to register their program and acquire the necessary information for the authentication. The process is described below. diff --git a/docs/dev/connectors/twitter.zh.md b/docs/dev/connectors/twitter.zh.md index f4d110d44c024a..ae09a7e6332b0f 100644 --- a/docs/dev/connectors/twitter.zh.md +++ b/docs/dev/connectors/twitter.zh.md @@ -35,7 +35,7 @@ Flink Streaming 通过一个内置的 `TwitterSource` 类来创建到 tweets 流 {% endhighlight %} -注意：当前的二进制发行版还没有这些连接器。集群执行请参考[这里]({{site.baseurl}}/zh/dev/projectsetup/dependencies.html). +注意：当前的二进制发行版还没有这些连接器。集群执行请参考[这里]({{site.baseurl}}/zh/dev/project-configuration.html). #### 认证 使用 Twitter 流，用户需要先注册自己的程序，获取认证相关的必要信息。过程如下： diff --git a/docs/dev/datastream_api.zh.md b/docs/dev/datastream_api.zh.md index c95a9101dcd010..8300827cfa7dd4 100644 --- a/docs/dev/datastream_api.zh.md +++ b/docs/dev/datastream_api.zh.md @@ -32,19 +32,221 @@ example write the data to files, or to standard output (for example the command terminal). Flink programs run in a variety of contexts, standalone, or embedded in other programs. The execution can happen in a local JVM, or on clusters of many machines. -Please see [basic concepts]({{ site.baseurl }}/dev/api_concepts.html) for an introduction -to the basic concepts of the Flink API. - -In order to create your own Flink DataStream program, we encourage you to start with -[anatomy of a Flink Program]({{ site.baseurl }}/dev/api_concepts.html#anatomy-of-a-flink-program) -and gradually add your own -[stream transformations]({{ site.baseurl }}/dev/stream/operators/index.html). The remaining sections act as references for additional -operations and advanced features. +In order to create your own Flink DataStream program, we encourage you to start +with [anatomy of a Flink Program](#anatomy-of-a-flink-program) and gradually +add your own [stream transformations]({{ site.baseurl +}}/dev/stream/operators/index.html). The remaining sections act as references +for additional operations and advanced features. * This will be replaced by the TOC {:toc} +What is a DataStream? +---------------------- + +The DataStream API gets its name from the special `DataStream` class that is +used to represent a collection of data in a Flink program. You can think of +them as immutable collections of data that can contain duplicates. This data +can either be finite or unbounded, the API that you use to work on them is the +same. + +A `DataStream` is similar to a regular Java `Collection` in terms of usage but +is quite different in some key ways. They are immutable, meaning that once they +are created you cannot add or remove elements. You can also not simply inspect +the elements inside but only work on them using the `DataStream` API +operations, which are also called transformations. + +You can create an initial `DataStream` by adding a source in a Flink program. +Then you can derive new streams from this and combine them by using API methods +such as `map`, `filter`, and so on. + +Anatomy of a Flink Program +-------------------------- + +Flink programs look like regular programs that transform `DataStreams`. Each +program consists of the same basic parts: + +1. Obtain an `execution environment`, +2. Load/create the initial data, +3. Specify transformations on this data, +4. Specify where to put the results of your computations, +5. Trigger the program execution + + ++{% highlight java %} + +public class PersonVectorizer extends Vectorizer+implements Serializable { + @Override + public void vectorize(Person element, VectorizedRowBatch batch) throws IOException { + ... + String metadataKey = ...; + ByteBuffer metadataValue = ...; + this.addUserMetadata(metadataKey, metadataValue); + } +} + +{% endhighlight %} + +{% highlight scala %} + +class PersonVectorizer(schema: String) extends Vectorizer[Person](schema) { + + override def vectorize(element: Person, batch: VectorizedRowBatch): Unit = { + ... + val metadataKey: String = ... + val metadataValue: ByteBuffer = ... + addUserMetadata(metadataKey, metadataValue) + } + +} + +{% endhighlight %} ++++ +Once you specified the complete program you need to **trigger the program +execution** by calling `execute()` on the `StreamExecutionEnvironment`. +Depending on the type of the `ExecutionEnvironment` the execution will be +triggered on your local machine or submit your program for execution on a +cluster. + +The `execute()` method will wait for the job to finish and then return a +`JobExecutionResult`, this contains execution times and accumulator results. + +If you don't want to wait for the job to finish, you can trigger asynchronous +job execution by calling `executeAysnc()` on the `StreamExecutionEnvironment`. +It will return a `JobClient` with which you can communicate with the job you +just submitted. For instance, here is how to implement the semantics of +`execute()` by using `executeAsync()`. + +{% highlight java %} +final JobClient jobClient = env.executeAsync(); + +final JobExecutionResult jobExecutionResult = jobClient.getJobExecutionResult(userClassloader).get(); +{% endhighlight %} + +That last part about program execution is crucial to understanding when and how +Flink operations are executed. All Flink programs are executed lazily: When the +program's main method is executed, the data loading and transformations do not +happen directly. Rather, each operation is created and added to a dataflow +graph. The operations are actually executed when the execution is explicitly +triggered by an `execute()` call on the execution environment. Whether the +program is executed locally or on a cluster depends on the type of execution +environment + +The lazy evaluation lets you construct sophisticated programs that Flink +executes as one holistically planned unit. + +{% top %} Example Program --------------- diff --git a/docs/dev/event_time.md b/docs/dev/event_time.md index a25122f1bdee3a..3e2573496ff51f 100644 --- a/docs/dev/event_time.md +++ b/docs/dev/event_time.md @@ -34,30 +34,22 @@ For information about how to use time in Flink programs refer to [ProcessFunction]({% link dev/stream/operators/process_function.md %}). -* toc -{:toc} +A prerequisite for using *event time* processing is setting the right *time +characteristic*. That setting defines how data stream sources behave (for +example, whether they will assign timestamps), and what notion of time should +be used by window operations like `KeyedStream.timeWindow(Time.seconds(30))`. -## Setting a Time Characteristic - -The first part of a Flink DataStream program usually sets the base *time characteristic*. That setting -defines how data stream sources behave (for example, whether they will assign timestamps), and what notion of -time should be used by window operations like `KeyedStream.timeWindow(Time.seconds(30))`. - -The following example shows a Flink program that aggregates events in hourly time windows. The behavior of the -windows adapts with the time characteristic. +You can set the time characteristic using +`StreamExecutionEnvironment.setStreamTimeCharacteristic()`:+ + +We will now give an overview of each of those steps, please refer to the +respective sections for more details. Note that all core classes of the Java +DataStream API can be found in {% gh_link +/flink-streaming-java/src/main/java/org/apache/flink/streaming/api +"org.apache.flink.streaming.api" %}. + +The `StreamExecutionEnvironment` is the basis for all Flink programs. You can +obtain one using these static methods on `StreamExecutionEnvironment`: + +{% highlight java %} +getExecutionEnvironment() + +createLocalEnvironment() + +createRemoteEnvironment(String host, int port, String... jarFiles) +{% endhighlight %} + +Typically, you only need to use `getExecutionEnvironment()`, since this will do +the right thing depending on the context: if you are executing your program +inside an IDE or as a regular Java program it will create a local environment +that will execute your program on your local machine. If you created a JAR file +from your program, and invoke it through the [command line]({{ site.baseurl +}}/ops/cli.html), the Flink cluster manager will execute your main method and +`getExecutionEnvironment()` will return an execution environment for executing +your program on a cluster. + +For specifying data sources the execution environment has several methods to +read from files using various methods: you can just read them line by line, as +CSV files, or using any of the other provided sources. To just read a text file +as a sequence of lines, you can use: + +{% highlight java %} +final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); + +DataStream+text = env.readTextFile("file:///path/to/file"); +{% endhighlight %} + +This will give you a DataStream on which you can then apply transformations to create new +derived DataStreams. + +You apply transformations by calling methods on DataStream with a +transformation functions. For example, a map transformation looks like this: + +{% highlight java %} +DataStream input = ...; + +DataStream parsed = input.map(new MapFunction + +We will now give an overview of each of those steps, please refer to the +respective sections for more details. Note that all core classes of the Scala +DataStream API can be found in {% gh_link +/flink-streaming-scala/src/main/scala/org/apache/flink/streaming/api/scala +"org.apache.flink.streaming.api.scala" %}. + +The `StreamExecutionEnvironment` is the basis for all Flink programs. You can +obtain one using these static methods on `StreamExecutionEnvironment`: + +{% highlight scala %} +getExecutionEnvironment() + +createLocalEnvironment() + +createRemoteEnvironment(host: String, port: Int, jarFiles: String*) +{% endhighlight %} + +Typically, you only need to use `getExecutionEnvironment()`, since this will do +the right thing depending on the context: if you are executing your program +inside an IDE or as a regular Java program it will create a local environment +that will execute your program on your local machine. If you created a JAR file +from your program, and invoke it through the [command line]({{ site.baseurl +}}/ops/cli.html), the Flink cluster manager will execute your main method and +`getExecutionEnvironment()` will return an execution environment for executing +your program on a cluster. + +For specifying data sources the execution environment has several methods to +read from files using various methods: you can just read them line by line, as +CSV files, or using any of the other provided sources. To just read a text file +as a sequence of lines, you can use: + +{% highlight scala %} +val env = StreamExecutionEnvironment.getExecutionEnvironment() + +val text: DataStream[String] = env.readTextFile("file:///path/to/file") +{% endhighlight %} + +This will give you a DataStream on which you can then apply transformations to +create new derived DataStreams. + +You apply transformations by calling methods on DataStream with a +transformation functions. For example, a map transformation looks like this: + +{% highlight scala %} +val input: DataSet[String] = ... + +val mapped = input.map { x => x.toInt } +{% endhighlight %} + +This will create a new DataStream by converting every String in the original +collection to an Integer. + +Once you have a DataStream containing your final results, you can write it to +an outside system by creating a sink. These are just some example methods for +creating a sink: + +{% highlight scala %} +writeAsText(path: String) + +print() +{% endhighlight %} + ++-Note that in order to run this example in *event time*, the program needs to either use sources -that directly define event time for the data and emit watermarks themselves, or the program must -inject a *Timestamp Assigner & Watermark Generator* after the sources. Those functions describe how to access -the event timestamps, and what degree of out-of-orderness the event stream exhibits. - -The section below describes the general mechanism behind *timestamps* and *watermarks*. For a guide on how -to use timestamp assignment and watermark generation in the Flink DataStream API, please refer to -[Generating Timestamps / Watermarks]({{ site.baseurl }}/dev/event_timestamps_watermarks.html). - -{% top %} - -## Idling sources - -Currently, with pure event time watermarks generators, watermarks can not progress if there are no elements -to be processed. That means in case of gap in the incoming data, event time will not progress and for -example the window operator will not be triggered and thus existing windows will not be able to produce any -output data. - -To circumvent this one can use periodic watermark assigners that don't only assign based on -element timestamps. An example solution could be an assigner that switches to using current processing time -as the time basis after not observing new events for a while. - -Sources can be marked as idle using `SourceFunction.SourceContext#markAsTemporarilyIdle`. For details please refer to the Javadoc of -this method as well as `StreamStatus`. - -## Debugging Watermarks - -Please refer to the [Debugging Windows & Event Time]({{ site.baseurl }}/monitoring/debugging_event_time.html) section for debugging -watermarks at runtime. - -## How operators are processing watermarks - -As a general rule, operators are required to completely process a given watermark before forwarding it downstream. For example, -`WindowOperator` will first evaluate which windows should be fired, and only after producing all of the output triggered by -the watermark will the watermark itself be sent downstream. In other words, all elements produced due to occurrence of a watermark -will be emitted before the watermark. - -The same rule applies to `TwoInputStreamOperator`. However, in this case the current watermark of the operator is defined as -the minimum of both of its inputs. - -The details of this behavior are defined by the implementations of the `OneInputStreamOperator#processWatermark`, -`TwoInputStreamOperator#processWatermark1` and `TwoInputStreamOperator#processWatermark2` methods. +Note that in order to run this example in *event time*, the program needs to +either use sources that directly define event time for the data and emit +watermarks themselves, or the program must inject a *Timestamp Assigner & +Watermark Generator* after the sources. Those functions describe how to access +the event timestamps, and what degree of out-of-orderness the event stream +exhibits. + +## Where to go next? + +* [Generating Watermarks]({% link dev/event_timestamps_watermarks.md + %}): Shows how to write timestamp assigners and watermark generators, which + are needed for event-time aware Flink applications. +* [Builtin Watermark Generators]({% link dev/event_timestamp_extractors.md %}): + Gives an overview of the builtin watermark generators. +* [Debugging Windows & Event Time]({{ site.baseurl + }}/monitoring/debugging_event_time.html): Show how to debug problems around + watermarks and timestamps in event-time Flink applications. {% top %} diff --git a/docs/dev/event_time.zh.md b/docs/dev/event_time.zh.md index dbe964fee2fb9c..50c8560d6dbe53 100644 --- a/docs/dev/event_time.zh.md +++ b/docs/dev/event_time.zh.md @@ -89,11 +89,11 @@ stream {% highlight python %} env = StreamExecutionEnvironment.get_execution_environment() -env.set_stream_time_characteristic(TimeCharacteristic.ProcessingTime) +env.set_stream_time_characteristic(TimeCharacteristic.EventTime) # alternatively: # env.set_stream_time_characteristic(TimeCharacteristic.IngestionTime) -# env.set_stream_time_characteristic(TimeCharacteristic.EventTime) +# env.set_stream_time_characteristic(TimeCharacteristic.ProcessingTime) {% endhighlight %} diff --git a/docs/dev/event_timestamp_extractors.md b/docs/dev/event_timestamp_extractors.md index 01b3634867edf3..417658c22c5664 100644 --- a/docs/dev/event_timestamp_extractors.md +++ b/docs/dev/event_timestamp_extractors.md @@ -1,5 +1,5 @@ --- -title: "Pre-defined Timestamp Extractors / Watermark Emitters" +title: "Builtin Watermark Generators" nav-parent_id: event_time nav-pos: 2 --- @@ -25,83 +25,73 @@ under the License. * toc {:toc} -As described in [timestamps and watermark handling]({{ site.baseurl }}/dev/event_timestamps_watermarks.html), -Flink provides abstractions that allow the programmer to assign their own timestamps and emit their own watermarks. More specifically, -one can do so by implementing one of the `AssignerWithPeriodicWatermarks` and `AssignerWithPunctuatedWatermarks` interfaces, depending -on the use case. In a nutshell, the first will emit watermarks periodically, while the second does so based on some property of -the incoming records, e.g. whenever a special element is encountered in the stream. +As described in [Generating Watermarks]({{ site.baseurl +}}/dev/event_timestamps_watermarks.html), Flink provides abstractions that +allow the programmer to assign their own timestamps and emit their own +watermarks. More specifically, one can do so by implementing the +`WatermarkGenerator` interface. -In order to further ease the programming effort for such tasks, Flink comes with some pre-implemented timestamp assigners. -This section provides a list of them. Apart from their out-of-the-box functionality, their implementation can serve as an example -for custom implementations. +In order to further ease the programming effort for such tasks, Flink comes +with some pre-implemented timestamp assigners. This section provides a list of +them. Apart from their out-of-the-box functionality, their implementation can +serve as an example for custom implementations. -### **Assigners with ascending timestamps** +## Monotonously Increasing Timestamps -The simplest special case for *periodic* watermark generation is the case where timestamps seen by a given source task -occur in ascending order. In that case, the current timestamp can always act as a watermark, because no earlier timestamps will -arrive. +The simplest special case for *periodic* watermark generation is the when +timestamps seen by a given source task occur in ascending order. In that case, +the current timestamp can always act as a watermark, because no earlier +timestamps will arrive. -Note that it is only necessary that timestamps are ascending *per parallel data source task*. For example, if -in a specific setup one Kafka partition is read by one parallel data source instance, then it is only necessary that -timestamps are ascending within each Kafka partition. Flink's watermark merging mechanism will generate correct -watermarks whenever parallel streams are shuffled, unioned, connected, or merged. +Note that it is only necessary that timestamps are ascending *per parallel data +source task*. For example, if in a specific setup one Kafka partition is read +by one parallel data source instance, then it is only necessary that timestamps +are ascending within each Kafka partition. Flink's watermark merging mechanism +will generate correct watermarks whenever parallel streams are shuffled, +unioned, connected, or merged.{% highlight java %} final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); -env.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime); - -// alternatively: -// env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime); -// env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime); +env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime); -DataStreamstream = env.addSource(new FlinkKafkaConsumer010 (topic, schema, props)); +DataStream stream = env.addSource(new FlinkKafkaConsumer (topic, schema, props)); stream .keyBy( (event) -> event.getUser() ) @@ -70,13 +62,9 @@ stream {% highlight scala %} val env = StreamExecutionEnvironment.getExecutionEnvironment -env.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime) +env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime) -// alternatively: -// env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime) -// env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime) - -val stream: DataStream[MyEvent] = env.addSource(new FlinkKafkaConsumer010[MyEvent](topic, schema, props)) +val stream: DataStream[MyEvent] = env.addSource(new FlinkKafkaConsumer[MyEvent](topic, schema, props)) stream .keyBy( _.getUser ) @@ -89,56 +77,31 @@ stream {% highlight python %} env = StreamExecutionEnvironment.get_execution_environment() -env.set_stream_time_characteristic(TimeCharacteristic.ProcessingTime) +env.set_stream_time_characteristic(TimeCharacteristic.EventTime) # alternatively: # env.set_stream_time_characteristic(TimeCharacteristic.IngestionTime) -# env.set_stream_time_characteristic(TimeCharacteristic.EventTime) +# env.set_stream_time_characteristic(TimeCharacteristic.ProcessingTime) {% endhighlight %} -### **Assigners allowing a fixed amount of lateness** - -Another example of periodic watermark generation is when the watermark lags behind the maximum (event-time) timestamp -seen in the stream by a fixed amount of time. This case covers scenarios where the maximum lateness that can be encountered in a -stream is known in advance, e.g. when creating a custom source containing elements with timestamps spread within a fixed period of -time for testing. For these cases, Flink provides the `BoundedOutOfOrdernessTimestampExtractor` which takes as an argument -the `maxOutOfOrderness`, i.e. the maximum amount of time an element is allowed to be late before being ignored when computing the -final result for the given window. Lateness corresponds to the result of `t - t_w`, where `t` is the (event-time) timestamp of an -element, and `t_w` that of the previous watermark. If `lateness > 0` then the element is considered late and is, by default, ignored when computing -the result of the job for its corresponding window. See the documentation about [allowed lateness]({{ site.baseurl }}/dev/stream/operators/windows.html#allowed-lateness) -for more information about working with late elements. +## Fixed Amount of Lateness + +Another example of periodic watermark generation is when the watermark lags +behind the maximum (event-time) timestamp seen in the stream by a fixed amount +of time. This case covers scenarios where the maximum lateness that can be +encountered in a stream is known in advance, e.g. when creating a custom source +containing elements with timestamps spread within a fixed period of time for +testing. For these cases, Flink provides the `BoundedOutOfOrdernessWatermarks` +generator which takes as an argument the `maxOutOfOrderness`, i.e. the maximum +amount of time an element is allowed to be late before being ignored when +computing the final result for the given window. Lateness corresponds to the +result of `t - t_w`, where `t` is the (event-time) timestamp of an element, and +`t_w` that of the previous watermark. If `lateness > 0` then the element is +considered late and is, by default, ignored when computing the result of the +job for its corresponding window. See the documentation about [allowed +lateness]({{ site.baseurl +}}/dev/stream/operators/windows.html#allowed-lateness) for more information +about working with late elements.{% highlight java %} -DataStreamstream = ... - -DataStream withTimestampsAndWatermarks = - stream.assignTimestampsAndWatermarks(new AscendingTimestampExtractor () { - - @Override - public long extractAscendingTimestamp(MyEvent element) { - return element.getCreationTime(); - } -}); +WatermarkStrategy.forMonotonousTimestamps(); {% endhighlight %} {% highlight scala %} -val stream: DataStream[MyEvent] = ... - -val withTimestampsAndWatermarks = stream.assignAscendingTimestamps( _.getCreationTime ) +WatermarkStrategy.forMonotonousTimestamps() {% endhighlight %}diff --git a/docs/dev/event_timestamps_watermarks.md b/docs/dev/event_timestamps_watermarks.md index 28e1f2f343d1e8..3b8d7661eedb09 100644 --- a/docs/dev/event_timestamps_watermarks.md +++ b/docs/dev/event_timestamps_watermarks.md @@ -1,5 +1,5 @@ --- -title: "Generating Timestamps / Watermarks" +title: "Generating Watermarks" nav-parent_id: event_time nav-pos: 1 --- @@ -22,115 +22,106 @@ specific language governing permissions and limitations under the License. --> +In this section you will learn about the APIs that Flink provides for working +with **event time** timestamps and watermarks. For an introduction to *event +time*, *processing time*, and *ingestion time*, please refer to the +[introduction to event time]({{ site.baseurl }}/dev/event_time.html). + * toc {:toc} +## Introduction to Watermark Strategies -This section is relevant for programs running on **event time**. For an introduction to *event time*, -*processing time*, and *ingestion time*, please refer to the [introduction to event time]({{ site.baseurl }}/dev/event_time.html). - -To work with *event time*, streaming programs need to set the *time characteristic* accordingly. - -{% highlight java %} -DataStreamstream = ... - -DataStream withTimestampsAndWatermarks = - stream.assignTimestampsAndWatermarks(new BoundedOutOfOrdernessTimestampExtractor (Time.seconds(10)) { - - @Override - public long extractTimestamp(MyEvent element) { - return element.getCreationTime(); - } -}); +WatermarkStrategy.forBoundedOutOfOrderness(Duration.ofSeconds(10)); {% endhighlight %} {% highlight scala %} -val stream: DataStream[MyEvent] = ... - -val withTimestampsAndWatermarks = stream.assignTimestampsAndWatermarks(new BoundedOutOfOrdernessTimestampExtractor[MyEvent](Time.seconds(10))( _.getCreationTime )) +WatermarkStrategy + .forBoundedOutOfOrderness(Duration.ofSeconds(10)) {% endhighlight %}-- -## Assigning Timestamps - -In order to work with *event time*, Flink needs to know the events' *timestamps*, meaning each element in the -stream needs to have its event timestamp *assigned*. This is usually done by accessing/extracting the -timestamp from some field in the element. +In order to work with *event time*, Flink needs to know the events +*timestamps*, meaning each element in the stream needs to have its event +timestamp *assigned*. This is usually done by accessing/extracting the +timestamp from some field in the element by using a `TimestampAssigner`. -Timestamp assignment goes hand-in-hand with generating watermarks, which tell the system about -progress in event time. +Timestamp assignment goes hand-in-hand with generating watermarks, which tell +the system about progress in event time. You can configure this by specifying a +`WatermarkGenerator`. -There are two ways to assign timestamps and generate watermarks: +The Flink API expects a `WatermarkStrategy` that contains both a +`TimestampAssigner` and `WatermarkGenerator`. A number of common strategies +are available out of the box as static methods on `WatermarkStrategy`, but +users can also build their own strategies when required. - 1. Directly in the data stream source - 2. Via a timestamp assigner / watermark generator: in Flink, timestamp assigners also define the watermarks to be emitted +Here is the interface for completeness' sake: -Attention Both timestamps and watermarks are specified as -milliseconds since the Java epoch of 1970-01-01T00:00:00Z. - -### Source Functions with Timestamps and Watermarks +{% highlight java %} +public interface WatermarkStrategy-{% highlight java %} -final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); -env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime); -{% endhighlight %} ---{% highlight scala %} -val env = StreamExecutionEnvironment.getExecutionEnvironment -env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime) -{% endhighlight %} ---{% highlight python %} -env = StreamExecutionEnvironment.get_execution_environment() -env.set_stream_time_characteristic(TimeCharacteristic.EventTime) -{% endhighlight %} --extends TimestampAssignerSupplier , WatermarkGeneratorSupplier { -Stream sources can directly assign timestamps to the elements they produce, and they can also emit watermarks. -When this is done, no timestamp assigner is needed. -Note that if a timestamp assigner is used, any timestamps and watermarks provided by the source will be overwritten. + /** + * Instantiates a {@link TimestampAssigner} for assigning timestamps according to this + * strategy. + */ + @Override + TimestampAssigner createTimestampAssigner(TimestampAssignerSupplier.Context context); -To assign a timestamp to an element in the source directly, the source must use the `collectWithTimestamp(...)` -method on the `SourceContext`. To generate watermarks, the source must call the `emitWatermark(Watermark)` function. + /** + * Instantiates a WatermarkGenerator that generates watermarks according to this strategy. + */ + @Override + WatermarkGenerator createWatermarkGenerator(WatermarkGeneratorSupplier.Context context); +} +{% endhighlight %} -Below is a simple example of a *(non-checkpointed)* source that assigns timestamps and generates watermarks: +As mentioned, you usually don't implement this interface yourself but use the +static helper methods on `WatermarkStrategy` for common watermark strategies or +to bundle together a custom `TimestampAssigner` with a `WatermarkGenerator`. +For example, to use bounded-out-of-orderness watermarks and a lambda function as a +timestamp assigner you use this: +Specifying a `TimestampAssigner` is optional and in most cases you don't +actually want to specify one. For example, when using Kafka or Kinesis you +would get timestamps directly from the Kafka/Kinesis records. + +We will look at the `WatermarkGenerator` interface later in [Writing +WatermarkGenerators](#writing-watermarkgenerators). -### Timestamp Assigners / Watermark Generators +{% highlight java %} -@Override -public void run(SourceContextctx) throws Exception { - while (/* condition */) { - MyType next = getNext(); - ctx.collectWithTimestamp(next, next.getEventTimestamp()); - - if (next.hasWatermarkTime()) { - ctx.emitWatermark(new Watermark(next.getWatermarkTime())); - } - } -} +WatermarkStrategy + . {% highlight scala %} -override def run(ctx: SourceContext[MyType]): Unit = { - while (/* condition */) { - val next: MyType = getNext() - ctx.collectWithTimestamp(next, next.eventTimestamp) - - if (next.hasWatermarkTime) { - ctx.emitWatermark(new Watermark(next.getWatermarkTime)) - } - } -} +WatermarkStrategy + .forBoundedOutOfOrderness[(Long, String)](Duration.ofSeconds(20)) + .withTimestampAssigner(new SerializableTimestampAssigner[(Long, String)] { + override def extractTimestamp(element: (Long, String), recordTimestamp: Long): Long = element._1 + }) {% endhighlight %} + +(Using Scala Lambdas here currently doesn't work because Scala is stupid and it's hard to support this. #fus)+Attention: Both timestamps and watermarks +are specified as milliseconds since the Java epoch of 1970-01-01T00:00:00Z. +-Timestamp assigners take a stream and produce a new stream with timestamped elements and watermarks. If the -original stream had timestamps and/or watermarks already, the timestamp assigner overwrites them. +## Using Watermark Strategies -Timestamp assigners are usually specified immediately after the data source, but it is not strictly required to do so. -A common pattern, for example, is to parse (*MapFunction*) and filter (*FilterFunction*) before the timestamp assigner. -In any case, the timestamp assigner needs to be specified before the first operation on event time -(such as the first window operation). As a special case, when using Kafka as the source of a streaming job, -Flink allows the specification of a timestamp assigner / watermark emitter inside -the source (or consumer) itself. More information on how to do so can be found in the -[Kafka Connector documentation]({{ site.baseurl }}/dev/connectors/kafka.html). +There are two places in Flink applications where a `WatermarkStrategy` can be +used: 1) directly on sources and 2) after non-source operation. +The first option is preferable, because it allows sources to exploit knowledge +about shards/partitions/splits in the watermarking logic. Sources can usually +then track watermarks at a finer level and the overall watermark produced by a +source will be more accurate. Specifying a `WatermarkStrategy` directly on the +source usually means you have to use a source specific interface/ Refer to +[Watermark Strategies and the Kafka +Connector](#watermark-strategies-and-the-kafka-connector) for how this works on +a Kafka Connector and for more details about how per-partition watermarking +works there. -**NOTE:** The remainder of this section presents the main interfaces a programmer has -to implement in order to create her own timestamp extractors/watermark emitters. -To see the pre-implemented extractors that ship with Flink, please refer to the -[Pre-defined Timestamp Extractors / Watermark Emitters]({{ site.baseurl }}/dev/event_timestamp_extractors.html) page. +The second option (setting a `WatermarkStrategy` after arbitrary operations) +should only be used if you cannot set a strategy directly on the source:+Using a `WatermarkStrategy` this way takes a stream and produce a new stream +with timestamped elements and watermarks. If the original stream had timestamps +and/or watermarks already, the timestamp assigner overwrites them. + +## Dealing With Idle Sources + +If one of the input splits/partitions/shards does not carry events for a while +this means that the `WatermarkGenerator` also does not get any new information +on which to base a watermark. We call this an *idle input* or an *idle source*. +This is a problem because it can happen that some of your partitions do still +carry events. In that case, the watermark will be held back, because it is +computed as the minimum over all the different parallel watermarks. + +To deal with this, you can use a `WatermarkStrategy` that will detect idleness +and mark an input as idle. `WatermarkStrategy` provides a convenience helper +for this: + +@@ -144,7 +135,7 @@ DataStreamstream = env.readFile( DataStream withTimestampsAndWatermarks = stream .filter( event -> event.severity() == WARNING ) - .assignTimestampsAndWatermarks(new MyTimestampsAndWatermarks()); + .assignTimestampsAndWatermarks( ); withTimestampsAndWatermarks .keyBy( (event) -> event.getGroup() ) @@ -164,7 +155,7 @@ val stream: DataStream[MyEvent] = env.readFile( val withTimestampsAndWatermarks: DataStream[MyEvent] = stream .filter( _.severity == WARNING ) - .assignTimestampsAndWatermarks(new MyTimestampsAndWatermarks()) + .assignTimestampsAndWatermarks( ) withTimestampsAndWatermarks .keyBy( _.getGroup ) @@ -175,18 +166,103 @@ withTimestampsAndWatermarks ++ -#### **With Periodic Watermarks** +## Writing WatermarkGenerators -`AssignerWithPeriodicWatermarks` assigns timestamps and generates watermarks periodically (possibly depending -on the stream elements, or purely based on processing time). +A `TimestampAssigner` is a simple function that extracts a field from an event, we therefore don't need to look at them in detail. A `WatermarkGenerator`, on the other hand, is a bit more complicated to write and we will look at how you can do that in the next two sections. This is the `WatermarkGenerator` interface: -The interval (every *n* milliseconds) in which the watermark will be generated is defined via -`ExecutionConfig.setAutoWatermarkInterval(...)`. The assigner's `getCurrentWatermark()` method will be -called each time, and a new watermark will be emitted if the returned watermark is non-null and larger than the previous -watermark. +{% highlight java %} +/** + * The {@code WatermarkGenerator} generates watermarks either based on events or + * periodically (in a fixed interval). + * + *+{% highlight java %} +WatermarkStrategy + .++{% highlight scala %} +WatermarkStrategy + .forBoundedOutOfOrderness[(Long, String)](Duration.ofSeconds(20)) + .withIdleness(Duration.ofMinutes(1)) +{% endhighlight %} ++Note: This WatermarkGenerator subsumes the previous distinction between the + * {@code AssignerWithPunctuatedWatermarks} and the {@code AssignerWithPeriodicWatermarks}. + */ +@Public +public interface WatermarkGenerator
{ + + /** + * Called for every event, allows the watermark generator to examine and remember the + * event timestamps, or to emit a watermark based on the event itself. + */ + void onEvent(T event, long eventTimestamp, WatermarkOutput output); + + /** + * Called periodically, and might emit a new watermark, or not. + * + * The interval in which this method is called and Watermarks are generated + * depends on {@link ExecutionConfig#getAutoWatermarkInterval()}. + */ + void onPeriodicEmit(WatermarkOutput output); +} +{% endhighlight %} -Here we show two simple examples of timestamp assigners that use periodic watermark generation. Note that Flink ships with a `BoundedOutOfOrdernessTimestampExtractor` similar to the `BoundedOutOfOrdernessGenerator` shown below, which you can read about [here]({{ site.baseurl }}/dev/event_timestamp_extractors.html#assigners-allowing-a-fixed-amount-of-lateness). +There are two different styles of watermark generation: *periodic* and +*punctuated*. + +A periodic generator usually observes to the incoming events via `onEvent()` +and then emits a watermark when the framework calls `onPeriodicEmit()`. + +A puncutated generator will look at events in `onEvent()` and wait for special +*marker events* or *punctuations* that carry watermark information in the +stream. When it sees one of these events it emits a watermark immediately. +Usually, punctuated generators don't emit a watermark from `onPeriodicEmit()`. + +We will look at how to implement generators for each style next. + +### Writing a Periodic WatermarkGenerator + +A periodic generator observes stream events and generates +watermarks periodically (possibly depending on the stream elements, or purely +based on processing time). + +The interval (every *n* milliseconds) in which the watermark will be generated +is defined via `ExecutionConfig.setAutoWatermarkInterval(...)`. The +generators's `onPeriodicEmit()` method will be called each time, and a new +watermark will be emitted if the returned watermark is non-null and larger than +the previous watermark. + +Here we show two simple examples of watermark generators that use periodic +watermark generation. Note that Flink ships with +`BoundedOutOfOrdernessWatermarks`, which is a `WatermarkGenerator` that works +similarly to the `BoundedOutOfOrdernessGenerator` shown below. You can read +about using that [here]({{ site.baseurl +}}/dev/event_timestamp_extractors.html#assigners-allowing-a-fixed-amount-of-lateness).
-#### **With Punctuated Watermarks** +### Writing a Punctuated WatermarkGenerator -To generate watermarks whenever a certain event indicates that a new watermark might be generated, use -`AssignerWithPunctuatedWatermarks`. For this class Flink will first call the `extractTimestamp(...)` method -to assign the element a timestamp, and then immediately call the -`checkAndGetNextWatermark(...)` method on that element. +A punctuated watermark generator will observe the stream of +events and emit a watermark whenever it sees a special element that carries +watermark information. -The `checkAndGetNextWatermark(...)` method is passed the timestamp that was assigned in the `extractTimestamp(...)` -method, and can decide whether it wants to generate a watermark. Whenever the `checkAndGetNextWatermark(...)` -method returns a non-null watermark, and that watermark is larger than the latest previous watermark, that -new watermark will be emitted. +This is how you can implement a punctuated generator that emits a watermark +whenever an event indicates that it carries a certain marker:@@ -196,44 +272,42 @@ Here we show two simple examples of timestamp assigners that use periodic waterm * but only to a certain degree. The latest elements for a certain timestamp t will arrive * at most n milliseconds after the earliest elements for timestamp t. */ -public class BoundedOutOfOrdernessGenerator implements AssignerWithPeriodicWatermarks@@ -250,15 +324,13 @@ class BoundedOutOfOrdernessGenerator extends AssignerWithPeriodicWatermarks[MyEv var currentMaxTimestamp: Long = _ - override def extractTimestamp(element: MyEvent, previousElementTimestamp: Long): Long = { - val timestamp = element.getCreationTime() - currentMaxTimestamp = max(timestamp, currentMaxTimestamp) - timestamp + override def onEvent(element: MyEvent, eventTimestamp: Long): Unit = { + currentMaxTimestamp = max(eventTimestamp, currentMaxTimestamp) } - override def getCurrentWatermark(): Watermark = { - // return the watermark as current highest timestamp minus the out-of-orderness bound - new Watermark(currentMaxTimestamp - maxOutOfOrderness) + override def onPeriodicEmit(): Unit = { + // emit the watermark as current highest timestamp minus the out-of-orderness bound + output.emitWatermark(new Watermark(currentMaxTimestamp - maxOutOfOrderness - 1)); } } @@ -270,45 +342,43 @@ class TimeLagWatermarkGenerator extends AssignerWithPeriodicWatermarks[MyEvent] val maxTimeLag = 5000L // 5 seconds - override def extractTimestamp(element: MyEvent, previousElementTimestamp: Long): Long = { - element.getCreationTime + override def onEvent(element: MyEvent, eventTimestamp: Long): Unit = { + // don't need to do anything because we work on processing time } - override def getCurrentWatermark(): Watermark = { - // return the watermark as current time minus the maximum time lag - new Watermark(System.currentTimeMillis() - maxTimeLag) + override def onPeriodicEmit(): Unit = { + output.emitWatermark(new Watermark(System.currentTimeMillis() - maxTimeLag)); } } {% endhighlight %}{ +public class BoundedOutOfOrdernessGenerator implements WatermarkGenerator { private final long maxOutOfOrderness = 3500; // 3.5 seconds private long currentMaxTimestamp; @Override - public long extractTimestamp(MyEvent element, long previousElementTimestamp) { - long timestamp = element.getCreationTime(); - currentMaxTimestamp = Math.max(timestamp, currentMaxTimestamp); - return timestamp; + public void onEvent(MyEvent event, long eventTimestamp, WatermarkOutput output) { + currentMaxTimestamp = Math.max(currentMaxTimestamp, eventTimestamp); } @Override - public Watermark getCurrentWatermark() { - // return the watermark as current highest timestamp minus the out-of-orderness bound - return new Watermark(currentMaxTimestamp - maxOutOfOrderness); + public void onPeriodicEmit(WatermarkOutput output) { + // emit the watermark as current highest timestamp minus the out-of-orderness bound + output.emitWatermark(new Watermark(currentMaxTimestamp - maxOutOfOrderness - 1)); } + } /** * This generator generates watermarks that are lagging behind processing time by a fixed amount. * It assumes that elements arrive in Flink after a bounded delay. */ -public class TimeLagWatermarkGenerator implements AssignerWithPeriodicWatermarks { +public class TimeLagWatermarkGenerator implements WatermarkGenerator { - private final long maxTimeLag = 5000; // 5 seconds + private final long maxTimeLag = 5000; // 5 seconds - @Override - public long extractTimestamp(MyEvent element, long previousElementTimestamp) { - return element.getCreationTime(); - } + @Override + public void onEvent(MyEvent event, long eventTimestamp, WatermarkOutput output) { + // don't need to do anything because we work on processing time + } - @Override - public Watermark getCurrentWatermark() { - // return the watermark as current time minus the maximum time lag - return new Watermark(System.currentTimeMillis() - maxTimeLag); - } + @Override + public void onPeriodicEmit(WatermarkOutput output) { + output.emitWatermark(new Watermark(System.currentTimeMillis() - maxTimeLag)); + } } {% endhighlight %} -*Note:* It is possible to generate a watermark on every single event. However, because each watermark causes some -computation downstream, an excessive number of watermarks degrades performance. - - -## Timestamps per Kafka Partition +{% highlight java %} -public class PunctuatedAssigner implements AssignerWithPunctuatedWatermarks@@ -316,59 +386,69 @@ public class PunctuatedAssigner implements AssignerWithPunctuatedWatermarks{ +public class PunctuatedAssigner implements WatermarkGenerator { - @Override - public long extractTimestamp(MyEvent element, long previousElementTimestamp) { - return element.getCreationTime(); - } + @Override + public void onEvent(MyEvent event, long eventTimestamp, WatermarkOutput output) { + if (event.hasWatermarkMarker()) { + output.emitWatermark(new Watermark(event.getWatermarkTimestamp())); + } + } - @Override - public Watermark checkAndGetNextWatermark(MyEvent lastElement, long extractedTimestamp) { - return lastElement.hasWatermarkMarker() ? new Watermark(extractedTimestamp) : null; - } + @Override + public void onPeriodicEmit(WatermarkOutput output) { + // don't need to do anything because we emit in reaction to events above + } } {% endhighlight %} +Note: It is possible to +generate a watermark on every single event. However, because each watermark +causes some computation downstream, an excessive number of watermarks degrades +performance. +-When using [Apache Kafka](connectors/kafka.html) as a data source, each Kafka partition may have a simple event time pattern (ascending -timestamps or bounded out-of-orderness). However, when consuming streams from Kafka, multiple partitions often get consumed in parallel, -interleaving the events from the partitions and destroying the per-partition patterns (this is inherent in how Kafka's consumer clients work). +## Watermark Strategies and the Kafka Connector -In that case, you can use Flink's Kafka-partition-aware watermark generation. Using that feature, watermarks are generated inside the -Kafka consumer, per Kafka partition, and the per-partition watermarks are merged in the same way as watermarks are merged on stream shuffles. +When using [Apache Kafka](connectors/kafka.html) as a data source, each Kafka +partition may have a simple event time pattern (ascending timestamps or bounded +out-of-orderness). However, when consuming streams from Kafka, multiple +partitions often get consumed in parallel, interleaving the events from the +partitions and destroying the per-partition patterns (this is inherent in how +Kafka's consumer clients work). -For example, if event timestamps are strictly ascending per Kafka partition, generating per-partition watermarks with the -[ascending timestamps watermark generator](event_timestamp_extractors.html#assigners-with-ascending-timestamps) will result in perfect overall watermarks. +In that case, you can use Flink's Kafka-partition-aware watermark generation. +Using that feature, watermarks are generated inside the Kafka consumer, per +Kafka partition, and the per-partition watermarks are merged in the same way as +watermarks are merged on stream shuffles. -The illustrations below show how to use the per-Kafka-partition watermark generation, and how watermarks propagate through the -streaming dataflow in that case. +For example, if event timestamps are strictly ascending per Kafka partition, +generating per-partition watermarks with the [ascending timestamps watermark +generator](event_timestamp_extractors.html#assigners-with-ascending-timestamps) +will result in perfect overall watermarks. Note, that we don't provide a +`TimestampAssigner` in the example, the timestamps of the Kafka records +themselves will be used instead. +The illustrations below show how to use the per-Kafka-partition watermark +generation, and how watermarks propagate through the streaming dataflow in that +case.@@ -123,7 +120,6 @@ table = table_env.from_elements(..., ['user', 'product', 'amount']) result = table_env \ .sql_query("SELECT SUM(amount) FROM %s WHERE product LIKE '%%Rubber%%'" % table) -# SQL 更新已经注册的表 # 创建并注册 TableSink t_env.connect(FileSystem().path("/path/to/file"))) .with_format(Csv() @@ -134,16 +130,106 @@ t_env.connect(FileSystem().path("/path/to/file"))) .field("amount", DataTypes.BIGINT())) .create_temporary_table("RubberOrders") -# 在表上执行 SQL 更新操作，并把结果发出到 TableSink +# 在表上执行插入操作，并把结果发出到 TableSink table_env \ - .sql_update("INSERT INTO RubberOrders SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'") + .execute_sql("INSERT INTO RubberOrders SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'") +{% endhighlight %} + + + +{% top %} + +## 执行查询 +SELECT 语句或者 VALUES 语句可以通过 `TableEnvironment.executeSql()` 方法来执行，将选择的结果收集到本地。该方法返回 `TableResult` 对象用于包装查询的结果。和 SELECT 语句很像，一个 `Table` 对象可以通过 `Table.execute()` 方法执行从而将 `Table` 的内容收集到本地客户端。 +`TableResult.collect()` 方法返回一个可以关闭的行迭代器。除非所有的数据都被收集到本地，否则一个查询作业永远不会结束。所以我们应该通过 `CloseableIterator#close()` 方法主动地关闭作业以防止资源泄露。 +我们还可以通过 `TableResult.print()` 方法将查询结果打印到本地控制台。`TableResult` 中的结果数据只能被访问一次，因此一个 `TableResult` 实例中，`collect()` 方法和 `print()` 方法不能被同时使用。 + +对于流模式，`TableResult.collect()` 方法或者 `TableResult.print` 方法保证端到端精确一次的数据交付。这就要求开启 checkpointing。默认情况下 checkpointing 是禁止的，我们可以通过 `TableConfig` 设置 checkpointing 相关属性（请参考 checkpointing 配置）来开启 checkpointing。 +因此一条结果数据只有在其对应的 checkpointing 完成后才能在客户端被访问。 + +**注意：** 对于流模式，当前仅支持追加模式的查询语句，并且应该开启 checkpoint。因为一条结果只有在其对应的 checkpoint 完成之后才能被客户端访问到。 + +{% highlight java %} -FlinkKafkaConsumer010kafkaSource = new FlinkKafkaConsumer010<>("myTopic", schema, props); -kafkaSource.assignTimestampsAndWatermarks(new AscendingTimestampExtractor () { - - @Override - public long extractAscendingTimestamp(MyType element) { - return element.eventTimestamp(); - } -}); +FlinkKafkaConsumer kafkaSource = new FlinkKafkaConsumer<>("myTopic", schema, props); +kafkaSource.assignTimestampsAndWatermarks( + WatermarkStrategy. + .forBoundedOutOfOrderness(Duration.ofSeconds(20))); DataStream stream = env.addSource(kafkaSource); {% endhighlight %} {% highlight scala %} -val kafkaSource = new FlinkKafkaConsumer010[MyType]("myTopic", schema, props) -kafkaSource.assignTimestampsAndWatermarks(new AscendingTimestampExtractor[MyType] { - def extractAscendingTimestamp(element: MyType): Long = element.eventTimestamp -}) +val kafkaSource = new FlinkKafkaConsumer[MyType]("myTopic", schema, props) +kafkaSource.assignTimestampsAndWatermarks( + WatermarkStrategy + .forBoundedOutOfOrderness(Duration.ofSeconds(20))) val stream: DataStream[MyType] = env.addSource(kafkaSource) {% endhighlight %} @@ -377,4 +457,31 @@ val stream: DataStream[MyType] = env.addSource(kafkaSource)@@ -88,14 +87,12 @@ val table = ds.toTable(tableEnv, $"user", $"product", $"amount") val result = tableEnv.sqlQuery( s"SELECT SUM(amount) FROM $table WHERE product LIKE '%Rubber%'") -// SQL 查询一个已经注册的表 // 使用名称 "Orders" 注册一个 DataStream tableEnv.createTemporaryView("Orders", ds, $"user", $"product", $"amount") // 在表上执行 SQL 查询并得到以新表返回的结果 val result2 = tableEnv.sqlQuery( "SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'") -// 使用 SQL 更新一个已经注册的表 // 创建并注册一个 TableSink val schema = new Schema() .field("product", DataTypes.STRING()) @@ -106,8 +103,8 @@ tableEnv.connect(new FileSystem("/path/to/file")) .withSchema(schema) .createTemporaryTable("RubberOrders") -// 在表上执行 SQL 更新操作，并把结果发出到 TableSink -tableEnv.sqlUpdate( +// 在表上执行插入操作，并把结果发出到 TableSink +tableEnv.executeSql( "INSERT INTO RubberOrders SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'") {% endhighlight %}
+## How Operators Process Watermarks
+
+As a general rule, operators are required to completely process a given
+watermark before forwarding it downstream. For example, `WindowOperator` will
+first evaluate all windows that should be fired, and only after producing all of
+the output triggered by the watermark will the watermark itself be sent
+downstream. In other words, all elements produced due to occurrence of a
+watermark will be emitted before the watermark.
+
+The same rule applies to `TwoInputStreamOperator`. However, in this case the
+current watermark of the operator is defined as the minimum of both of its
+inputs.
+
+The details of this behavior are defined by the implementations of the
+`OneInputStreamOperator#processWatermark`,
+`TwoInputStreamOperator#processWatermark1` and
+`TwoInputStreamOperator#processWatermark2` methods.
+
+## The Deprecated AssignerWithPeriodicWatermarks and AssignerWithPunctuatedWatermarks
+
+Prior to introducing the current abstraction of `WatermarkStrategy`,
+`TimestampAssigner`, and `WatermarkGenerator`, Flink used
+`AssignerWithPeriodicWatermarks` and `AssignerWithPeriodicWatermarks`. You will
+still see them in the API but it is recommended to use the new interfaces
+because they offer a clearer separation of concerns and also unify periodic and
+punctuated styles of watermark generation.
+
{% top %}
diff --git a/docs/dev/index.md b/docs/dev/index.md
index 0a5616b8092cbb..b58120471b371c 100644
--- a/docs/dev/index.md
+++ b/docs/dev/index.md
@@ -5,7 +5,6 @@ nav-title: ' Applica
nav-parent_id: root
nav-pos: 5
section-break: true
-always-expand: true
---
+
+Every Flink application depends on a set of Flink libraries. At the bare minimum, the application depends
+on the Flink APIs. Many applications depend in addition on certain connector libraries (like Kafka, Cassandra, etc.).
+When running Flink applications (either in a distributed deployment, or in the IDE for testing), the Flink
+runtime library must be available as well.
+
+* This will be replaced by the TOC
+{:toc}
+
+## Flink Core and Application Dependencies
+
+As with most systems that run user-defined applications, there are two broad categories of dependencies and libraries in Flink:
+
+ - **Flink Core Dependencies**: Flink itself consists of a set of classes and dependencies that are needed to run the system, for example
+ coordination, networking, checkpoints, failover, APIs, operations (such as windowing), resource management, etc.
+ The set of all these classes and dependencies forms the core of Flink's runtime and must be present when a Flink
+ application is started.
+
+ These core classes and dependencies are packaged in the `flink-dist` jar. They are part of Flink's `lib` folder and
+ part of the basic Flink container images. Think of these dependencies as similar to Java's core library (`rt.jar`, `charsets.jar`, etc.),
+ which contains the classes like `String` and `List`.
+
+ The Flink Core Dependencies do not contain any connectors or libraries (CEP, SQL, ML, etc.) in order to avoid having an excessive
+ number of dependencies and classes in the classpath by default. In fact, we try to keep the core dependencies as slim as possible
+ to keep the default classpath small and avoid dependency clashes.
+
+ - The **User Application Dependencies** are all connectors, formats, or libraries that a specific user application needs.
+
+ The user application is typically packaged into an *application jar*, which contains the application code and the required
+ connector and library dependencies.
+
+ The user application dependencies explicitly do not include the Flink DataStream APIs and runtime dependencies,
+ because those are already part of Flink's Core Dependencies.
+
+
+## Setting up a Project: Basic Dependencies
+
+Every Flink application needs as the bare minimum the API dependencies, to develop against.
+
+When setting up a project manually, you need to add the following dependencies for the Java/Scala API
+(here presented in Maven syntax, but the same dependencies apply to other build tools (Gradle, SBT, etc.) as well.
+
+++ +**Important:** Please note that all these dependencies have their scope set to *provided*. +That means that they are needed to compile against, but that they should not be packaged into the +project's resulting application jar file - these dependencies are Flink Core Dependencies, +which are already available in any setup. + +It is highly recommended keeping the dependencies in scope *provided*. If they are not set to *provided*, +the best case is that the resulting JAR becomes excessively large, because it also contains all Flink core +dependencies. The worst case is that the Flink core dependencies that are added to the application's jar file +clash with some of your own dependency versions (which is normally avoided through inverted classloading). + +**Note on IntelliJ:** To make the applications run within IntelliJ IDEA it is necessary to tick the +`Include dependencies with "Provided" scope` box in the run configuration. +If this option is not available (possibly due to using an older IntelliJ IDEA version), then a simple workaround +is to create a test that calls the applications `main()` method. + + +## Adding Connector and Library Dependencies + +Most applications need specific connectors or libraries to run, for example a connector to Kafka, Cassandra, etc. +These connectors are not part of Flink's core dependencies and must be added as dependencies to the application. + +Below is an example adding the connector for Kafka as a dependency (Maven syntax): +{% highlight xml %} ++{% highlight xml %} +++ +{% endhighlight %} +org.apache.flink +flink-streaming-java{{ site.scala_version_suffix }} +{{site.version }} +provided ++{% highlight xml %} +++ +{% endhighlight %} +org.apache.flink +flink-streaming-scala{{ site.scala_version_suffix }} +{{site.version }} +provided ++ +{% endhighlight %} + +We recommend packaging the application code and all its required dependencies into one *jar-with-dependencies* which +we refer to as the *application jar*. The application jar can be submitted to an already running Flink cluster, +or added to a Flink application container image. + +Projects created from the [Java Project Template]({{ site.baseurl }}/dev/project-configuration.html) or +[Scala Project Template]({{ site.baseurl }}/dev/project-configuration) are configured to automatically include +the application dependencies into the application jar when running `mvn clean package`. For projects that are +not set up from those templates, we recommend adding the Maven Shade Plugin (as listed in the Appendix below) +to build the application jar with all required dependencies. + +**Important:** For Maven (and other build tools) to correctly package the dependencies into the application jar, +these application dependencies must be specified in scope *compile* (unlike the core dependencies, which +must be specified in scope *provided*). + + +## Scala Versions + +Scala versions (2.11, 2.12, etc.) are not binary compatible with one another. +For that reason, Flink for Scala 2.11 cannot be used with an application that uses +Scala 2.12. + +All Flink dependencies that (transitively) depend on Scala are suffixed with the +Scala version that they are built for, for example `flink-streaming-scala_2.11`. + +Developers that only use Java can pick any Scala version, Scala developers need to +pick the Scala version that matches their application's Scala version. + +Please refer to the [build guide]({{ site.baseurl }}/flinkDev/building.html#scala-versions) +for details on how to build Flink for a specific Scala version. + +## Hadoop Dependencies + +**General rule: It should never be necessary to add Hadoop dependencies directly to your application.** +*(The only exception being when using existing Hadoop input-/output formats with Flink's Hadoop compatibility wrappers)* + +If you want to use Flink with Hadoop, you need to have a Flink setup that includes the Hadoop dependencies, rather than +adding Hadoop as an application dependency. Please refer to the [Hadoop Setup Guide]({{ site.baseurl }}/ops/deployment/hadoop.html) +for details. + +There are two main reasons for that design: + + - Some Hadoop interaction happens in Flink's core, possibly before the user application is started, for example + setting up HDFS for checkpoints, authenticating via Hadoop's Kerberos tokens, or deployment on YARN. + + - Flink's inverted classloading approach hides many transitive dependencies from the core dependencies. That applies not only + to Flink's own core dependencies, but also to Hadoop's dependencies when present in the setup. + That way, applications can use different versions of the same dependencies without running into dependency conflicts (and + trust us, that's a big deal, because Hadoops dependency tree is huge.) + +If you need Hadoop dependencies during testing or development inside the IDE (for example for HDFS access), please configure +these dependencies similar to the scope of the dependencies to *test* or to *provided*. + +## Maven Quickstart + +#### Requirements + +The only requirements are working __Maven 3.0.4__ (or higher) and __Java 8.x__ installations. + +#### Create Project + +Use one of the following commands to __create a project__: + + +org.apache.flink +flink-connector-kafka{{ site.scala_version_suffix }} +{{site.version }} +++ +We recommend you __import this project into your IDE__ to develop and +test it. IntelliJ IDEA supports Maven projects out of the box. +If you use Eclipse, the [m2e plugin](http://www.eclipse.org/m2e/) +allows to [import Maven projects](http://books.sonatype.com/m2eclipse-book/reference/creating-sect-importing-projects.html#fig-creating-import). +Some Eclipse bundles include that plugin by default, others require you +to install it manually. + +*Please note*: The default JVM heapsize for Java may be too +small for Flink. You have to manually increase it. +In Eclipse, choose `Run Configurations -> Arguments` and write into the `VM Arguments` box: `-Xmx800m`. +In IntelliJ IDEA recommended way to change JVM options is from the `Help | Edit Custom VM Options` menu. See [this article](https://intellij-support.jetbrains.com/hc/en-us/articles/206544869-Configuring-JVM-options-and-platform-properties) for details. + +#### Build Project + +If you want to __build/package your project__, go to your project directory and +run the '`mvn clean package`' command. +You will __find a JAR file__ that contains your application, plus connectors and libraries +that you may have added as dependencies to the application: `target/+{% highlight bash %} +$ mvn archetype:generate \ + -DarchetypeGroupId=org.apache.flink \ + -DarchetypeArtifactId=flink-quickstart-java \{% unless site.is_stable %} + -DarchetypeCatalog=https://repository.apache.org/content/repositories/snapshots/ \{% endunless %} + -DarchetypeVersion={{site.version}} +{% endhighlight %} + This allows you to name your newly created project. It will interactively ask you for the groupId, artifactId, and package name. +++{% highlight bash %} +{% if site.is_stable %} +$ curl https://flink.apache.org/q/quickstart.sh | bash -s {{site.version}} +{% else %} +$ curl https://flink.apache.org/q/quickstart-SNAPSHOT.sh | bash -s {{site.version}} +{% endif %} +{% endhighlight %} + ++ {% unless site.is_stable %} ++ Note: For Maven 3.0 or higher, it is no longer possible to specify the repository (-DarchetypeCatalog) via the command line. For details about this change, please refer to Maven official document + If you wish to use the snapshot repository, you need to add a repository entry to your settings.xml. For example: +{% highlight bash %} +
+ {% endunless %} ++ +{% endhighlight %} ++ +apache ++ ++ +apache ++ ++ +apache-snapshots +https://repository.apache.org/content/repositories/snapshots/ +- .jar`. + +__Note:__ If you use a different class than *StreamingJob* as the application's main class / entry point, +we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, Flink +can run the application from the JAR file without additionally specifying the main class. + +## Gradle + +#### Requirements + +The only requirements are working __Gradle 3.x__ (or higher) and __Java 8.x__ installations. + +#### Create Project + +Use one of the following commands to __create a project__: + + + ++ +We recommend you __import this project into your IDE__ to develop and +test it. IntelliJ IDEA supports Gradle projects after installing the `Gradle` plugin. +Eclipse does so via the [Eclipse Buildship](https://projects.eclipse.org/projects/tools.buildship) plugin +(make sure to specify a Gradle version >= 3.0 in the last step of the import wizard; the `shadow` plugin requires it). +You may also use [Gradle's IDE integration](https://docs.gradle.org/current/userguide/userguide.html#ide-integration) +to create project files from Gradle. + + +*Please note*: The default JVM heapsize for Java may be too +small for Flink. You have to manually increase it. +In Eclipse, choose `Run Configurations -> Arguments` and write into the `VM Arguments` box: `-Xmx800m`. +In IntelliJ IDEA recommended way to change JVM options is from the `Help | Edit Custom VM Options` menu. See [this article](https://intellij-support.jetbrains.com/hc/en-us/articles/206544869-Configuring-JVM-options-and-platform-properties) for details. + +#### Build Project + +If you want to __build/package your project__, go to your project directory and +run the '`gradle clean shadowJar`' command. +You will __find a JAR file__ that contains your application, plus connectors and libraries +that you may have added as dependencies to the application: `build/libs/+ + ++ ++ +++{% highlight gradle %} +buildscript { + repositories { + jcenter() // this applies only to the Gradle 'Shadow' plugin + } + dependencies { + classpath 'com.github.jengelman.gradle.plugins:shadow:2.0.4' + } +} + +plugins { + id 'java' + id 'application' + // shadow plugin to produce fat JARs + id 'com.github.johnrengelman.shadow' version '2.0.4' +} + + +// artifact properties +group = 'org.myorg.quickstart' +version = '0.1-SNAPSHOT' +mainClassName = 'org.myorg.quickstart.StreamingJob' +description = """Flink Quickstart Job""" + +ext { + javaVersion = '1.8' + flinkVersion = '{{ site.version }}' + scalaBinaryVersion = '{{ site.scala_version }}' + slf4jVersion = '1.7.15' + log4jVersion = '2.12.1' +} + + +sourceCompatibility = javaVersion +targetCompatibility = javaVersion +tasks.withType(JavaCompile) { + options.encoding = 'UTF-8' +} + +applicationDefaultJvmArgs = ["-Dlog4j.configurationFile=log4j2.properties"] + +task wrapper(type: Wrapper) { + gradleVersion = '3.1' +} + +// declare where to find the dependencies of your project +repositories { + mavenCentral() + maven { url "https://repository.apache.org/content/repositories/snapshots/" } +} + +// NOTE: We cannot use "compileOnly" or "shadow" configurations since then we could not run code +// in the IDE or with "gradle run". We also cannot exclude transitive dependencies from the +// shadowJar yet (see https://github.com/johnrengelman/shadow/issues/159). +// -> Explicitly define the // libraries we want to be included in the "flinkShadowJar" configuration! +configurations { + flinkShadowJar // dependencies which go into the shadowJar + + // always exclude these (also from transitive dependencies) since they are provided by Flink + flinkShadowJar.exclude group: 'org.apache.flink', module: 'force-shading' + flinkShadowJar.exclude group: 'com.google.code.findbugs', module: 'jsr305' + flinkShadowJar.exclude group: 'org.slf4j' + flinkShadowJar.exclude group: 'org.apache.logging.log4j' +} + +// declare the dependencies for your production and test code +dependencies { + // -------------------------------------------------------------- + // Compile-time dependencies that should NOT be part of the + // shadow jar and are provided in the lib folder of Flink + // -------------------------------------------------------------- + compile "org.apache.flink:flink-streaming-java_${scalaBinaryVersion}:${flinkVersion}" + + // -------------------------------------------------------------- + // Dependencies that should be part of the shadow jar, e.g. + // connectors. These must be in the flinkShadowJar configuration! + // -------------------------------------------------------------- + //flinkShadowJar "org.apache.flink:flink-connector-kafka-0.11_${scalaBinaryVersion}:${flinkVersion}" + + compile "org.apache.logging.log4j:log4j-api:${log4jVersion}" + compile "org.apache.logging.log4j:log4j-core:${log4jVersion}" + compile "org.apache.logging.log4j:log4j-slf4j-impl:${log4jVersion}" + compile "org.slf4j:slf4j-log4j12:${slf4jVersion}" + + // Add test dependencies here. + // testCompile "junit:junit:4.12" +} + +// make compileOnly dependencies available for tests: +sourceSets { + main.compileClasspath += configurations.flinkShadowJar + main.runtimeClasspath += configurations.flinkShadowJar + + test.compileClasspath += configurations.flinkShadowJar + test.runtimeClasspath += configurations.flinkShadowJar + + javadoc.classpath += configurations.flinkShadowJar +} + +run.classpath = sourceSets.main.runtimeClasspath + +jar { + manifest { + attributes 'Built-By': System.getProperty('user.name'), + 'Build-Jdk': System.getProperty('java.version') + } +} + +shadowJar { + configurations = [project.configurations.flinkShadowJar] +} +{% endhighlight %} +++{% highlight gradle %} +rootProject.name = 'quickstart' +{% endhighlight %} +++{% highlight bash %} +bash -c "$(curl https://flink.apache.org/q/gradle-quickstart.sh)" -- {{site.version}} {{site.scala_version}} +{% endhighlight %} + This allows you to name your newly created project. It will interactively ask + you for the project name, organization (also used for the package name), project version, + Scala and Flink version. ++- -all.jar`. + +__Note:__ If you use a different class than *StreamingJob* as the application's main class / entry point, +we recommend you change the `mainClassName` setting in the `build.gradle` file accordingly. That way, Flink +can run the application from the JAR file without additionally specifying the main class. + +## SBT + +#### Create Project + +You can scaffold a new project via either of the following two methods: + + + + ++ +#### Build Project + +In order to build your project you simply have to issue the `sbt clean assembly` command. +This will create the fat-jar __your-project-name-assembly-0.1-SNAPSHOT.jar__ in the directory __target/scala_your-major-scala-version/__. + +#### Run Project + +In order to run your project you have to issue the `sbt run` command. + +Per default, this will run your job in the same JVM as `sbt` is running. +In order to run your job in a distinct JVM, add the following line to `build.sbt` + +{% highlight scala %} +fork in run := true +{% endhighlight %} + + +#### IntelliJ + +We recommend using [IntelliJ](https://www.jetbrains.com/idea/) for your Flink job development. +In order to get started, you have to import your newly created project into IntelliJ. +You can do this via `File -> New -> Project from Existing Sources...` and then choosing your project's directory. +IntelliJ will then automatically detect the `build.sbt` file and set everything up. + +In order to run your Flink job, it is recommended to choose the `mainRunner` module as the classpath of your __Run/Debug Configuration__. +This will ensure, that all dependencies which are set to _provided_ will be available upon execution. +You can configure the __Run/Debug Configurations__ via `Run -> Edit Configurations...` and then choose `mainRunner` from the _Use classpath of module_ dropbox. + +#### Eclipse + +In order to import the newly created project into [Eclipse](https://eclipse.org/), you first have to create Eclipse project files for it. +These project files can be created via the [sbteclipse](https://github.com/typesafehub/sbteclipse) plugin. +Add the following line to your `PROJECT_DIR/project/plugins.sbt` file: + +{% highlight bash %} +addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "4.0.0") +{% endhighlight %} + +In `sbt` use the following command to create the Eclipse project files + +{% highlight bash %} +> eclipse +{% endhighlight %} + +Now you can import the project into Eclipse via `File -> Import... -> Existing Projects into Workspace` and then select the project directory. + + +## Appendix: Template for building a Jar with Dependencies + +To build an application JAR that contains all dependencies required for declared connectors and libraries, +you can use the following shade plugin definition: + +{% highlight xml %} ++{% highlight bash %} +$ sbt new tillrohrmann/flink-project.g8 +{% endhighlight %} + This will prompt you for a couple of parameters (project name, flink version...) and then create a Flink project from the flink-project template. + You need sbt >= 0.13.13 to execute this command. You can follow this installation guide to obtain it if necessary. +++{% highlight bash %} +$ bash <(curl https://flink.apache.org/q/sbt-quickstart.sh) +{% endhighlight %} + This will create a Flink project in the specified project directory. +++ +{% endhighlight %} + +{% top %} diff --git a/docs/dev/project-configuration.zh.md b/docs/dev/project-configuration.zh.md new file mode 100644 index 00000000000000..8d5c58abb2363a --- /dev/null +++ b/docs/dev/project-configuration.zh.md @@ -0,0 +1,559 @@ +--- +title: "Project Configuration" +nav-parent_id: streaming +nav-pos: 301 +--- + + +Every Flink application depends on a set of Flink libraries. At the bare minimum, the application depends +on the Flink APIs. Many applications depend in addition on certain connector libraries (like Kafka, Cassandra, etc.). +When running Flink applications (either in a distributed deployment, or in the IDE for testing), the Flink +runtime library must be available as well. + +* This will be replaced by the TOC +{:toc} + +## Flink Core and Application Dependencies + +As with most systems that run user-defined applications, there are two broad categories of dependencies and libraries in Flink: + + - **Flink Core Dependencies**: Flink itself consists of a set of classes and dependencies that are needed to run the system, for example + coordination, networking, checkpoints, failover, APIs, operations (such as windowing), resource management, etc. + The set of all these classes and dependencies forms the core of Flink's runtime and must be present when a Flink + application is started. + + These core classes and dependencies are packaged in the `flink-dist` jar. They are part of Flink's `lib` folder and + part of the basic Flink container images. Think of these dependencies as similar to Java's core library (`rt.jar`, `charsets.jar`, etc.), + which contains the classes like `String` and `List`. + + The Flink Core Dependencies do not contain any connectors or libraries (CEP, SQL, ML, etc.) in order to avoid having an excessive + number of dependencies and classes in the classpath by default. In fact, we try to keep the core dependencies as slim as possible + to keep the default classpath small and avoid dependency clashes. + + - The **User Application Dependencies** are all connectors, formats, or libraries that a specific user application needs. + + The user application is typically packaged into an *application jar*, which contains the application code and the required + connector and library dependencies. + + The user application dependencies explicitly do not include the Flink DataStream APIs and runtime dependencies, + because those are already part of Flink's Core Dependencies. + + +## Setting up a Project: Basic Dependencies + +Every Flink application needs as the bare minimum the API dependencies, to develop against. + +When setting up a project manually, you need to add the following dependencies for the Java/Scala API +(here presented in Maven syntax, but the same dependencies apply to other build tools (Gradle, SBT, etc.) as well. + ++ ++ +org.apache.maven.plugins +maven-shade-plugin +3.1.1 ++ ++ +package ++ +shade ++ ++ ++ +com.google.code.findbugs:jsr305 +org.slf4j:* +log4j:* ++ ++ + +*:* ++ +META-INF/*.SF +META-INF/*.DSA +META-INF/*.RSA ++ ++ +my.programs.main.clazz +++ +**Important:** Please note that all these dependencies have their scope set to *provided*. +That means that they are needed to compile against, but that they should not be packaged into the +project's resulting application jar file - these dependencies are Flink Core Dependencies, +which are already available in any setup. + +It is highly recommended keeping the dependencies in scope *provided*. If they are not set to *provided*, +the best case is that the resulting JAR becomes excessively large, because it also contains all Flink core +dependencies. The worst case is that the Flink core dependencies that are added to the application's jar file +clash with some of your own dependency versions (which is normally avoided through inverted classloading). + +**Note on IntelliJ:** To make the applications run within IntelliJ IDEA it is necessary to tick the +`Include dependencies with "Provided" scope` box in the run configuration. +If this option is not available (possibly due to using an older IntelliJ IDEA version), then a simple workaround +is to create a test that calls the applications `main()` method. + + +## Adding Connector and Library Dependencies + +Most applications need specific connectors or libraries to run, for example a connector to Kafka, Cassandra, etc. +These connectors are not part of Flink's core dependencies and must be added as dependencies to the application. + +Below is an example adding the connector for Kafka as a dependency (Maven syntax): +{% highlight xml %} ++{% highlight xml %} +++ +{% endhighlight %} +org.apache.flink +flink-streaming-java{{ site.scala_version_suffix }} +{{site.version }} +provided ++{% highlight xml %} +++ +{% endhighlight %} +org.apache.flink +flink-streaming-scala{{ site.scala_version_suffix }} +{{site.version }} +provided ++ +{% endhighlight %} + +We recommend packaging the application code and all its required dependencies into one *jar-with-dependencies* which +we refer to as the *application jar*. The application jar can be submitted to an already running Flink cluster, +or added to a Flink application container image. + +Projects created from the [Java Project Template]({{ site.baseurl }}/dev/project-configuration.html) or +[Scala Project Template]({{ site.baseurl }}/dev/project-configuration) are configured to automatically include +the application dependencies into the application jar when running `mvn clean package`. For projects that are +not set up from those templates, we recommend adding the Maven Shade Plugin (as listed in the Appendix below) +to build the application jar with all required dependencies. + +**Important:** For Maven (and other build tools) to correctly package the dependencies into the application jar, +these application dependencies must be specified in scope *compile* (unlike the core dependencies, which +must be specified in scope *provided*). + + +## Scala Versions + +Scala versions (2.11, 2.12, etc.) are not binary compatible with one another. +For that reason, Flink for Scala 2.11 cannot be used with an application that uses +Scala 2.12. + +All Flink dependencies that (transitively) depend on Scala are suffixed with the +Scala version that they are built for, for example `flink-streaming-scala_2.11`. + +Developers that only use Java can pick any Scala version, Scala developers need to +pick the Scala version that matches their application's Scala version. + +Please refer to the [build guide]({{ site.baseurl }}/flinkDev/building.html#scala-versions) +for details on how to build Flink for a specific Scala version. + +## Hadoop Dependencies + +**General rule: It should never be necessary to add Hadoop dependencies directly to your application.** +*(The only exception being when using existing Hadoop input-/output formats with Flink's Hadoop compatibility wrappers)* + +If you want to use Flink with Hadoop, you need to have a Flink setup that includes the Hadoop dependencies, rather than +adding Hadoop as an application dependency. Please refer to the [Hadoop Setup Guide]({{ site.baseurl }}/ops/deployment/hadoop.html) +for details. + +There are two main reasons for that design: + + - Some Hadoop interaction happens in Flink's core, possibly before the user application is started, for example + setting up HDFS for checkpoints, authenticating via Hadoop's Kerberos tokens, or deployment on YARN. + + - Flink's inverted classloading approach hides many transitive dependencies from the core dependencies. That applies not only + to Flink's own core dependencies, but also to Hadoop's dependencies when present in the setup. + That way, applications can use different versions of the same dependencies without running into dependency conflicts (and + trust us, that's a big deal, because Hadoops dependency tree is huge.) + +If you need Hadoop dependencies during testing or development inside the IDE (for example for HDFS access), please configure +these dependencies similar to the scope of the dependencies to *test* or to *provided*. + +## Maven Quickstart + +#### Requirements + +The only requirements are working __Maven 3.0.4__ (or higher) and __Java 8.x__ installations. + +#### Create Project + +Use one of the following commands to __create a project__: + + +org.apache.flink +flink-connector-kafka{{ site.scala_version_suffix }} +{{site.version }} +++ +We recommend you __import this project into your IDE__ to develop and +test it. IntelliJ IDEA supports Maven projects out of the box. +If you use Eclipse, the [m2e plugin](http://www.eclipse.org/m2e/) +allows to [import Maven projects](http://books.sonatype.com/m2eclipse-book/reference/creating-sect-importing-projects.html#fig-creating-import). +Some Eclipse bundles include that plugin by default, others require you +to install it manually. + +*Please note*: The default JVM heapsize for Java may be too +small for Flink. You have to manually increase it. +In Eclipse, choose `Run Configurations -> Arguments` and write into the `VM Arguments` box: `-Xmx800m`. +In IntelliJ IDEA recommended way to change JVM options is from the `Help | Edit Custom VM Options` menu. See [this article](https://intellij-support.jetbrains.com/hc/en-us/articles/206544869-Configuring-JVM-options-and-platform-properties) for details. + +#### Build Project + +If you want to __build/package your project__, go to your project directory and +run the '`mvn clean package`' command. +You will __find a JAR file__ that contains your application, plus connectors and libraries +that you may have added as dependencies to the application: `target/+{% highlight bash %} +$ mvn archetype:generate \ + -DarchetypeGroupId=org.apache.flink \ + -DarchetypeArtifactId=flink-quickstart-java \{% unless site.is_stable %} + -DarchetypeCatalog=https://repository.apache.org/content/repositories/snapshots/ \{% endunless %} + -DarchetypeVersion={{site.version}} +{% endhighlight %} + This allows you to name your newly created project. It will interactively ask you for the groupId, artifactId, and package name. +++{% highlight bash %} +{% if site.is_stable %} +$ curl https://flink.apache.org/q/quickstart.sh | bash -s {{site.version}} +{% else %} +$ curl https://flink.apache.org/q/quickstart-SNAPSHOT.sh | bash -s {{site.version}} +{% endif %} +{% endhighlight %} + ++ {% unless site.is_stable %} ++ Note: For Maven 3.0 or higher, it is no longer possible to specify the repository (-DarchetypeCatalog) via the command line. For details about this change, please refer to Maven official document + If you wish to use the snapshot repository, you need to add a repository entry to your settings.xml. For example: +{% highlight bash %} +
+ {% endunless %} ++ +{% endhighlight %} ++ +apache ++ ++ +apache ++ ++ +apache-snapshots +https://repository.apache.org/content/repositories/snapshots/ +- .jar`. + +__Note:__ If you use a different class than *StreamingJob* as the application's main class / entry point, +we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, Flink +can run the application from the JAR file without additionally specifying the main class. + +## Gradle + +#### Requirements + +The only requirements are working __Gradle 3.x__ (or higher) and __Java 8.x__ installations. + +#### Create Project + +Use one of the following commands to __create a project__: + + + ++ +We recommend you __import this project into your IDE__ to develop and +test it. IntelliJ IDEA supports Gradle projects after installing the `Gradle` plugin. +Eclipse does so via the [Eclipse Buildship](https://projects.eclipse.org/projects/tools.buildship) plugin +(make sure to specify a Gradle version >= 3.0 in the last step of the import wizard; the `shadow` plugin requires it). +You may also use [Gradle's IDE integration](https://docs.gradle.org/current/userguide/userguide.html#ide-integration) +to create project files from Gradle. + + +*Please note*: The default JVM heapsize for Java may be too +small for Flink. You have to manually increase it. +In Eclipse, choose `Run Configurations -> Arguments` and write into the `VM Arguments` box: `-Xmx800m`. +In IntelliJ IDEA recommended way to change JVM options is from the `Help | Edit Custom VM Options` menu. See [this article](https://intellij-support.jetbrains.com/hc/en-us/articles/206544869-Configuring-JVM-options-and-platform-properties) for details. + +#### Build Project + +If you want to __build/package your project__, go to your project directory and +run the '`gradle clean shadowJar`' command. +You will __find a JAR file__ that contains your application, plus connectors and libraries +that you may have added as dependencies to the application: `build/libs/+ + ++ ++ +++{% highlight gradle %} +buildscript { + repositories { + jcenter() // this applies only to the Gradle 'Shadow' plugin + } + dependencies { + classpath 'com.github.jengelman.gradle.plugins:shadow:2.0.4' + } +} + +plugins { + id 'java' + id 'application' + // shadow plugin to produce fat JARs + id 'com.github.johnrengelman.shadow' version '2.0.4' +} + + +// artifact properties +group = 'org.myorg.quickstart' +version = '0.1-SNAPSHOT' +mainClassName = 'org.myorg.quickstart.StreamingJob' +description = """Flink Quickstart Job""" + +ext { + javaVersion = '1.8' + flinkVersion = '{{ site.version }}' + scalaBinaryVersion = '{{ site.scala_version }}' + slf4jVersion = '1.7.15' + log4jVersion = '2.12.1' +} + + +sourceCompatibility = javaVersion +targetCompatibility = javaVersion +tasks.withType(JavaCompile) { + options.encoding = 'UTF-8' +} + +applicationDefaultJvmArgs = ["-Dlog4j.configurationFile=log4j2.properties"] + +task wrapper(type: Wrapper) { + gradleVersion = '3.1' +} + +// declare where to find the dependencies of your project +repositories { + mavenCentral() + maven { url "https://repository.apache.org/content/repositories/snapshots/" } +} + +// NOTE: We cannot use "compileOnly" or "shadow" configurations since then we could not run code +// in the IDE or with "gradle run". We also cannot exclude transitive dependencies from the +// shadowJar yet (see https://github.com/johnrengelman/shadow/issues/159). +// -> Explicitly define the // libraries we want to be included in the "flinkShadowJar" configuration! +configurations { + flinkShadowJar // dependencies which go into the shadowJar + + // always exclude these (also from transitive dependencies) since they are provided by Flink + flinkShadowJar.exclude group: 'org.apache.flink', module: 'force-shading' + flinkShadowJar.exclude group: 'com.google.code.findbugs', module: 'jsr305' + flinkShadowJar.exclude group: 'org.slf4j' + flinkShadowJar.exclude group: 'org.apache.logging.log4j' +} + +// declare the dependencies for your production and test code +dependencies { + // -------------------------------------------------------------- + // Compile-time dependencies that should NOT be part of the + // shadow jar and are provided in the lib folder of Flink + // -------------------------------------------------------------- + compile "org.apache.flink:flink-streaming-java_${scalaBinaryVersion}:${flinkVersion}" + + // -------------------------------------------------------------- + // Dependencies that should be part of the shadow jar, e.g. + // connectors. These must be in the flinkShadowJar configuration! + // -------------------------------------------------------------- + //flinkShadowJar "org.apache.flink:flink-connector-kafka-0.11_${scalaBinaryVersion}:${flinkVersion}" + + compile "org.apache.logging.log4j:log4j-api:${log4jVersion}" + compile "org.apache.logging.log4j:log4j-core:${log4jVersion}" + compile "org.apache.logging.log4j:log4j-slf4j-impl:${log4jVersion}" + compile "org.slf4j:slf4j-log4j12:${slf4jVersion}" + + // Add test dependencies here. + // testCompile "junit:junit:4.12" +} + +// make compileOnly dependencies available for tests: +sourceSets { + main.compileClasspath += configurations.flinkShadowJar + main.runtimeClasspath += configurations.flinkShadowJar + + test.compileClasspath += configurations.flinkShadowJar + test.runtimeClasspath += configurations.flinkShadowJar + + javadoc.classpath += configurations.flinkShadowJar +} + +run.classpath = sourceSets.main.runtimeClasspath + +jar { + manifest { + attributes 'Built-By': System.getProperty('user.name'), + 'Build-Jdk': System.getProperty('java.version') + } +} + +shadowJar { + configurations = [project.configurations.flinkShadowJar] +} +{% endhighlight %} +++{% highlight gradle %} +rootProject.name = 'quickstart' +{% endhighlight %} +++{% highlight bash %} +bash -c "$(curl https://flink.apache.org/q/gradle-quickstart.sh)" -- {{site.version}} {{site.scala_version}} +{% endhighlight %} + This allows you to name your newly created project. It will interactively ask + you for the project name, organization (also used for the package name), project version, + Scala and Flink version. ++- -all.jar`. + +__Note:__ If you use a different class than *StreamingJob* as the application's main class / entry point, +we recommend you change the `mainClassName` setting in the `build.gradle` file accordingly. That way, Flink +can run the application from the JAR file without additionally specifying the main class. + +## SBT + +#### Create Project + +You can scaffold a new project via either of the following two methods: + + + + ++ +#### Build Project + +In order to build your project you simply have to issue the `sbt clean assembly` command. +This will create the fat-jar __your-project-name-assembly-0.1-SNAPSHOT.jar__ in the directory __target/scala_your-major-scala-version/__. + +#### Run Project + +In order to run your project you have to issue the `sbt run` command. + +Per default, this will run your job in the same JVM as `sbt` is running. +In order to run your job in a distinct JVM, add the following line to `build.sbt` + +{% highlight scala %} +fork in run := true +{% endhighlight %} + + +#### IntelliJ + +We recommend using [IntelliJ](https://www.jetbrains.com/idea/) for your Flink job development. +In order to get started, you have to import your newly created project into IntelliJ. +You can do this via `File -> New -> Project from Existing Sources...` and then choosing your project's directory. +IntelliJ will then automatically detect the `build.sbt` file and set everything up. + +In order to run your Flink job, it is recommended to choose the `mainRunner` module as the classpath of your __Run/Debug Configuration__. +This will ensure, that all dependencies which are set to _provided_ will be available upon execution. +You can configure the __Run/Debug Configurations__ via `Run -> Edit Configurations...` and then choose `mainRunner` from the _Use classpath of module_ dropbox. + +#### Eclipse + +In order to import the newly created project into [Eclipse](https://eclipse.org/), you first have to create Eclipse project files for it. +These project files can be created via the [sbteclipse](https://github.com/typesafehub/sbteclipse) plugin. +Add the following line to your `PROJECT_DIR/project/plugins.sbt` file: + +{% highlight bash %} +addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "4.0.0") +{% endhighlight %} + +In `sbt` use the following command to create the Eclipse project files + +{% highlight bash %} +> eclipse +{% endhighlight %} + +Now you can import the project into Eclipse via `File -> Import... -> Existing Projects into Workspace` and then select the project directory. + + +## Appendix: Template for building a Jar with Dependencies + +To build an application JAR that contains all dependencies required for declared connectors and libraries, +you can use the following shade plugin definition: + +{% highlight xml %} ++{% highlight bash %} +$ sbt new tillrohrmann/flink-project.g8 +{% endhighlight %} + This will prompt you for a couple of parameters (project name, flink version...) and then create a Flink project from the flink-project template. + You need sbt >= 0.13.13 to execute this command. You can follow this installation guide to obtain it if necessary. +++{% highlight bash %} +$ bash <(curl https://flink.apache.org/q/sbt-quickstart.sh) +{% endhighlight %} + This will create a Flink project in the specified project directory. +++ +{% endhighlight %} + +{% top %} diff --git a/docs/dev/stream/operators/index.zh.md b/docs/dev/stream/operators/index.zh.md index 2f543352342280..29d61b5a8b0261 100644 --- a/docs/dev/stream/operators/index.zh.md +++ b/docs/dev/stream/operators/index.zh.md @@ -100,7 +100,7 @@ dataStream.filter(new FilterFunction+ ++ +org.apache.maven.plugins +maven-shade-plugin +3.1.1 ++ ++ +package ++ +shade ++ ++ ++ +com.google.code.findbugs:jsr305 +org.slf4j:* +log4j:* ++ ++ + +*:* ++ +META-INF/*.SF +META-INF/*.DSA +META-INF/*.RSA ++ ++ +my.programs.main.clazz +() { KeyBy
DataStream → KeyedStream- Logically partitions a stream into disjoint partitions. All records with the same key are assigned to the same partition. Internally, keyBy() is implemented with hash partitioning. There are different ways to specify keys.
+Logically partitions a stream into disjoint partitions. All records with the same key are assigned to the same partition. Internally, keyBy() is implemented with hash partitioning. There are different ways to specify keys.
This transformation returns a KeyedStream, which is, among other things, required to use keyed state.
{% highlight java %} @@ -500,7 +500,7 @@ dataStream.filter { _ != 0 }KeyBy
DataStream → KeyedStreamLogically partitions a stream into disjoint partitions, each partition containing elements of the same key. - Internally, this is implemented with hash partitioning. See keys on how to specify keys. + Internally, this is implemented with hash partitioning. See keys on how to specify keys. This transformation returns a KeyedStream.
{% highlight scala %} dataStream.keyBy("someKey") // Key by field "someKey" diff --git a/docs/dev/stream/operators/windows.zh.md b/docs/dev/stream/operators/windows.zh.md index 72269a45596d99..517f7b33863080 100644 --- a/docs/dev/stream/operators/windows.zh.md +++ b/docs/dev/stream/operators/windows.zh.md @@ -94,7 +94,7 @@ Using the `keyBy(...)` will split your infinite stream into logical keyed stream stream is not keyed. In the case of keyed streams, any attribute of your incoming events can be used as a key -(more details [here]({{ site.baseurl }}/dev/api_concepts.html#specifying-keys)). Having a keyed stream will +(more details [here]({% link dev/stream/state/state.zh.md %}#keyed-datastream)). Having a keyed stream will allow your windowed computation to be performed in parallel by multiple tasks, as each logical keyed stream can be processed independently from the rest. All elements referring to the same key will be sent to the same parallel task. diff --git a/docs/dev/stream/sources.md b/docs/dev/stream/sources.md new file mode 100644 index 00000000000000..3c3db9085e12bf --- /dev/null +++ b/docs/dev/stream/sources.md @@ -0,0 +1,391 @@ +--- +title: "Data Sources" +nav-title: "Data Sources" +nav-parent_id: streaming +nav-pos: 10 +--- + + + +* This will be replaced by the TOC +{:toc} + +++ + +This page describes Flink's Data Source API and the concepts and architecture behind it. +**Read this, if you are interested in how data sources in Flink work, or if you want to implement a new Data Source.** + +If you are looking for pre-defined source connectors, please check the [Connector Docs]({{ site.baseurl }}/dev/connectors/). + + +## Data Source Concepts + +**Core Components** + +A Data Source has three core components: *Splits*, the *SplitEnumerator*, and the *SourceReader*. + + - A **Split** is a portion of data consumed by the source, like a file or a log partition. Splits are granularity by which the source distributes the work and parallelizes the data reading. + + - The **SourceReader** requests *Splits* and processes them, for example by reading the file or log partition represented by the *Split*. The *SourceReader* run in parallel on the Task Managers in the `SourceOperators` and produces the parallel stream of events/records. + + - The **SplitEnumerator** generates the *Splits* and assignes them to the *SourceReaders*. It runs as a single instance on the Job Manager and is responsible for maintaining the backlog of pending *Splits* and assigning them to the readers in a balanced manner. + +The [Source]() class is API entry point that ties the above three components together. + +Note: This describes the new Data Source API, introduced in Flink 1.11 as part of FLIP-27. + This new API is currently in BETA status.
+Most of the existing source connectors are not yet (as of Flink 1.11) implemented using this new API, + but using the previous API, based on SourceFunction.
+++ + +**Unified Across Streaming and Batch** + +The Data Source API supports both unbounded streaming sources and bounded batch sources, in a unified way. + +The difference between both cases is minimal: In the bounded/batch case, the enumerator generates a fix set of splits, and each split is necessarily finite. In the unbounded streaming case, one of the two is not true (splits are not finite, or the enumerator keep generating new splits). + +#### Examples + +Here are some simplified conceptual examples to illustrate how the data source components interact, in streaming and batch cases. + +*Note that this does not the accurately describe how the Kafka and File source implementations work; parts are simplified, for illustrative purposes.* + +**Bounded File Source** + +The source has the URI/Path of a directory to read, and a *Format* that defines how to parse the files. + + - A *Split* is a file, or a region of a file (if the data format supports splitting the file). + - The *SplitEnumerator* lists all files under the given directory path. It assigns Splits to the next reader that requests a Split. Once all Splits are assigned, it responds to requests with *NoMoreSplits*. + - The *SourceReader* requests a Split and reads the assigned Split (file or file region) and parses it using the given Format. If it does not get another Split, but a *NoMoreSplits* message, it finishes. + +**Unbounded Streaming File Source** + +This source works the same way as described above, except that the *SplitEnumerator* never responds with *NoMoreSplits* and periodically lists the contents under the given URI/Path to check for new files. Once it finds new files, it generates new Splits for them and can assign them to the available SourceReaders. + +**Unbounded Streaming Kafka Source** + +The source has a Kafka Topic (or list of Topics or Topic regex) and a *Deserializer* to parse the records. + + - A *Split* is a Kafka Topic Partition. + - The *SplitEnumerator* connects to the brokers to list all topic partitions involved in the subscribed topics. The enumerator can optionally repeat this operation to discover newly added topics/partitions. + - The *SourceReader* reads the assigned Splits (Topic Partitions) using the KafkaConsumer and deserializes the records using the provided Deserializer. The splits (Topic Partitions) do not have an end, so the reader never reaches the end of the data. + +**Bounded Kafka Source** + +Same as above, except that each Split (Topic Partition) has a defined end offset. Once the *SourceReader* reaches the end offset for a Split, it finishes that Split. Once all assigned Splits are finished, the SourceReader finishes. + +## The Data Source API +This section describes the major interfaces of the new Source API introduced in FLIP-27, and provides tips to the developers on the Source development. + +### Source +The [Source](https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/connector/source/Source.java) API is a factory style interface to create the following components. + + - *Split Enumerator* + - *Source Reader* + - *Split Serializer* + - *Enumerator Checkpoint Serializer* + +In addition to that, the Source provides the [boundedness](https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/connector/source/Boundedness.java) attribute of the source, so that Flink can choose appropriate mode to run the Flink jobs. + +The Source implementations should be serializable as the Source instances are serialized and uploaded to the Flink cluster at runtime. + +### SplitEnumerator +The [SplitEnumerator](https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/connector/source/SplitEnumerator.java) is expected to be the "brain" of the Source. Typical implementations of the `SplitEnumerator` do the following: + + - `SourceReader` registration handling + - `SourceReader` failure handling + - The `addSplitsBack()` method will be invoked when a `SourceReader` fails. The SplitEnumerator should take back the split assignments that have not been acknowledged by the failed `SourceReader`. + - `SourceEvent` handling + - `SourceEvent`s are custom events sent between `SplitEnumerator` and `SourceReader`. The implementation can leverage this mechanism to perform sophisticated coordination. + - Split discovery and assignment + - The `SplitEnumerator` can assign splits to the `SourceReader`s in response to various events, including discovery of new splits, new `SourceReader` registration, `SourceReader` failure, etc. + +A `SplitEnumerator` can accomplish the above work with the help of the [SplitEnumeratorContext](https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/connector/source/SplitEnumeratorContext.java) which is provided to the `Source` on creation or restore of the `SplitEnumerator`. +The `SplitEnumeratorContext` allows a `SplitEnumerator` to retrieve necessary information of the readers and perform coordination actions. +The `Source` implementation is expected to pass the `SplitEnumeratorContext` to the `SplitEnumerator` instance. + +While a `SplitEnumerator` implementation can work well in a reactive way by only taking coordination actions when its method is invoked, some `SplitEnumerator` implementations might want to take actions actively. For example, a `SplitEnumerator` may want to periodically run split discovery and assign the new splits to the `SourceReaders`. +Such implementations may find that the `callAsync()` method `SplitEnumeratorContext` is handy. The code snippet below shows how the `SplitEnumerator` implementation can achieve that without maintaining its own threads. +
++{% highlight java %} +class MySplitEnumerator implements SplitEnumerator+ +### SourceReader + +The [SourceReader](https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/connector/source/SourceReader.java) is a component running in the Task Managers to consume the records from the Splits. + +The `SourceReader` exposes a pull-based consumption interface. A Flink task keeps calling `pollNext(ReaderOutput)` in a loop to poll records from the `SourceReader`. The return value of the `pollNext(ReaderOutput)` method indicates the status of the source reader. + + - `MORE_AVAILABLE` - The SourceReader has more records available immediately. + - `NOTHING_AVAILABLE` - The SourceReader does not have more records available at this point, but may have more records in the future. + - `END_OF_INPUT` - The SourceReader has exhausted all the records and reached the end of data. This means the SourceReader can be closed. + +In the interest of performance, a `ReaderOutput` is provided to the `pollNext(ReaderOutput)` method, so a `SourceReader` can emit multiple records in a single call of pollNext() if it has to. For example, sometimes the external system works at the granularity of blocks. A block may contain multiple records but the source can only checkpoint at the block boundaries. In this case the `SourceReader` can emit all the records in one block at a time to the `ReaderOutput`. +**However, the `SourceReader` implementation should avoid emitting multiple records in a single `pollNext(ReaderOutput)` invocation unless necessary.** This is because the task thread that is polling from the `SourceReader` works in an event-loop and cannot block. + +All the state of a `SourceReader` should be maintained inside the `SourceSplit`s which are returned at the `snapshotState()` invocation. Doing this allows the `SourceSplit`s to be reassigned to other `SourceReaders` when needed. + +A `SourceReaderContext` is provided to the `Source` upon a `SourceReader` creation. It is expected that the `Source` will pass the context to the `SourceReader` instance. The `SourceReader` can send `SourceEvent` to its `SplitEnumerator` through the `SourceReaderContext`. A typical design pattern of the `Source` is letting the `SourceReader`s report their local information to the `SplitEnumerator` who has a global view to make decisions. + +The `SourceReader` API is a low level API that allows users to deal with the splits manually and have their own threading model to fetch and handover the records. To facilitate the `SourceReader` implementation, Flink has provided a [SourceReaderBase](https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/reader/SourceReaderBase.java) class which significantly reduces the amount the work needed to write a `SourceReader`. +**It is highly recommended for the connector developers to take advantage of the `SourceReaderBase` instead of writing the `SourceReader`s from scratch**. For more details please check the [Split Reader API](#the-split-reader-api) section. + +### Use the Source +In order to create a `DataStream` from a `Source`, one needs to pass the `Source` to a `StreamExecutionEnvironment`. For example, + +{ + private final long DISCOVER_INTERVAL = 60_000L; + + /** + * A method to discover the splits. + */ + private List discoverSplits() {...} + + @Override + public void start() { + ... + enumContext.callAsync(this::discoverSplits, splits -> { + Map ++ +---- +---- + +## The Split Reader API + +The core SourceReader API is fully asynchronous and requires implementations to manage asynchronous split reading manually. +However, in practice, most sources use perform blocking operations, like blocking *poll()* calls on clients (for example the `KafkaConsumer`), or blocking I/O operations on distributed file systems (HDFS, S3, ...). To make this compatible with the asynchronous Source API, these blocking (synchronous) operations need to happen in separate threads, which hand over the data to the asynchronous part of the reader. + +The [SplitReader](https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/reader/splitreader/SplitReader.java) is the high-level API for simple synchronous reading/polling-based source implementations, like file reading, Kafka, etc. + +The core is the `SourceReaderBase` class, which takes a `SplitReader` and creates fetcher threads running the SplitReader, supporting different consumption threading models. + +### SplitReader + +The `SplitReader` API only has three methods: + - A blocking fetch method to return a [RecordsWithSplitIds](https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/reader/RecordsWithSplitIds.java) + - A non-blocking method to handle split changes. + - A non-blocking wake up method to wake up the blocking fetch operation. + +The `SplitReader` only focuses on reading the records from the external system, therefore is much simpler compared with `SourceReader`. +Please check the Java doc of the class for more details. + +### SourceReaderBase + +It is quite common that a `SourceReader` implementation does the following: + + - Have a pool of threads fetching from splits of the external system in a blocking way. + - Handle the synchronization between the internal fetching threads and other methods invocations such as `pollNext(ReaderOutput)`. + - Maintain the per split watermark for watermark alignment. + - Maintain the state of each split for checkpoint. + +In order to reduce the work of writing a new `SourceReader`, Flink provides a [SourceReaderBase](https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/reader/SourceReaderBase.java) class to serve as a base implementation of the `SourceReader`. +`SourceReaderBase` has all the above work done out of the box. To write a new `SourceReader`, one can just let the `SourceReader` implementation inherit from the `SourceReaderBase`, fill in a few methods and implement a high level [SplitReader](https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/reader/splitreader/SplitReader.java). + +### SplitFetcherManager + +The `SourceReaderBase` supports a few threading models out of the box, depending on the behavior of the [SplitFetcherManager](https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/reader/fetcher/SplitFetcherManager.java) it works with. +The `SplitFetcherManager` helps create and maintain a pool of `SplitFetcher`s each fetching with a `SplitReader`. It also determines how to assign splits to each split fetcher. + +As an example, as illustrated below, a `SplitFetcherManager` may have a fixed number of threads, each fetching from some splits assigned to the `SourceReader`. ++{% highlight java %} +final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); + +Source mySource = new MySource(...); + +DataStream+stream = env.fromSource( + mySource, + WatermarkStrategy.noWatermarks(), + "MySourceName"); +... +{% endhighlight %} + +{% highlight scala %} +val env = StreamExecutionEnvironment.getExecutionEnvironment() + +val mySource = new MySource(...) + +val stream = env.fromSource( + mySource, + WatermarkStrategy.noWatermarks(), + "MySourceName") +... +{% endhighlight %} ++++The following code snippet implements the this threading model. +
++{% highlight java %} +/** + * A SplitFetcherManager that has a fixed size of split fetchers and assign splits + * to the split fetchers based on the hash code of split IDs. + */ +public class FixedSizeSplitFetcherManager+ +And a `SourceReader` using this threading model can be created like following: +splitsToAdd) { + // Group splits by their owner fetchers. + Map +{% highlight java %} +public class FixedFetcherSizeSourceReader+ +Apparently, the `SourceReader` implementations can also implement their own threading model easily on top of the `SplitFetcherManager` and `SourceReaderBase`. + +## Event Time and Watermarks + +*Event Time* assignment and *Watermark Generation* happen as part of the data sources. The event streams leaving the Source Readers have event timestamps and (during streaming execution) contain watermarks. See [Timely Stream Processing]({{ site.baseurl }}/concepts/timely-stream-processing.html) for an introduction to Event Time and Watermarks. + +Important Applications based on the legacy [SourceFunction](https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/SourceFunction.java) typically generate timestamps and watermarks in a separate later step via `stream.assignTimestampsAndWatermarks(WatermarkStrategy)`. This function should not be used with the new sources, because timestamps will be already assigned, and it will override the previous split-aware watermarks. + +#### API + +The `WatermarkStrategy` is passed to the Source during creation in the DataStream API and creates both the [TimestampAssigner](https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/common/eventtime/TimestampAssigner.java) and [WatermarkGenerator](https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/common/eventtime/WatermarkGenerator.java). + +{% highlight java %} +environment.fromSource( + SourcefinishedSplitIds) { + // Do something in the callback for the finished splits. + } + + @Override + protected SplitStateT initializedState(SplitT split) { + ... + } + + @Override + protected SplitT toSplitType(String splitId, SplitStateT splitState) { + ... + } +} +{% endhighlight %} + timestampsAndWatermarks, + String sourceName) +{% endhighlight %} + +The `TimestampAssigner` and `WatermarkGenerator` run transparently as part of the `ReaderOutput`(or `SourceOutput`) so source implementors do not have to implement any timestamp extraction and watermark generation code. + +#### Event Timestamps + +Event timestamps are assigned in two steps: + + 1. The SourceReader may attach the *source record timestamp* to the event, by calling `SourceOutput.collect(event, timestamp)`. + This is relevant only for data sources that are record-based and have timestamps, such as Kafka, Kinesis, Pulsar, or Pravega. + Sources that are not based on records with timestamps (like files) do not have a *source record timestamp*. + This step is part of the source connector implementation and not parameterized by the application that uses the source. + + 2. The `TimestampAssigner`, which is configured by the application, assigns the final timestamp. + The `TimestampAssigner` sees the original *source record timestamp* and the event. The assigner can use the *source record timestamp* or access a field of the event obtain the final event timestamp. + +This two-step approach allows users to reference both timestamps from the source systems and timestamps in the event's data as the event timestamp. + +*Note:* When using a data source without *source record timestamps* (like files) and selecting the *source record timestamp* as the final event timestamp, events will get a default timestamp equal to `LONG_MIN` *(=-9,223,372,036,854,775,808)*. + +#### Watermark Generation + +Watermark Generators are only active during streaming execution. Batch execution deactivates Watermark Generators; all related operations described below become effectively no-ops. + +The data source API supports running watermark generators individually *per split*. That allows Flink to observe the event time progress per split individually, which is important to handle *event time skew* properly and prevent *idle partitions* from holding back the event time progress of the entire application. + + ++ +When implementing a source connector using the *Split Reader API*, this is automatically handled. All implementations based on the Split Reader API have split-aware watermarks out-of-the-box. + +For an implementation of the lower level `SourceReader` API to use split-aware watermark generation, the implementation must ouput events from different splits to different outputs: the *Split-local SourceOutputs*. Split-local outputs can be created and released on the main [ReaderOutput](https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/connector/source/ReaderOutput.java) via the `createOutputForSplit(splitId)` and `releaseOutputForSplit(splitId)` methods. Please refer to the JavaDocs of the class and methods for details. diff --git a/docs/dev/stream/sources.zh.md b/docs/dev/stream/sources.zh.md new file mode 100644 index 00000000000000..a063ecb54b2dae --- /dev/null +++ b/docs/dev/stream/sources.zh.md @@ -0,0 +1,391 @@ +--- +title: "Data Sources" +nav-title: "Data Sources" +nav-parent_id: streaming +nav-pos: 10 +--- + + + +* This will be replaced by the TOC +{:toc} + +
+++ + +This page describes Flink's Data Source API and the concepts and architecture behind it. +**Read this, if you are interested in how data sources in Flink work, or if you want to implement a new Data Source.** + +If you are looking for pre-defined source connectors, please check the [Connector Docs]({{ site.baseurl }}/dev/connectors/). + + +## Data Source Concepts + +**Core Components** + +A Data Source has three core components: *Splits*, the *SplitEnumerator*, and the *SourceReader*. + + - A **Split** is a portion of data consumed by the source, like a file or a log partition. Splits are granularity by which the source distributes the work and parallelizes the data reading. + + - The **SourceReader** requests *Splits* and processes them, for example by reading the file or log partition represented by the *Split*. The *SourceReader* run in parallel on the Task Managers in the `SourceOperators` and produces the parallel stream of events/records. + + - The **SplitEnumerator** generates the *Splits* and assignes them to the *SourceReaders*. It runs as a single instance on the Job Manager and is responsible for maintaining the backlog of pending *Splits* and assigning them to the readers in a balanced manner. + +The [Source]() class is API entry point that ties the above three components together. + +Note: This describes the new Data Source API, introduced in Flink 1.11 as part of FLIP-27. + This new API is currently in BETA status.
+Most of the existing source connectors are not yet (as of Flink 1.11) implemented using this new API, + but using the previous API, based on SourceFunction.
+++ + +**Unified Across Streaming and Batch** + +The Data Source API supports both unbounded streaming sources and bounded batch sources, in a unified way. + +The difference between both cases is minimal: In the bounded/batch case, the enumerator generates a fix set of splits, and each split is necessarily finite. In the unbounded streaming case, one of the two is not true (splits are not finite, or the enumerator keep generating new splits). + +#### Examples + +Here are some simplified conceptual examples to illustrate how the data source components interact, in streaming and batch cases. + +*Note that this does not the accurately describe how the Kafka and File source implementations work; parts are simplified, for illustrative purposes.* + +**Bounded File Source** + +The source has the URI/Path of a directory to read, and a *Format* that defines how to parse the files. + + - A *Split* is a file, or a region of a file (if the data format supports splitting the file). + - The *SplitEnumerator* lists all files under the given directory path. It assigns Splits to the next reader that requests a Split. Once all Splits are assigned, it responds to requests with *NoMoreSplits*. + - The *SourceReader* requests a Split and reads the assigned Split (file or file region) and parses it using the given Format. If it does not get another Split, but a *NoMoreSplits* message, it finishes. + +**Unbounded Streaming File Source** + +This source works the same way as described above, except that the *SplitEnumerator* never responds with *NoMoreSplits* and periodically lists the contents under the given URI/Path to check for new files. Once it finds new files, it generates new Splits for them and can assign them to the available SourceReaders. + +**Unbounded Streaming Kafka Source** + +The source has a Kafka Topic (or list of Topics or Topic regex) and a *Deserializer* to parse the records. + + - A *Split* is a Kafka Topic Partition. + - The *SplitEnumerator* connects to the brokers to list all topic partitions involved in the subscribed topics. The enumerator can optionally repeat this operation to discover newly added topics/partitions. + - The *SourceReader* reads the assigned Splits (Topic Partitions) using the KafkaConsumer and deserializes the records using the provided Deserializer. The splits (Topic Partitions) do not have an end, so the reader never reaches the end of the data. + +**Bounded Kafka Source** + +Same as above, except that each Split (Topic Partition) has a defined end offset. Once the *SourceReader* reaches the end offset for a Split, it finishes that Split. Once all assigned Splits are finished, the SourceReader finishes. + +## The Data Source API +This section describes the major interfaces of the new Source API introduced in FLIP-27, and provides tips to the developers on the Source development. + +### Source +The [Source](https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/connector/source/Source.java) API is a factory style interface to create the following components. + + - *Split Enumerator* + - *Source Reader* + - *Split Serializer* + - *Enumerator Checkpoint Serializer* + +In addition to that, the Source provides the [boundedness](https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/connector/source/Boundedness.java) attribute of the source, so that Flink can choose appropriate mode to run the Flink jobs. + +The Source implementations should be serializable as the Source instances are serialized and uploaded to the Flink cluster at runtime. + +### SplitEnumerator +The [SplitEnumerator](https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/connector/source/SplitEnumerator.java) is expected to be the "brain" of the Source. Typical implementations of the `SplitEnumerator` do the following: + + - `SourceReader` registration handling + - `SourceReader` failure handling + - The `addSplitsBack()` method will be invoked when a `SourceReader` fails. The SplitEnumerator should take back the split assignments that have not been acknowledged by the failed `SourceReader`. + - `SourceEvent` handling + - `SourceEvent`s are custom events sent between `SplitEnumerator` and `SourceReader`. The implementation can leverage this mechanism to perform sophisticated coordination. + - Split discovery and assignment + - The `SplitEnumerator` can assign splits to the `SourceReader`s in response to various events, including discovery of new splits, new `SourceReader` registration, `SourceReader` failure, etc. + +A `SplitEnumerator` can accomplish the above work with the help of the [SplitEnumeratorContext](https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/connector/source/SplitEnumeratorContext.java) which is provided to the `Source` on creation or restore of the `SplitEnumerator`. +The `SplitEnumeratorContext` allows a `SplitEnumerator` to retrieve necessary information of the readers and perform coordination actions. +The `Source` implementation is expected to pass the `SplitEnumeratorContext` to the `SplitEnumerator` instance. + +While a `SplitEnumerator` implementation can work well in a reactive way by only taking coordination actions when its method is invoked, some `SplitEnumerator` implementations might want to take actions actively. For example, a `SplitEnumerator` may want to periodically run split discovery and assign the new splits to the `SourceReaders`. +Such implementations may find that the `callAsync()` method `SplitEnumeratorContext` is handy. The code snippet below shows how the `SplitEnumerator` implementation can achieve that without maintaining its own threads. +
++{% highlight java %} +class MySplitEnumerator implements SplitEnumerator+ +### SourceReader + +The [SourceReader](https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/connector/source/SourceReader.java) is a component running in the Task Managers to consume the records from the Splits. + +The `SourceReader` exposes a pull-based consumption interface. A Flink task keeps calling `pollNext(ReaderOutput)` in a loop to poll records from the `SourceReader`. The return value of the `pollNext(ReaderOutput)` method indicates the status of the source reader. + + - `MORE_AVAILABLE` - The SourceReader has more records available intermediately. + - `NOTHING_AVAILABLE` - The SourceReader does not have more records available at this point, but may have more records in the future. + - `END_OF_INPUT` - The SourceReader has exhausted all the records and reached the end of data. This means the SourceReader can be closed. + +In the interest of performance, a `ReaderOutput` is provided to the `pollNext(ReaderOutput)` method, so a `SourceReader` can emit multiple records in a single call of pollNext() if it has to. For example, sometimes the external system works at the granularity of blocks. A block may contain multiple records but the source can only checkpoint at the block boundaries. In this case the `SourceReader` can emit one block at a time to the `ReaderOutput`. +**However, the `SourceReader` implementation should avoid emitting multiple records in a single `pollNext(ReaderOutput)` invocation unless necessary.** + +All the state of a `SourceReader` should be maintained inside the `SourceSplit`s which are returned at the `snapshotState()` invocation. Doing this allows the `SourceSplit`s to be reassigned to other `SourceReaders` when needed. + +A `SourceReaderContext` is provided to the `Source` upon a `SourceReader` creation. It is expected that the `Source` will pass the context to the `SourceReader` instance. The `SourceReader` can send `SourceEvent` to its `SplitEnumerator`. A typical design pattern for the `Source` is letting the `SourceReader`s report their local information to the `SplitEnumerator` who has a global view to make decisions. + +The `SourceReader` API is a low level API that allows users to deal with the splits manually and have their own threading model to fetch and handover the records. To facilitate the `SourceReader` implementation, Flink has provided a [SourceReaderBase](https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/reader/SourceReaderBase.java) class which significantly reduce the amount the work needed to write a `SourceReader`. +**It is highly recommended for the connector developers to take advantage of the `SourceReaderBase` instead of writing the `SourceReader`s by themselves from scratch**. For more details please check the [Split Reader API](#the-split-reader-api) section. + +### Use the Source +In order to create a `DataStream` from a `Source`, one needs to pass the `Source` to a `StreamExecutionEnvironment`. For example, + +{ + private final long DISCOVER_INTERVAL = 60_000L; + + /** + * A method to discover the splits. + */ + private List discoverSplits() {...} + + @Override + public void start() { + ... + enumContext.callAsync(this::discoverSplits, splits -> { + Map ++ +---- +---- + +## The Split Reader API + +The core SourceReader API is fully asynchronous and requires implementations to manage asynchronous split reading manually. +However, in practice, most sources use perform blocking operations, like blocking *poll()* calls on clients (for example the `KafkaConsumer`), or blocking I/O operations on distributed file systems (HDFS, S3, ...). To make this compatible with the asynchronous Source API, these blocking (synchronous) operations need to happen in separate threads, which hand over the data to the asynchronous part of the reader. + +The [SplitReader](https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/reader/splitreader/SplitReader.java) is the high-level API for simple synchronous reading/polling-based source implementations, like file reading, Kafka, etc. + +The core is the `SourceReaderBase` class, which takes a `SplitReader` and creates fetcher threads running the SplitReader, supporting different consumption threading models. + +### SplitReader + +The `SplitReader` API only has three methods: + - A blocking fetch method to return a [RecordsWithSplitIds](https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/reader/RecordsWithSplitIds.java) + - A non-blocking method to handle split changes. + - A non-blocking wake up method to wake up the blocking fetch operation. + +The `SplitReader` only focuses on reading the records from the external system, therefore is much simpler compared with `SourceReader`. +Please check the Java doc of the class for more details. + +### SourceReaderBase + +It is quite common that a `SourceReader` implementation does the following: + + - Have a pool of threads fetching from splits of the external system in a blocking way. + - Handle the synchronization between the fetching threads mentioned above and the `pollNext(ReaderOutput)` invocation. + - Maintain the per split watermark for watermark alignment. + - Maintain the state of each split for checkpoint + +In order to reduce the work of writing a new `SourceReader`, Flink provides a [SourceReaderBase](https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/reader/SourceReaderBase.java) class to serve as a base implementation of the `SourceReader`. +`SourceReaderBase` has all the above work done out of the box. To write a new `SourceReader`, one can just let the `SourceReader` implementation inherit from the `SourceReaderBase`, fill in a few methods and implement a high level [SplitReader](https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/reader/splitreader/SplitReader.java). + +### SplitFetcherManager + +The `SourceReaderBase` supports a few threading models out of the box, depending on the behavior of the [SplitFetcherManager](https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/reader/fetcher/SplitFetcherManager.java) it works with. +The `SplitFetcherManager` helps create and maintain a pool of `SplitFetcher`s each fetching with a `SplitReader`. It also determines how to assign splits to each split fetcher. + +As an example, as illustrated below, a `SplitFetcherManager` may have a fixed number of threads, each fetching from some splits assigned to the `SourceReader`. ++{% highlight java %} +final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); + +Source mySource = new MySource(...); + +DataStream+stream = env.fromSource( + mySource, + WatermarkStrategy.noWatermarks(), + "MySourceName"); +... +{% endhighlight %} + +{% highlight scala %} +val env = StreamExecutionEnvironment.getExecutionEnvironment() + +val mySource = new MySource(...) + +val stream = env.fromSource( + mySource, + WatermarkStrategy.noWatermarks(), + "MySourceName") +... +{% endhighlight %} ++++The following code snippet implements the this threading model. +
++{% highlight java %} +/** + * A SplitFetcherManager that has a fixed size of split fetchers and assign splits + * to the split fetchers based on the hash code of split IDs. + */ +public class FixedSizeSplitFetcherManager+ +And a `SourceReader` using this threading model can be created like following: +splitsToAdd) { + // Group splits by their owner fetchers. + Map +{% highlight java %} +public class FixedFetcherSizeSourceReader+ +Apparently, the `SourceReader` implementations can also implement their own threading model easily on top of the `SplitFetcherManager` and `SourceReaderBase`. + +## Event Time and Watermarks + +*Event Time* assignment and *Watermark Generation* happen as part of the data sources. The event streams leaving the Source Readers have event timestamps and (during streaming execution) contain watermarks. See [Timely Stream Processing]({{ site.baseurl }}/concepts/timely-stream-processing.html) for an introduction to Event Time and Watermarks. + +Important Applications based on the legacy [SourceFunction](https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/SourceFunction.java) typically generate timestamps and watermarks in a separate later step via `stream.assignTimestampsAndWatermarks(WatermarkStrategy)`. This function should not be used with the new sources, because timestamps will be already assigned, and it will override the previous split-aware watermarks. + +#### API + +The `WatermarkStrategy` is passed to the Source during creation in the DataStream API and creates both the [TimestampAssigner](https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/common/eventtime/TimestampAssigner.java) and [WatermarkGenerator](https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/common/eventtime/WatermarkGenerator.java). + +{% highlight java %} +environment.fromSource( + SourcefinishedSplitIds) { + // Do something in the callback for the finished splits. + } + + @Override + protected SplitStateT initializedState(SplitT split) { + ... + } + + @Override + protected SplitT toSplitType(String splitId, SplitStateT splitState) { + ... + } +} +{% endhighlight %} + timestampsAndWatermarks, + String sourceName) +{% endhighlight %} + +The `TimestampAssigner` and `WatermarkGenerator` run transparently as part of the `ReaderOutput`(or `SourceOutput`) so source implementors do not have to implement any timestamp extraction and watermark generation code. + +#### Event Timestamps + +Event timestamps are assigned in two steps: + + 1. The SourceReader may attach the *source record timestamp* to the event, by calling `SourceOutput.collect(event, timestamp)`. + This is relevant only for data sources that are record-based and have timestamps, such as Kafka, Kinesis, Pulsar, or Pravega. + Sources that are not based on records with timestamps (like files) do not have a *source record timestamp*. + This step is part of the source connector implementation and not parameterized by the application that uses the source. + + 2. The `TimestampAssigner`, which is configured by the application, assigns the final timestamp. + The `TimestampAssigner` sees the original *source record timestamp* and the event. The assigner can use the *source record timestamp* or access a field of the event obtain the final event timestamp. + +This two-step approach allows users to reference both timestamps from the source systems and timestamps in the event's data as the event timestamp. + +*Note:* When using a data source without *source record timestamps* (like files) and selecting the *source record timestamp* as the final event timestamp, events will get a default timestamp equal to `LONG_MIN` *(=-9,223,372,036,854,775,808)*. + +#### Watermark Generation + +Watermark Generators are only active during streaming execution. Batch execution deactivates Watermark Generators; all related operations described below become effectively no-ops. + +The data source API supports running watermark generators individually *per split*. That allows Flink to observe the event time progress per split individually, which is important to handle *event time skew* properly and prevent *idle partitions* from holding back the event time progress of the entire application. + + ++ +When implementing a source connector using the *Split Reader API*, this is automatically handled. All implementations based on the Split Reader API have split-aware watermarks out-of-the-box. + +For an implementation of the lower level `SourceReader` API to use split-aware watermark generation, the implementation must ouput events from different splits to different outputs: the *Split-local SourceOutputs*. Split-local outputs can be created and released on the main [ReaderOutput](https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/connector/source/ReaderOutput.java) via the `createOutputForSplit(splitId)` and `releaseOutputForSplit(splitId)` methods. Please refer to the JavaDocs of the class and methods for details. diff --git a/docs/dev/stream/state/checkpointing.md b/docs/dev/stream/state/checkpointing.md index c193fc37d71ed5..43dc4b9a94442b 100644 --- a/docs/dev/stream/state/checkpointing.md +++ b/docs/dev/stream/state/checkpointing.md @@ -32,7 +32,7 @@ any type of more elaborate operation. In order to make state fault tolerant, Flink needs to **checkpoint** the state. Checkpoints allow Flink to recover state and positions in the streams to give the application the same semantics as a failure-free execution. -The [documentation on streaming fault tolerance]({{ site.baseurl }}/internals/stream_checkpointing.html) describes in detail the technique behind Flink's streaming fault tolerance mechanism. +The [documentation on streaming fault tolerance]({% link learn-flink/fault_tolerance.md %}) describes in detail the technique behind Flink's streaming fault tolerance mechanism. ## Prerequisites @@ -72,12 +72,14 @@ Other parameters for checkpointing include: This option cannot be used when a minimum time between checkpoints is defined. - - *externalized checkpoints*: You can configure periodic checkpoints to be persisted externally. Externalized checkpoints write their meta data out to persistent storage and are *not* automatically cleaned up when the job fails. This way, you will have a checkpoint around to resume from if your job fails. There are more details in the [deployment notes on externalized checkpoints]({{ site.baseurl }}/ops/state/checkpoints.html#externalized-checkpoints). + - *externalized checkpoints*: You can configure periodic checkpoints to be persisted externally. Externalized checkpoints write their meta data out to persistent storage and are *not* automatically cleaned up when the job fails. This way, you will have a checkpoint around to resume from if your job fails. There are more details in the [deployment notes on externalized checkpoints]({% link ops/state/checkpoints.md %}#externalized-checkpoints). - *fail/continue task on checkpoint errors*: This determines if a task will be failed if an error occurs in the execution of the task's checkpoint procedure. This is the default behaviour. Alternatively, when this is disabled, the task will simply decline the checkpoint to the checkpoint coordinator and continue running. - *prefer checkpoint for recovery*: This determines if a job will fallback to latest checkpoint even when there are more recent savepoints available to potentially reduce recovery time. + - *unaligned checkpoints*: You can enable [unaligned checkpoints]({% link ops/state/checkpoints.md %}#unaligned-checkpoints) to greatly reduce checkpointing times under backpressure. Only works for exactly-once checkpoints and with number of concurrent checkpoints of 1. +
+### Related Config Options -Some more parameters and/or defaults may be set via `conf/flink-conf.yaml` (see [configuration]({{ site.baseurl }}/ops/config.html) for a full guide): +Some more parameters and/or defaults may be set via `conf/flink-conf.yaml` (see [configuration]({% link ops/config.md %}) for a full guide): {% include generated/checkpointing_configuration.html %} @@ -173,7 +184,7 @@ Some more parameters and/or defaults may be set via `conf/flink-conf.yaml` (see ## Selecting a State Backend -Flink's [checkpointing mechanism]({{ site.baseurl }}/internals/stream_checkpointing.html) stores consistent snapshots +Flink's [checkpointing mechanism]({% link learn-flink/fault_tolerance.md %}) stores consistent snapshots of all the state in timers and stateful operators, including connectors, windows, and any [user-defined state](state.html). Where the checkpoints are stored (e.g., JobManager memory, file system, database) depends on the configured **State Backend**. @@ -181,7 +192,7 @@ Where the checkpoints are stored (e.g., JobManager memory, file system, database By default, state is kept in memory in the TaskManagers and checkpoints are stored in memory in the JobManager. For proper persistence of large state, Flink supports various approaches for storing and checkpointing state in other state backends. The choice of state backend can be configured via `StreamExecutionEnvironment.setStateBackend(…)`. -See [state backends]({{ site.baseurl }}/ops/state/state_backends.html) for more details on the available state backends and options for job-wide and cluster-wide configuration. +See [state backends]({% link ops/state/state_backends.md %}) for more details on the available state backends and options for job-wide and cluster-wide configuration. ## State Checkpoints in Iterative Jobs @@ -196,7 +207,7 @@ Please note that records in flight in the loop edges (and the state changes asso ## Restart Strategies Flink supports different restart strategies which control how the jobs are restarted in case of a failure. For more -information, see [Restart Strategies]({{ site.baseurl }}/dev/restart_strategies.html). +information, see [Restart Strategies]({% link dev/task_failure_recovery.md %}). {% top %} diff --git a/docs/dev/stream/state/checkpointing.zh.md b/docs/dev/stream/state/checkpointing.zh.md index d4aa989523a680..f9f0592a9e69d1 100644 --- a/docs/dev/stream/state/checkpointing.zh.md +++ b/docs/dev/stream/state/checkpointing.zh.md @@ -29,7 +29,7 @@ Flink 中的每个方法或算子都能够是**有状态的**（阅读 [working 状态化的方法在处理单个 元素/事件 的时候存储数据，让状态成为使各个类型的算子更加精细的重要部分。 为了让状态容错，Flink 需要为状态添加 **checkpoint（检查点）**。Checkpoint 使得 Flink 能够恢复状态和在流中的位置，从而向应用提供和无故障执行时一样的语义。 -[容错文档]({{ site.baseurl }}/zh/internals/stream_checkpointing.html) 中介绍了 Flink 流计算容错机制内部的技术原理。 +[容错文档]({{ site.baseurl }}/zh/learn-flink/fault_tolerance.html) 中介绍了 Flink 流计算容错机制内部的技术原理。 ## 前提条件 @@ -165,7 +165,7 @@ env.get_checkpoint_config().set_prefer_checkpoint_for_recovery(True) ## 选择一个 State Backend -Flink 的 [checkpointing 机制]({{ site.baseurl }}/zh/internals/stream_checkpointing.html) 会将 timer 以及 stateful 的 operator 进行快照，然后存储下来， +Flink 的 [checkpointing 机制]({{ site.baseurl }}/zh/learn-flink/fault_tolerance.html) 会将 timer 以及 stateful 的 operator 进行快照，然后存储下来， 包括连接器（connectors），窗口（windows）以及任何用户[自定义的状态](state.html)。 Checkpoint 存储在哪里取决于所配置的 **State Backend**（比如 JobManager memory、 file system、 database）。 @@ -184,7 +184,7 @@ Flink 现在为没有迭代（iterations）的作业提供一致性的处理保 ## 重启策略 -Flink 支持不同的重启策略，来控制 job 万一故障时该如何重启。更多信息请阅读 [重启策略]({{ site.baseurl }}/zh/dev/restart_strategies.html)。 +Flink 支持不同的重启策略，来控制 job 万一故障时该如何重启。更多信息请阅读 [重启策略]({{ site.baseurl }}/zh/dev/task_failure_recovery.html)。 {% top %} diff --git a/docs/dev/stream/state/custom_serialization.md b/docs/dev/stream/state/custom_serialization.md index 79f05a36067f14..e7413022476417 100644 --- a/docs/dev/stream/state/custom_serialization.md +++ b/docs/dev/stream/state/custom_serialization.md @@ -309,7 +309,7 @@ the nested element serializer. In these cases, an additional three methods need to be implemented on the `CompositeTypeSerializerSnapshot`: * `#writeOuterSnapshot(DataOutputView)`: defines how the outer snapshot information is written. * `#readOuterSnapshot(int, DataInputView, ClassLoader)`: defines how the outer snapshot information is read. - * `#isOuterSnapshotCompatible(TypeSerializer)`: checks whether the outer snapshot information remains identical. + * `#resolveOuterSchemaCompatibility(TypeSerializer)`: checks the compatibility based on the outer snapshot information. By default, the `CompositeTypeSerializerSnapshot` assumes that there isn't any outer snapshot information to read / write, and therefore have empty default implementations for the above methods. If the subclass @@ -351,8 +351,10 @@ public final class GenericArraySerializerSnapshot{% highlight java %} @@ -105,6 +107,9 @@ env.getCheckpointConfig().enableExternalizedCheckpoints(ExternalizedCheckpointCl // allow job recovery fallback to checkpoint when there is a more recent savepoint env.getCheckpointConfig().setPreferCheckpointForRecovery(true); + +// enables the experimental unaligned checkpoints +env.getCheckpointConfig.enableUnalignedCheckpoints(); {% endhighlight %}@@ -130,6 +135,9 @@ env.getCheckpointConfig.setFailTasksOnCheckpointingErrors(false) // allow only one checkpoint to be in progress at the same time env.getCheckpointConfig.setMaxConcurrentCheckpoints(1) + +// enables the experimental unaligned checkpoints +env.getCheckpointConfig.enableUnalignedCheckpoints() {% endhighlight %}@@ -158,13 +166,16 @@ env.get_checkpoint_config().enable_externalized_checkpoints(ExternalizedCheckpoi # allow job recovery fallback to checkpoint when there is a more recent savepoint env.get_checkpoint_config().set_prefer_checkpoint_for_recovery(True) + +// enables the experimental unaligned checkpoints +env.get_checkpoint_config().enable_unaligned_checkpoints() {% endhighlight %}extends CompositeTypeSerial } @Override - protected boolean isOuterSnapshotCompatible(GenericArraySerializer newSerializer) { - return this.componentClass == newSerializer.getComponentClass(); + protected boolean resolveOuterSchemaCompatibility(GenericArraySerializer newSerializer) { + return (this.componentClass == newSerializer.getComponentClass()) + ? OuterSchemaCompatibility.COMPATIBLE_AS_IS + : OuterSchemaCompatibility.INCOMPATIBLE; } @Override diff --git a/docs/dev/stream/state/custom_serialization.zh.md b/docs/dev/stream/state/custom_serialization.zh.md index daeea171d6fd19..2706e8088837d9 100644 --- a/docs/dev/stream/state/custom_serialization.zh.md +++ b/docs/dev/stream/state/custom_serialization.zh.md @@ -87,7 +87,7 @@ type and the *serialized binary format* of a state type. The schema, generally s 1. Data schema of the state type has evolved, i.e. adding or removing a field from a POJO that is used as state. 2. Generally speaking, after a change to the data schema, the serialization format of the serializer will need to be upgraded. 3. Configuration of the serializer has changed. - + In order for the new execution to have information about the *written schema* of state and detect whether or not the schema has changed, upon taking a savepoint of an operator's state, a *snapshot* of the state serializer needs to be written along with the state bytes. This is abstracted a `TypeSerializerSnapshot`, explained in the next subsection. @@ -108,10 +108,10 @@ public interface TypeSerializerSnapshot { {% highlight java %} -public abstract class TypeSerializer@@ -470,9 +487,44 @@ val counts: DataStream[(String, Int)] = stream }) {% endhighlight %} -## 使用 Managed Operator State +## Operator State + +*Operator State* (or *non-keyed state*) is state that is is bound to one +parallel operator instance. The [Kafka Connector]({% link +dev/connectors/kafka.zh.md %}) is a good motivating example for the use of +Operator State in Flink. Each parallel instance of the Kafka consumer maintains +a map of topic partitions and offsets as its Operator State. + +The Operator State interfaces support redistributing state among parallel +operator instances when the parallelism is changed. There are different schemes +for doing this redistribution. + +In a typical stateful Flink Application you don't need operators state. It is +mostly a special type of state that is used in source/sink implementations and +scenarios where you don't have a key by which state can be partitioned. + +## Broadcast State + +*Broadcast State* is a special type of *Operator State*. It was introduced to +support use cases where records of one stream need to be broadcasted to all +downstream tasks, where they are used to maintain the same state among all +subtasks. This state can then be accessed while processing records of a second +stream. As an example where broadcast state can emerge as a natural fit, one +can imagine a low-throughput stream containing a set of rules which we want to +evaluate against all elements coming from another stream. Having the above type +of use cases in mind, broadcast state differs from the rest of operator states +in that: + + 1. it has a map format, + 2. it is only available to specific operators that have as inputs a + *broadcasted* stream and a *non-broadcasted* one, and + 3. such an operator can have *multiple broadcast states* with different names. + +{% top %} + +## 使用 Operator State -用户可以通过实现 `CheckpointedFunction` 或 `ListCheckpointed{ - +public abstract class TypeSerializer { + // ... - + public abstract TypeSerializerSnapshot snapshotConfiguration(); } {% endhighlight %} @@ -140,7 +140,7 @@ which can be one of the following: 2. **`TypeSerializerSchemaCompatibility.compatibleAfterMigration()`**: this result signals that the new serializer has a different serialization schema, and it is possible to migrate from the old schema by using the previous serializer (which recognizes the old schema) to read bytes into state objects, and then rewriting the object back to bytes with - the new serializer (which recognizes the new schema). + the new serializer (which recognizes the new schema). 3. **`TypeSerializerSchemaCompatibility.incompatible()`**: this result signals that the new serializer has a different serialization schema, but it is not possible to migrate from the old schema. @@ -170,13 +170,13 @@ to the implementation of state serializers and their serializer snapshots. - Upon receiving the new serializer, it is provided to the restored previous serializer's snapshot via the `TypeSerializer#resolveSchemaCompatibility` to check for schema compatibility. 4. **Migrate state bytes in backend from schema _A_ to schema _B_** - - If the compatibility resolution reflects that the schema has changed and migration is possible, schema migration is + - If the compatibility resolution reflects that the schema has changed and migration is possible, schema migration is performed. The previous state serializer which recognizes schema _A_ will be obtained from the serializer snapshot, via `TypeSerializerSnapshot#restoreSerializer()`, and is used to deserialize state bytes to objects, which in turn are re-written again with the new serializer, which recognizes schema _B_ to complete the migration. All entries of the accessed state is migrated all-together before processing continues. - If the resolution signals incompatibility, then the state access fails with an exception. - + #### Heap state backends (e.g. `MemoryStateBackend`, `FsStateBackend`) 1. **Register new state with a state serializer that has schema _A_** @@ -218,7 +218,7 @@ as your serializer's snapshot class: - `TypeSerializerSchemaCompatibility.compatibleAsIs()`, if the new serializer class remains identical, or - `TypeSerializerSchemaCompatibility.incompatible()`, if the new serializer class is different then the previous one. - + Below is an example of how the `SimpleTypeSerializerSnapshot` is used, using Flink's `IntSerializer` as an example: {% highlight java %} @@ -309,7 +309,7 @@ the nested element serializer. In these cases, an additional three methods need to be implemented on the `CompositeTypeSerializerSnapshot`: * `#writeOuterSnapshot(DataOutputView)`: defines how the outer snapshot information is written. * `#readOuterSnapshot(int, DataInputView, ClassLoader)`: defines how the outer snapshot information is read. - * `#isOuterSnapshotCompatible(TypeSerializer)`: checks whether the outer snapshot information remains identical. + * `#resolveOuterSchemaCompatibility(TypeSerializer)`: checks the compatibility based on the outer snapshot information. By default, the `CompositeTypeSerializerSnapshot` assumes that there isn't any outer snapshot information to read / write, and therefore have empty default implementations for the above methods. If the subclass @@ -351,8 +351,10 @@ public final class GenericArraySerializerSnapshot-For more on this, you can check how to [set up a Flink program]({{ site.baseurl }}/dev/projectsetup/dependencies.html). +For more on this, you can check how to [set up a Flink program]({{ site.baseurl }}/dev/project-configuration.html). The `QueryableStateClient` will submit your query to the internal proxy, which will then process your query and return the final result. The only requirement to initialize the client is to provide a valid `TaskManager` hostname (remember diff --git a/docs/dev/stream/state/queryable_state.zh.md b/docs/dev/stream/state/queryable_state.zh.md index 3c14c7e8360a92..4b76b4d40f6c25 100644 --- a/docs/dev/stream/state/queryable_state.zh.md +++ b/docs/dev/stream/state/queryable_state.zh.md @@ -1,5 +1,5 @@ --- -title: "可查询状态" +title: "Queryable State" nav-parent_id: streaming_state nav-pos: 4 is_beta: true @@ -27,75 +27,52 @@ under the License. {:toc}extends CompositeTypeSerial } @Override - protected boolean isOuterSnapshotCompatible(GenericArraySerializer newSerializer) { - return this.componentClass == newSerializer.getComponentClass(); + protected boolean resolveOuterSchemaCompatibility(GenericArraySerializer newSerializer) { + return (this.componentClass == newSerializer.getComponentClass()) + ? OuterSchemaCompatibility.COMPATIBLE_AS_IS + : OuterSchemaCompatibility.INCOMPATIBLE; } @Override diff --git a/docs/dev/stream/state/index.zh.md b/docs/dev/stream/state/index.zh.md index 1b5444873a0a28..97e2d97a4022d9 100644 --- a/docs/dev/stream/state/index.zh.md +++ b/docs/dev/stream/state/index.zh.md @@ -25,34 +25,17 @@ specific language governing permissions and limitations under the License. --> -Stateful functions and operators store data across the processing of individual elements/events, making state a critical building block for -any type of more elaborate operation. - -For example: - - - When an application searches for certain event patterns, the state will store the sequence of events encountered so far. - - When aggregating events per minute/hour/day, the state holds the pending aggregates. - - When training a machine learning model over a stream of data points, the state holds the current version of the model parameters. - - When historic data needs to be managed, the state allows efficient access to events that occurred in the past. - -Flink needs to be aware of the state in order to make state fault tolerant using [checkpoints](checkpointing.html) and to allow [savepoints]({{ site.baseurl }}/ops/state/savepoints.html) of streaming applications. - -Knowledge about the state also allows for rescaling Flink applications, meaning that Flink takes care of redistributing state across parallel instances. - -The [queryable state](queryable_state.html) feature of Flink allows you to access state from outside of Flink during runtime. - -When working with state, it might also be useful to read about [Flink's state backends]({{ site.baseurl }}/ops/state/state_backends.html). Flink provides different state backends that specify how and where state is stored. State can be located on Java's heap or off-heap. Depending on your state backend, Flink can also *manage* the state for the application, meaning Flink deals with the memory management (possibly spilling to disk if necessary) to allow applications to hold very large state. State backends can be configured without changing your application logic. - +你将在本节中了解到 Flink 提供的用于编写有状态程序的 API，想了解更多有状态流处理的概念，请查看[有状态的流处理]({% link concepts/stateful-stream-processing.zh.md %}) {% top %} -Where to go next? +接下来看什么? ----------------- -* [Working with State](state.html): Shows how to use state in a Flink application and explains the different kinds of state. -* [The Broadcast State Pattern](broadcast_state.html): Explains how to connect a broadcast stream with a non-broadcast stream and use state to exchange information between them. -* [Checkpointing](checkpointing.html): Describes how to enable and configure checkpointing for fault tolerance. -* [Queryable State](queryable_state.html): Explains how to access state from outside of Flink during runtime. -* [State Schema Evolution](schema_evolution.html): Shows how schema of state types can be evolved. -* [Custom Serialization for Managed State](custom_serialization.html): Discusses how to implement custom serializers, especially for schema evolution. +* [Working with State](state.html): 描述了如何在 Flink 应用程序中使用状态，以及不同类型的状态。 +* [The Broadcast State 模式](broadcast_state.html): 描述了如何将广播流和非广播流进行连接从而交换数据。 +* [Checkpointing](checkpointing.html): 介绍了如何开启和配置 checkpoint，以实现状态容错。 +* [Queryable State](queryable_state.html): 介绍了如何从外围访问 Flink 的状态。 +* [状态数据结构升级](schema_evolution.html): 介绍了状态数据结构升级相关的内容。 +* [Managed State 的自定义序列化器](custom_serialization.html): 介绍了如何实现自定义的序列化器，尤其是如何支持状态数据结构升级。 {% top %} diff --git a/docs/dev/stream/state/queryable_state.md b/docs/dev/stream/state/queryable_state.md index d4c287921f235e..03a24729c90dc5 100644 --- a/docs/dev/stream/state/queryable_state.md +++ b/docs/dev/stream/state/queryable_state.md @@ -180,7 +180,7 @@ jar which must be explicitly included as a dependency in the `pom.xml` of your p {% endhighlight %} - Note: The client APIs for queryable state are currently in an evolving state and - there are no guarantees made about stability of the provided interfaces. It is - likely that there will be breaking API changes on the client side in the upcoming Flink versions. + 注意: 目前 querable state 的客户端 API 还在不断演进，不保证现有接口的稳定性。在后续的 Flink 版本中有可能发生 API 变化。-In a nutshell, this feature exposes Flink's managed keyed (partitioned) state -(see [Working with State]({{ site.baseurl }}/dev/stream/state/state.html)) to the outside world and -allows the user to query a job's state from outside Flink. For some scenarios, queryable state -eliminates the need for distributed operations/transactions with external systems such as key-value -stores which are often the bottleneck in practice. In addition, this feature may be particularly -useful for debugging purposes. +简而言之, 这个特性将 Flink 的 managed keyed (partitioned) state +(参考 [Working with State]({{ site.baseurl }}/zh/dev/stream/state/state.html)) 暴露给外部，从而用户可以在 Flink 外部查询作业 state。 +在某些场景中，Queryable State 消除了对外部系统的分布式操作以及事务的需求，比如 KV 存储系统，而这些外部系统往往会成为瓶颈。除此之外，这个特性对于调试作业非常有用。- Attention: When querying a state object, that object is accessed from a concurrent - thread without any synchronization or copying. This is a design choice, as any of the above would lead - to increased job latency, which we wanted to avoid. Since any state backend using Java heap space, - e.g.-## Architecture +## 架构 -Before showing how to use the Queryable State, it is useful to briefly describe the entities that compose it. -The Queryable State feature consists of three main entities: +在展示如何使用 Queryable State 之前，先简单描述一下该特性的组成部分，主要包括以下三部分: - 1. the `QueryableStateClient`, which (potentially) runs outside the Flink cluster and submits the user queries, - 2. the `QueryableStateClientProxy`, which runs on each `TaskManager` (*i.e.* inside the Flink cluster) and is responsible - for receiving the client's queries, fetching the requested state from the responsible Task Manager on his behalf, and - returning it to the client, and - 3. the `QueryableStateServer` which runs on each `TaskManager` and is responsible for serving the locally stored state. + 1. `QueryableStateClient`，默认运行在 Flink 集群外部，负责提交用户的查询请求； + 2. `QueryableStateClientProxy`，运行在每个 `TaskManager` 上(*即* Flink 集群内部)，负责接收客户端的查询请求，从所负责的 Task Manager 获取请求的 state，并返回给客户端； + 3. `QueryableStateServer`, 运行在 `TaskManager` 上，负责服务本地存储的 state。 -The client connects to one of the proxies and sends a request for the state associated with a specific -key, `k`. As stated in [Working with State]({{ site.baseurl }}/dev/stream/state/state.html), keyed state is organized in -*Key Groups*, and each `TaskManager` is assigned a number of these key groups. To discover which `TaskManager` is -responsible for the key group holding `k`, the proxy will ask the `JobManager`. Based on the answer, the proxy will -then query the `QueryableStateServer` running on that `TaskManager` for the state associated with `k`, and forward the -response back to the client. +客户端连接到一个代理，并发送请求获取特定 `k` 对应的 state。 如 [Working with State]({{ site.baseurl }}/zh/dev/stream/state/state.html) 所述，keyed state 按照 +*Key Groups* 进行划分，每个 `TaskManager` 会分配其中的一些 key groups。代理会询问 `JobManager` 以找到 `k` 所属 key group 的 TaskManager。根据返回的结果, 代理将会向运行在 `TaskManager` 上的 `QueryableStateServer` 查询 `k` 对应的 state， 并将结果返回给客户端。 -## Activating Queryable State +## 激活 Queryable State -To enable queryable state on your Flink cluster, you need to do the following: +为了在 Flink 集群上使用 queryable state，需要进行以下操作： - 1. copy the `flink-queryable-state-runtime{{ site.scala_version_suffix }}-{{site.version }}.jar` -from the `opt/` folder of your [Flink distribution](https://flink.apache.org/downloads.html "Apache Flink: Downloads"), -to the `lib/` folder. - 2. set the property `queryable-state.enable` to `true`. See the [Configuration]({{ site.baseurl }}/ops/config.html#queryable-state) documentation for details and additional parameters. + 1. 将 `flink-queryable-state-runtime{{ site.scala_version_suffix }}-{{site.version }}.jar` +从 [Flink distribution](https://flink.apache.org/downloads.html "Apache Flink: Downloads") 的 `opt/` 目录拷贝到 `lib/` 目录； + 2. 将参数 `queryable-state.enable` 设置为 `true`。详细信息以及其它配置可参考文档 [Configuration]({{ site.baseurl }}/zh/ops/config.html#queryable-state)。 -To verify that your cluster is running with queryable state enabled, check the logs of any -task manager for the line: `"Started the Queryable State Proxy Server @ ..."`. +为了验证集群的 queryable state 已经被激活，可以检查任意 task manager 的日志中是否包含 "Started the Queryable State Proxy Server @ ..."。 -## Making State Queryable +## 将 state 设置为可查询的 -Now that you have activated queryable state on your cluster, it is time to see how to use it. In order for a state to -be visible to the outside world, it needs to be explicitly made queryable by using: +激活集群的 queryable state 功能后，还要将 state 设置为可查询的才能对外可见，可以通过以下两种方式进行设置： -* either a `QueryableStateStream`, a convenience object which acts as a sink and offers its incoming values as queryable -state, or -* the `stateDescriptor.setQueryable(String queryableStateName)` method, which makes the keyed state represented by the - state descriptor, queryable. +* 创建 `QueryableStateStream`，它会作为一个 sink，并将输入数据转化为 queryable state； +* 通过 `stateDescriptor.setQueryable(String queryableStateName)` 将 state 描述符所表示的 keyed state 设置成可查询的。 -The following sections explain the use of these two approaches. +接下来的部分将详细解释这两种方式。 ### Queryable State Stream -Calling `.asQueryableState(stateName, stateDescriptor)` on a `KeyedStream` returns a `QueryableStateStream` which offers -its values as queryable state. Depending on the type of state, there are the following variants of the `asQueryableState()` -method: +在 `KeyedStream` 上调用 `.asQueryableState(stateName, stateDescriptor)` 将会返回一个 `QueryableStateStream`， 它会将流数据转化为 queryable state。 +对应不同的 state 类型，`asQueryableState()` 有以下一些方法变体： {% highlight java %} // ValueState @@ -119,28 +96,23 @@ QueryableStateStream asQueryableState(MemoryStateBackendorFsStateBackend, does not work - with copies when retrieving values but instead directly references the stored values, read-modify-write - patterns are unsafe and may cause the queryable state server to fail due to concurrent modifications. - TheRocksDBStateBackendis safe from these issues. + 注意: 进行查询时，state 会在并发线程中被访问，但 state 不会进行同步和拷贝。这种设计是为了避免同步和拷贝带来的作业延时。对于使用 Java 堆内存的 state backend， + 比如MemoryStateBackend或者FsStateBackend，它们获取状态时不会进行拷贝，而是直接引用状态对象，所以对状态的 read-modify-write 是不安全的，并且可能会因为并发修改导致查询失败。但RocksDBStateBackend是安全的，不会遇到上述问题。- Note: There is no queryable-The returned `QueryableStateStream` can be seen as a sink and **cannot** be further transformed. Internally, a -`QueryableStateStream` gets translated to an operator which uses all incoming records to update the queryable state -instance. The updating logic is implied by the type of the `StateDescriptor` provided in the `asQueryableState` call. -In a program like the following, all records of the keyed stream will be used to update the state instance via the -`ValueState.update(value)`: +返回的 `QueryableStateStream` 可以被视作一个sink，而且**不能再**被进一步转换。在内部实现上，一个 `QueryableStateStream` 被转换成一个 operator，使用输入的数据来更新 queryable state。state 如何更新是由 `asQueryableState` 提供的 `StateDescriptor` 来决定的。在下面的代码中, keyed stream 的所有数据将会通过 `ValueState.update(value)` 来更新状态： {% highlight java %} stream.keyBy(0).asQueryableState("query-name") {% endhighlight %} -This acts like the Scala API's `flatMapWithState`. +这个行为类似于 Scala API 中的 `flatMapWithState`。 ### Managed Keyed State -Managed keyed state of an operator -(see [Using Managed Keyed State]({{ site.baseurl }}/dev/stream/state/state.html#using-managed-keyed-state)) -can be made queryable by making the appropriate state descriptor queryable via -`StateDescriptor.setQueryable(String queryableStateName)`, as in the example below: +operator 中的 Managed keyed state +(参考 [Using Managed Keyed State]({{ site.baseurl }}/zh/dev/stream/state/state.html#using-managed-keyed-state)) +可以通过 `StateDescriptor.setQueryable(String queryableStateName)` 将 state descriptor 设置成可查询的，从而使 state 可查询，如下所示： + {% highlight java %} ValueStateDescriptorListStatesink as it would result in an ever-growing - list which may not be cleaned up and thus will eventually consume too much memory. + 注意: 没有可查询的ListStatesink，因为这种情况下 list 会不断增长，并且可能不会被清理，最终会消耗大量的内存。- Note: The-This variant has no limitations as to which type of state can be made queryable. This means that this can be used for -any `ValueState`, `ReduceState`, `ListState`, `MapState`, `AggregatingState`, and the currently deprecated `FoldingState`. +这种方式不会限制 state 类型，即任意的 `ValueState`、`ReduceState`、`ListState`、`MapState`、`AggregatingState` 以及已弃用的 `FoldingState` +均可作为 queryable state。 -## Querying State +## 查询 state -So far, you have set up your cluster to run with queryable state and you have declared (some of) your state as -queryable. Now it is time to see how to query this state. +目前为止，你已经激活了集群的 queryable state 功能，并且将一些 state 设置成了可查询的，接下来将会展示如何进行查询。 -For this you can use the `QueryableStateClient` helper class. This is available in the `flink-queryable-state-client` -jar which must be explicitly included as a dependency in the `pom.xml` of your project along with `flink-core`, as shown below: +为了进行查询，可以使用辅助类 `QueryableStateClient`，这个类位于 `flink-queryable-state-client` 的 jar 中，在项目的 `pom.xml` 需要显示添加对 `flink-queryable-state-client` 和 `flink-core` 的依赖, 如下所示：queryableStateNameparameter may be chosen arbitrarily and is only - used for queries. It does not have to be identical to the state's own name. + 注意: 参数queryableStateName可以任意选取，并且只被用来进行查询，它可以和 state 的名称不同。{% highlight xml %} @@ -180,18 +149,16 @@ jar which must be explicitly included as a dependency in the `pom.xml` of your p {% endhighlight %}-For more on this, you can check how to [set up a Flink program]({{ site.baseurl }}/dev/projectsetup/dependencies.html). +关于依赖的更多信息, 可以参考如何 [配置 Flink 项目]({{ site.baseurl }}/zh/dev/project-configuration.html). -The `QueryableStateClient` will submit your query to the internal proxy, which will then process your query and return -the final result. The only requirement to initialize the client is to provide a valid `TaskManager` hostname (remember -that there is a queryable state proxy running on each task manager) and the port where the proxy listens. More on how -to configure the proxy and state server port(s) in the [Configuration Section](#configuration). +`QueryableStateClient` 将提交你的请求到内部代理，代理会处理请求并返回结果。客户端的初始化只需要提供一个有效的 `TaskManager` 主机名 +(每个 task manager 上都运行着一个 queryable state 代理)，以及代理监听的端口号。关于如何配置代理以及端口号可以参考 [Configuration Section](#configuration). {% highlight java %} QueryableStateClient client = new QueryableStateClient(tmHostname, proxyPort); {% endhighlight %} -With the client ready, to query a state of type `V`, associated with a key of type `K`, you can use the method: +客户端就绪后，为了查询类型为 `K` 的 key，以及类型为 `V` 的state，可以使用如下方法： {% highlight java %} CompletableFuturegetKvState( @@ -202,35 +169,29 @@ CompletableFuturegetKvState( StateDescriptor- Note: These state objects do not allow modifications to the contained state. You can use them to get - the actual value of the state, e.g. usingvalueState.get(), or iterate over - the containedmapState.entries(), but you cannot - modify them. As an example, calling theadd()method on a returned list state will throw an -UnsupportedOperationException. + 注意： 这些 state 对象不允许对其中的 state 进行修改。你可以通过valueState.get()获取实际的 state， + 或者通过mapState.entries()遍历所有add()+ 方法将会导致UnsupportedOperationException。- Note: The client is asynchronous and can be shared by multiple threads. It needs - to be shutdown via-### Example +### 示例 -The following example extends the `CountWindowAverage` example -(see [Using Managed Keyed State]({{ site.baseurl }}/dev/stream/state/state.html#using-managed-keyed-state)) -by making it queryable and shows how to query this value: +下面的例子扩展自 `CountWindowAverage` +(参考 [Using Managed Keyed State]({{ site.baseurl }}/zh/dev/stream/state/state.html#using-managed-keyed-state))， +将其中的 state 设置成可查询的，并展示了如何进行查询： {% highlight java %} public class CountWindowAverage extends RichFlatMapFunctionQueryableStateClient.shutdown()when unused in order to free - resources. + 注意: 客户端是异步的，并且可能被多个线程共享。客户端不再使用后需要通过QueryableStateClient.shutdown()+ 来终止，从而释放资源。getReducingState(ReducingStateDescriptor )` * `ListState getListState(ListStateDescriptor )` * `AggregatingState +-*Raw State* 则保存在算子自己的数据结构中。checkpoint 的时候，Flink 并不知晓具体的内容，仅仅写入一串字节序列到 checkpoint。 +### Tuple Keys and Expression Keys +{:.no_toc} -所有 datastream 的 function 都可以使用 managed state, 但是 raw state 则只能在实现算子的时候使用。 -由于 Flink 可以在修改并发时更好的分发状态数据，并且能够更好的管理内存，因此建议使用 managed state（而不是 raw state）。 +Flink also has two alternative ways of defining keys: tuple keys and expression +keys. With this you can specify keys using tuple field indices or expressions +for selecting fields of objects. We don't recommend using these today but you +can refer to the Javadoc of DataStream to learn about them. Using a KeySelector +function is strictly superior: with Java lambdas they are easy to use and they +have potentially less overhead at runtime. -注意 如果你的 managed state 需要定制化的序列化逻辑， -为了后续的兼容性请参考 [相应指南](custom_serialization.html)，Flink 的默认序列化器不需要用户做特殊的处理。 +{% top %} -## 使用 Managed Keyed State +## 使用 Keyed State -managed keyed state 接口提供不同类型状态的访问接口，这些状态都作用于当前输入数据的 key 下。换句话说，这些状态仅可在 `KeyedStream` +keyed state 接口提供不同类型状态的访问接口，这些状态都作用于当前输入数据的 key 下。换句话说，这些状态仅可在 `KeyedStream` 上使用，可以通过 `stream.keyBy(...)` 得到 `KeyedStream`. 接下来，我们会介绍不同类型的状态，然后介绍如何使用他们。所有支持的状态类型如下所示： @@ -83,32 +107,25 @@ managed keyed state 接口提供不同类型状态的访问接口，这些状态 * `AggregatingState+{% highlight java %} +// some ordinary POJO +public class WC { + public String word; + public int count; -*Keyed State* 和 *Operator State* 分别有两种存在形式：*managed* and *raw*. + public String getWord() { return word; } +} +DataStream+words = // [...] +KeyedStream keyed = words + .keyBy(WC::getWord); +{% endhighlight %} -*Managed State* 由 Flink 运行时控制的数据结构表示，比如内部的 hash table 或者 RocksDB。 -比如 "ValueState", "ListState" 等。Flink runtime 会对这些状态进行编码并写入 checkpoint。 + +{% highlight scala %} +// some ordinary case class +case class WC(word: String, count: Int) +val words: DataStream[WC] = // [...] +val keyed = words.keyBy( _.word ) +{% endhighlight %} ++getState(ValueStateDescriptor )` * `ReducingState getReducingState(ReducingStateDescriptor )` * `ListState getListState(ListStateDescriptor )` * `AggregatingState ` 接口来使用 managed operator state。 +用户可以通过实现 `CheckpointedFunction` 接口来使用 operator state。 #### CheckpointedFunction @@ -487,13 +539,14 @@ void initializeState(FunctionInitializationContext context) throws Exception; 进行 checkpoint 时会调用 `snapshotState()`。 用户自定义函数初始化时会调用 `initializeState()`，初始化包括第一次自定义函数初始化和从之前的 checkpoint 恢复。 因此 `initializeState()` 不仅是定义不同状态类型初始化的地方，也需要包括状态恢复的逻辑。 -当前，managed operator state 以 list 的形式存在。这些状态是一个 *可序列化* 对象的集合 `List`，彼此独立，方便在改变并发后进行状态的重新分派。 +当前 operator state 以 list 的形式存在。这些状态是一个 *可序列化* 对象的集合 `List`，彼此独立，方便在改变并发后进行状态的重新分派。 换句话说，这些对象是重新分配 non-keyed state 的最细粒度。根据状态的不同访问方式，有如下几种重新分配的模式： - **Even-split redistribution:** 每个算子都保存一个列表形式的状态集合，整个状态由所有的列表拼接而成。当作业恢复或重新分配的时候，整个状态会按照算子的并发度进行均匀分配。 比如说，算子 A 的并发读为 1，包含两个元素 `element1` 和 `element2`，当并发读增加为 2 时，`element1` 会被分到并发 0 上，`element2` 则会被分到并发 1 上。 - **Union redistribution:** 每个算子保存一个列表形式的状态集合。整个状态由所有的列表拼接而成。当作业恢复或重新分配时，每个算子都将获得所有的状态数据。 + Do not use this feature if your list may have high cardinality. Checkpoint metadata will store an offset to each list entry, which could lead to RPC framesize or out-of-memory errors. 下面的例子中的 `SinkFunction` 在 `CheckpointedFunction` 中进行数据缓存，然后统一发送到下游，这个例子演示了列表状态数据的 event-split redistribution。 diff --git a/docs/dev/table/catalogs.md b/docs/dev/table/catalogs.md index 961d0a68af791f..4ff391791b5f09 100644 --- a/docs/dev/table/catalogs.md +++ b/docs/dev/table/catalogs.md @@ -43,94 +43,8 @@ The `GenericInMemoryCatalog` is an in-memory implementation of a catalog. All ob ### JdbcCatalog -The `JdbcCatalog` enables users to connect Flink to relational databases over JDBC protocol. - -#### PostgresCatalog - -`PostgresCatalog` is the only implementation of JDBC Catalog at the moment. - -#### Usage of JdbcCatalog - -Set a `JdbcCatalog` with the following parameters: - -- name: required, name of the catalog -- default database: required, default database to connect to -- username: required, username of Postgres account -- password: required, password of the account -- base url: required, should be of format "jdbc:postgresql:// : ", and should not contain database name here - - -- +The `JdbcCatalog` enables users to connect Flink to relational databases over JDBC protocol. `PostgresCatalog` is the only implementation of JDBC Catalog at the moment. +See [JdbcCatalog documentation]({% link dev/table/connectors/jdbc.md %}) for more details on setting up the catalog. ### HiveCatalog @@ -154,8 +68,6 @@ The set of properties will be passed to a discovery service where the service tr Users can use SQL DDL to create tables in catalogs in both Table API and SQL. -For Table API: --{% highlight java %} - -EnvironmentSettings settings = EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build(); -TableEnvironment tableEnv = TableEnvironment.create(settings); - -String name = "mypg"; -String defaultDatabase = "mydb"; -String username = "..."; -String password = "..."; -String baseUrl = "..." - -JdbcCatalog catalog = new JdbcCatalog(name, defaultDatabase, username, password, baseUrl); -tableEnv.registerCatalog("mypg", catalog); - -// set the JdbcCatalog as the current catalog of the session -tableEnv.useCatalog("mypg"); -{% endhighlight %} ---{% highlight scala %} - -val settings = EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build() -val tableEnv = TableEnvironment.create(settings) - -val name = "mypg" -val defaultDatabase = "mydb" -val username = "..." -val password = "..." -val baseUrl = "..." - -val catalog = new JdbcCatalog(name, defaultDatabase, username, password, baseUrl) -tableEnv.registerCatalog("mypg", catalog) - -// set the JdbcCatalog as the current catalog of the session -tableEnv.useCatalog("mypg") -{% endhighlight %} ---{% highlight sql %} -CREATE CATALOG mypg WITH( - 'type'='jdbc', - 'default-database'='...', - 'username'='...', - 'password'='...', - 'base-url'='...' -); - -USE CATALOG mypg; -{% endhighlight %} ---{% highlight yaml %} - -execution: - planner: blink - ... - current-catalog: mypg # set the JdbcCatalog as the current catalog of the session - current-database: mydb - -catalogs: - - name: mypg - type: jdbc - default-database: mydb - username: ... - password: ... - base-url: ... -{% endhighlight %} --+ For detailed information, please check out [Flink SQL CREATE DDL]({{ site.baseurl }}/dev/table/sql/create.html). -### Using Java/Scala/Python API +### Using Java/Scala -Users can use Java, Scala, or Python API to create catalog tables programmatically. +Users can use Java or Scala to create catalog tables programmatically.{% highlight java %} @@ -168,19 +80,36 @@ Catalog catalog = new HiveCatalog("myhive", null, "+", " - +{% highlight scala %} +val tableEnv = ... -For SQL Client: +// Create a HiveCatalog +val catalog = new HiveCatalog("myhive", null, "+", " ") + +// Register the catalog +tableEnv.registerCatalog("myhive", catalog) + +// Create a catalog database +tableEnv.executeSql("CREATE DATABASE mydb WITH (...)") +// Create a catalog table +tableEnv.executeSql("CREATE TABLE mytable (name STRING, age INT) WITH (...)") + +tableEnv.listTables() // should return the tables in current catalog and database. + +{% endhighlight %} + {% highlight sql %} // the catalog should have been registered via yaml file Flink SQL> CREATE DATABASE mydb WITH (...); @@ -190,17 +119,25 @@ Flink SQL> CREATE TABLE mytable (name STRING, age INT) WITH (...); Flink SQL> SHOW TABLES; mytable {% endhighlight %} ++@@ -244,7 +224,7 @@ For detailed DDL information, please refer to [SQL CREATE DDL]({{ site.baseurl } ### Database operations{% highlight java %} -TableEnvironment tableEnv = ... +import org.apache.flink.table.api.*; +import org.apache.flink.table.catalog.*; +import org.apache.flink.table.catalog.hive.HiveCatalog; +import org.apache.flink.table.descriptors.Kafka; + +TableEnvironment tableEnv = TableEnvironment.create(EnvironmentSettings.newInstance().build()); // Create a HiveCatalog Catalog catalog = new HiveCatalog("myhive", null, "+", " "); @@ -209,7 +146,7 @@ Catalog catalog = new HiveCatalog("myhive", null, " ", " tables = catalog.listTables("mydb"); // tables should contain "mytable" {% endhighlight %} + +{% highlight scala %} +import org.apache.flink.table.api._ +import org.apache.flink.table.catalog._ +import org.apache.flink.table.catalog.hive.HiveCatalog +import org.apache.flink.table.descriptors.Kafka + +val tableEnv = TableEnvironment.create(EnvironmentSettings.newInstance.build) + +// Create a HiveCatalog +val catalog = new HiveCatalog("myhive", null, "", " ") + +// Register the catalog +tableEnv.registerCatalog("myhive", catalog) + +// Create a catalog database +catalog.createDatabase("mydb", new CatalogDatabaseImpl(...)) + +// Create a catalog table +val schema = TableSchema.builder() + .field("name", DataTypes.STRING()) + .field("age", DataTypes.INT()) + .build() + +catalog.createTable( + new ObjectPath("mydb", "mytable"), + new CatalogTableImpl( + schema, + new Kafka() + .version("0.11") + .... + .startFromEarlist() + .toProperties(), + "my comment" + ), + false + ) + +val tables = catalog.listTables("mydb") // tables should contain "mytable" +{% endhighlight %} -+{% highlight java %} // create database catalog.createDatabase("mydb", new CatalogDatabaseImpl(...), false); @@ -270,7 +250,7 @@ catalog.listDatabases("mycatalog"); ### Table operations-+{% highlight java %} // create table catalog.createTable(new ObjectPath("mydb", "mytable"), new CatalogTableImpl(...), false); @@ -299,7 +279,7 @@ catalog.listTables("mydb"); ### View operations--+{% highlight java %} // create view catalog.createTable(new ObjectPath("mydb", "myview"), new CatalogViewImpl(...), false); @@ -329,7 +309,7 @@ catalog.listViews("mydb"); ### Partition operations@@ -808,39 +806,42 @@ result.insert_into("CsvSinkTable") 两种计划器翻译和执行查询的方式是不同的。-+@@ -778,7 +777,7 @@ result.insertInto("CsvSinkTable") table_env = ... # see "Create a TableEnvironment" section # create a TableSink -t_env.connect(FileSystem().path("/path/to/file"))) +table_env.connect(FileSystem().path("/path/to/file"))) .with_format(Csv() .field_delimiter(',') .deriveSchema()) @@ -792,9 +791,8 @@ t_env.connect(FileSystem().path("/path/to/file"))) result = ... # emit the result Table to the registered TableSink -result.insert_into("CsvSinkTable") +result.execute_insert("CsvSinkTable") -# execute the program {% endhighlight %}{% highlight java %} // create view catalog.createPartition( @@ -370,7 +350,7 @@ catalog.listPartitions(new ObjectPath("mydb", "mytable"), Arrays.asList(epr1, .. ### Function operations@@ -766,9 +766,8 @@ tableEnv.connect(new FileSystem("/path/to/file")) val result: Table = ... // emit the result Table to the registered TableSink -result.insertInto("CsvSinkTable") +result.executeInsert("CsvSinkTable") -// execute the program {% endhighlight %}-@@ -716,7 +717,7 @@ Table API 和 SQL 查询的混用非常简单因为它们都返回 `Table` 对 请参考文档 [Table Sources & Sinks]({{ site.baseurl }}/zh/dev/table/sourceSinks.html) 以获取更多关于可用 Sink 的信息以及如何自定义 `TableSink`。 -方法 `Table.insertInto(String tableName)` 将 `Table` 发送至已注册的 `TableSink`。该方法通过名称在 catalog 中查找 `TableSink` 并确认`Table` schema 和 `TableSink` schema 一致。 +方法 `Table.executeInsert(String tableName)` 将 `Table` 发送至已注册的 `TableSink`。该方法通过名称在 catalog 中查找 `TableSink` 并确认`Table` schema 和 `TableSink` schema 一致。 下面的示例演示如何输出 `Table`： @@ -740,9 +741,8 @@ tableEnv.connect(new FileSystem("/path/to/file")) // compute a result Table using Table API operators and/or SQL queries Table result = ... // emit the result Table to the registered TableSink -result.insertInto("CsvSinkTable"); +result.executeInsert("CsvSinkTable"); -// execute the program {% endhighlight %}+{% highlight java %} // create function catalog.createFunction(new ObjectPath("mydb", "myfunc"), new CatalogFunctionImpl(...), false); diff --git a/docs/dev/table/catalogs.zh.md b/docs/dev/table/catalogs.zh.md index 23e0969672c8ac..5bd8e5630d7c73 100644 --- a/docs/dev/table/catalogs.zh.md +++ b/docs/dev/table/catalogs.zh.md @@ -39,94 +39,8 @@ Catalog 提供了元数据信息，例如数据库、表、分区、视图以及 ### JdbcCatalog -The `JdbcCatalog` enables users to connect Flink to relational databases over JDBC protocol. - -#### PostgresCatalog - -`PostgresCatalog` is the only implementation of JDBC Catalog at the moment. - -#### Usage of JdbcCatalog - -Set a `Jdbcatalog` with the following parameters: - -- name: required, name of the catalog -- default database: required, default database to connect to -- username: required, username of Postgres account -- password: required, password of the account -- base url: required, should be of format "jdbc:postgresql://-: ", and should not contain database name here - - -- +`JdbcCatalog` 使得用户可以将 Flink 通过 JDBC 协议连接到关系数据库。`PostgresCatalog` 是当前实现的唯一一种 JDBC Catalog。 +参考 [JdbcCatalog 文档]({% link dev/table/connectors/jdbc.zh.md %}) 获取关于配置 JDBC catalog 的详细信息。 ### HiveCatalog @@ -150,8 +64,6 @@ Catalog 是可扩展的，用户可以通过实现 `Catalog` 接口来开发自 用户可以使用 DDL 通过 Table API 或者 SQL Client 在 Catalog 中创建表。 -使用 Table API： --{% highlight java %} - -EnvironmentSettings settings = EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build(); -TableEnvironment tableEnv = TableEnvironment.create(settings); - -String name = "mypg"; -String defaultDatabase = "mydb"; -String username = "..."; -String password = "..."; -String baseUrl = "..." - -JdbcCatalog catalog = new JdbcCatalog(name, defaultDatabase, username, password, baseUrl); -tableEnv.registerCatalog("mypg", catalog); - -// set the JdbcCatalog as the current catalog of the session -tableEnv.useCatalog("mypg"); -{% endhighlight %} ---{% highlight scala %} - -val settings = EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build() -val tableEnv = TableEnvironment.create(settings) - -val name = "mypg" -val defaultDatabase = "mydb" -val username = "..." -val password = "..." -val baseUrl = "..." - -val catalog = new JdbcCatalog(name, defaultDatabase, username, password, baseUrl) -tableEnv.registerCatalog("mypg", catalog) - -// set the JdbcCatalog as the current catalog of the session -tableEnv.useCatalog("mypg") -{% endhighlight %} ---{% highlight sql %} -CREATE CATALOG mypg WITH( - 'type'='jdbc', - 'default-database'='...', - 'username'='...', - 'password'='...', - 'base-url'='...' -); - -USE CATALOG mypg; -{% endhighlight %} ---{% highlight yaml %} - -execution: - planner: blink - ... - current-catalog: mypg # set the JdbcCatalog as the current catalog of the session - current-database: mydb - -catalogs: - - name: mypg - type: jdbc - default-database: mydb - username: ... - password: ... - base-url: ... -{% endhighlight %} --+ 更多详细信息，请参考[Flink SQL CREATE DDL]({{ site.baseurl }}/zh/dev/table/sql/create.html)。 -### 使用 Java/Scala/Python API +### 使用 Java/Scala -用户可以用编程的方式使用Java、Scala 或者 Python API 来创建 Catalog 表。 +用户可以用编程的方式使用Java 或者 Scala 来创建 Catalog 表。{% highlight java %} @@ -164,19 +76,36 @@ Catalog catalog = new HiveCatalog("myhive", null, "+", " - +{% highlight scala %} +val tableEnv = ... + +// Create a HiveCatalog +val catalog = new HiveCatalog("myhive", null, "+", " "); -使用 SQL Client： +// Register the catalog +tableEnv.registerCatalog("myhive", catalog); +// Create a catalog database +tableEnv.executeSql("CREATE DATABASE mydb WITH (...)"); + +// Create a catalog table +tableEnv.executeSql("CREATE TABLE mytable (name STRING, age INT) WITH (...)"); + +tableEnv.listTables(); // should return the tables in current catalog and database. + +{% endhighlight %} + {% highlight sql %} // the catalog should have been registered via yaml file Flink SQL> CREATE DATABASE mydb WITH (...); @@ -186,17 +115,25 @@ Flink SQL> CREATE TABLE mytable (name STRING, age INT) WITH (...); Flink SQL> SHOW TABLES; mytable {% endhighlight %} ++@@ -240,7 +221,7 @@ List{% highlight java %} -TableEnvironment tableEnv = ... +import org.apache.flink.table.api.*; +import org.apache.flink.table.catalog.*; +import org.apache.flink.table.catalog.hive.HiveCatalog; +import org.apache.flink.table.descriptors.Kafka; + +TableEnvironment tableEnv = TableEnvironment.create(EnvironmentSettings.newInstance().build()); // Create a HiveCatalog Catalog catalog = new HiveCatalog("myhive", null, "+", " "); @@ -205,7 +142,7 @@ Catalog catalog = new HiveCatalog("myhive", null, " ", " tables = catalog.listTables("mydb"); // tables should contain "mytable" {% endhighlight %} + +{% highlight scala %} +import org.apache.flink.table.api._ +import org.apache.flink.table.catalog._ +import org.apache.flink.table.catalog.hive.HiveCatalog +import org.apache.flink.table.descriptors.Kafka + +val tableEnv = TableEnvironment.create(EnvironmentSettings.newInstance.build) + +// Create a HiveCatalog +val catalog = new HiveCatalog("myhive", null, "", " ") + +// Register the catalog +tableEnv.registerCatalog("myhive", catalog) + +// Create a catalog database +catalog.createDatabase("mydb", new CatalogDatabaseImpl(...)) + +// Create a catalog table +val schema = TableSchema.builder() + .field("name", DataTypes.STRING()) + .field("age", DataTypes.INT()) + .build() + +catalog.createTable( + new ObjectPath("mydb", "mytable"), + new CatalogTableImpl( + schema, + new Kafka() + .version("0.11") + .... + .startFromEarlist() + .toProperties(), + "my comment" + ), + false + ) + +val tables = catalog.listTables("mydb") // tables should contain "mytable" +{% endhighlight %} + tables = catalog.listTables("mydb"); // tables should contain "myta ### 数据库操作 -++Attention **Once the Table is converted to a DataSet, we must use the ExecutionEnvironment.execute method to execute the DataSet program.** + {% top %} ### Mapping of Data Types to Table Schema @@ -1405,16 +1402,7 @@ Query Optimization ------------------{% highlight java %} // create database catalog.createDatabase("mydb", new CatalogDatabaseImpl(...), false); @@ -266,7 +247,7 @@ catalog.listDatabases("mycatalog"); ### 表操作-@@ -829,40 +819,43 @@ Translate and Execute a Query The behavior of translating and executing a query is different for the two planners.+{% highlight java %} // create table catalog.createTable(new ObjectPath("mydb", "mytable"), new CatalogTableImpl(...), false); @@ -295,7 +276,7 @@ catalog.listTables("mydb"); ### 视图操作@@ -799,7 +790,7 @@ result.insertInto("CsvSinkTable") table_env = ... # see "Create a TableEnvironment" section # create a TableSink -t_env.connect(FileSystem().path("/path/to/file"))) +table_env.connect(FileSystem().path("/path/to/file"))) .with_format(Csv() .field_delimiter(',') .deriveSchema()) @@ -813,9 +804,8 @@ t_env.connect(FileSystem().path("/path/to/file"))) result = ... # emit the result Table to the registered TableSink -result.insert_into("CsvSinkTable") +result.execute_insert("CsvSinkTable") -# execute the program {% endhighlight %}-@@ -787,9 +779,8 @@ tableEnv.connect(new FileSystem("/path/to/file")) val result: Table = ... // emit the result Table to the registered TableSink -result.insertInto("CsvSinkTable") +result.executeInsert("CsvSinkTable") -// execute the program {% endhighlight %}+@@ -737,7 +729,7 @@ A batch `Table` can only be written to a `BatchTableSink`, while a streaming `Ta Please see the documentation about [Table Sources & Sinks]({{ site.baseurl }}/dev/table/sourceSinks.html) for details about available sinks and instructions for how to implement a custom `TableSink`. -The `Table.insertInto(String tableName)` method emits the `Table` to a registered `TableSink`. The method looks up the `TableSink` from the catalog by the name and validates that the schema of the `Table` is identical to the schema of the `TableSink`. +The `Table.executeInsert(String tableName)` method emits the `Table` to a registered `TableSink`. The method looks up the `TableSink` from the catalog by the name and validates that the schema of the `Table` is identical to the schema of the `TableSink`. The following examples shows how to emit a `Table`: @@ -760,10 +752,10 @@ tableEnv.connect(new FileSystem("/path/to/file")) // compute a result Table using Table API operators and/or SQL queries Table result = ... + // emit the result Table to the registered TableSink -result.insertInto("CsvSinkTable"); +result.executeInsert("CsvSinkTable"); -// execute the program {% endhighlight %}{% highlight java %} // create view catalog.createTable(new ObjectPath("mydb", "myview"), new CatalogViewImpl(...), false); @@ -325,7 +306,7 @@ catalog.listViews("mydb"); ### 分区操作-@@ -704,7 +697,7 @@ table_env = ... # see "Create a TableEnvironment" section # register "RevenueFrance" output table # compute revenue for all customers from France and emit to "RevenueFrance" -table_env.sql_update( +table_env.execute_sql( "INSERT INTO RevenueFrance " "SELECT cID, cName, SUM(revenue) AS revSum " "FROM Orders " @@ -712,7 +705,6 @@ table_env.sql_update( "GROUP BY cID, cName" ) -# execute query {% endhighlight %}+{% highlight java %} // create view catalog.createPartition( @@ -366,7 +347,7 @@ catalog.listPartitions(new ObjectPath("mydb", "mytable"), Arrays.asList(epr1, .. ### 函数操作@@ -552,7 +545,9 @@ val revenue = orders // execute query {% endhighlight %} -**Note:** The Scala Table API uses Scala Symbols, which start with a single tick (`'`) to reference the attributes of a `Table`. The Table API uses Scala implicits. Make sure to import `org.apache.flink.api.scala._` and `org.apache.flink.table.api.scala._` in order to use Scala implicit conversions. +**Note:** The Scala Table API uses Scala String interpolation that starts with a dollar sign (`$`) to reference the attributes of a `Table`. The Table API uses Scala implicits. Make sure to import +* `org.apache.flink.table.api._` - for implicit expression conversions +* `org.apache.flink.api.scala._` and `org.apache.flink.table.api.bridge.scala._` if you want to convert from/to DataStream.-@@ -157,7 +150,7 @@ If both planner jars are on the classpath (the default behavior), you should exp // ********************** import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; import org.apache.flink.table.api.EnvironmentSettings; -import org.apache.flink.table.api.java.StreamTableEnvironment; +import org.apache.flink.table.api.bridge.java.StreamTableEnvironment; EnvironmentSettings fsSettings = EnvironmentSettings.newInstance().useOldPlanner().inStreamingMode().build(); StreamExecutionEnvironment fsEnv = StreamExecutionEnvironment.getExecutionEnvironment(); @@ -168,7 +161,7 @@ StreamTableEnvironment fsTableEnv = StreamTableEnvironment.create(fsEnv, fsSetti // FLINK BATCH QUERY // ****************** import org.apache.flink.api.java.ExecutionEnvironment; -import org.apache.flink.table.api.java.BatchTableEnvironment; +import org.apache.flink.table.api.bridge.java.BatchTableEnvironment; ExecutionEnvironment fbEnv = ExecutionEnvironment.getExecutionEnvironment(); BatchTableEnvironment fbTableEnv = BatchTableEnvironment.create(fbEnv); @@ -178,7 +171,7 @@ BatchTableEnvironment fbTableEnv = BatchTableEnvironment.create(fbEnv); // ********************** import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; import org.apache.flink.table.api.EnvironmentSettings; -import org.apache.flink.table.api.java.StreamTableEnvironment; +import org.apache.flink.table.api.bridge.java.StreamTableEnvironment; StreamExecutionEnvironment bsEnv = StreamExecutionEnvironment.getExecutionEnvironment(); EnvironmentSettings bsSettings = EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build(); @@ -205,7 +198,7 @@ TableEnvironment bbTableEnv = TableEnvironment.create(bbSettings); // ********************** import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment import org.apache.flink.table.api.EnvironmentSettings -import org.apache.flink.table.api.scala.StreamTableEnvironment +import org.apache.flink.table.api.bridge.scala.StreamTableEnvironment val fsSettings = EnvironmentSettings.newInstance().useOldPlanner().inStreamingMode().build() val fsEnv = StreamExecutionEnvironment.getExecutionEnvironment @@ -216,7 +209,7 @@ val fsTableEnv = StreamTableEnvironment.create(fsEnv, fsSettings) // FLINK BATCH QUERY // ****************** import org.apache.flink.api.scala.ExecutionEnvironment -import org.apache.flink.table.api.scala.BatchTableEnvironment +import org.apache.flink.table.api.bridge.scala.BatchTableEnvironment val fbEnv = ExecutionEnvironment.getExecutionEnvironment val fbTableEnv = BatchTableEnvironment.create(fbEnv) @@ -226,7 +219,7 @@ val fbTableEnv = BatchTableEnvironment.create(fbEnv) // ********************** import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment import org.apache.flink.table.api.EnvironmentSettings -import org.apache.flink.table.api.scala.StreamTableEnvironment +import org.apache.flink.table.api.bridge.scala.StreamTableEnvironment val bsEnv = StreamExecutionEnvironment.getExecutionEnvironment val bsSettings = EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build() @@ -426,7 +419,7 @@ table_environment \+@@ -114,10 +109,8 @@ tapi_result = table_env.from_path("table1").select(...) sql_result = table_env.sql_query("SELECT ... FROM table1 ...") # emit a Table API result Table to a TableSink, same for SQL result -tapi_result.insert_into("outputTable") - -# execute -table_env.execute("python_job") +table_result = tapi_result.execute_insert("outputTable") +table_result... {% endhighlight %}{% highlight java %} // create function catalog.createFunction(new ObjectPath("mydb", "myfunc"), new CatalogFunctionImpl(...), false); diff --git a/docs/dev/table/common.md b/docs/dev/table/common.md index 907cb15ac1013f..e8a1861549b808 100644 --- a/docs/dev/table/common.md +++ b/docs/dev/table/common.md @@ -32,12 +32,11 @@ Main Differences Between the Two Planners 1. Blink treats batch jobs as a special case of streaming. As such, the conversion between Table and DataSet is also not supported, and batch jobs will not be translated into `DateSet` programs but translated into `DataStream` programs, the same as the streaming jobs. 2. The Blink planner does not support `BatchTableSource`, use bounded `StreamTableSource` instead of it. -3. The Blink planner only support the brand new `Catalog` and does not support `ExternalCatalog` which is deprecated. -4. The implementations of `FilterableTableSource` for the old planner and the Blink planner are incompatible. The old planner will push down `PlannerExpression`s into `FilterableTableSource`, while the Blink planner will push down `Expression`s. -5. String based key-value config options (Please see the documentation about [Configuration]({{ site.baseurl }}/dev/table/config.html) for details) are only used for the Blink planner. -6. The implementation(`CalciteConfig`) of `PlannerConfig` in two planners is different. -7. The Blink planner will optimize multiple-sinks into one DAG (supported only on `TableEnvironment`, not on `StreamTableEnvironment`). The old planner will always optimize each sink into a new DAG, where all DAGs are independent of each other. -8. The old planner does not support catalog statistics now, while the Blink planner does. +3. The implementations of `FilterableTableSource` for the old planner and the Blink planner are incompatible. The old planner will push down `PlannerExpression`s into `FilterableTableSource`, while the Blink planner will push down `Expression`s. +4. String based key-value config options (Please see the documentation about [Configuration]({{ site.baseurl }}/dev/table/config.html) for details) are only used for the Blink planner. +5. The implementation(`CalciteConfig`) of `PlannerConfig` in two planners is different. +6. The Blink planner will optimize multiple-sinks into one DAG on both `TableEnvironment` and `StreamTableEnvironment`. The old planner will always optimize each sink into a new DAG, where all DAGs are independent of each other. +7. The old planner does not support catalog statistics now, while the Blink planner does. Structure of Table API and SQL Programs @@ -63,10 +62,8 @@ Table tapiResult = tableEnv.from("table1").select(...); Table sqlResult = tableEnv.sqlQuery("SELECT ... FROM table1 ... "); // emit a Table API result Table to a TableSink, same for SQL result -tapiResult.insertInto("outputTable"); - -// execute -tableEnv.execute("java_job"); +TableResult tableResult = tapiResult.executeInsert("outputTable"); +tableResult... {% endhighlight %}@@ -88,10 +85,8 @@ val tapiResult = tableEnv.from("table1").select(...) val sqlResult = tableEnv.sqlQuery("SELECT ... FROM table1 ...") // emit a Table API result Table to a TableSink, same for SQL result -tapiResult.insertInto("outputTable") - -// execute -tableEnv.execute("scala_job") +val tableResult = tapiResult.executeInsert("outputTable") +tableResult... {% endhighlight %}{% highlight sql %} -tableEnvironment.sqlUpdate("CREATE [TEMPORARY] TABLE MyTable (...) WITH (...)") +tableEnvironment.executeSql("CREATE [TEMPORARY] TABLE MyTable (...) WITH (...)") {% endhighlight %}@@ -661,7 +656,7 @@ TableEnvironment tableEnv = ...; // see "Create a TableEnvironment" section // register "RevenueFrance" output table // compute revenue for all customers from France and emit to "RevenueFrance" -tableEnv.sqlUpdate( +tableEnv.executeSql( "INSERT INTO RevenueFrance " + "SELECT cID, cName, SUM(revenue) AS revSum " + "FROM Orders " + @@ -669,7 +664,6 @@ tableEnv.sqlUpdate( "GROUP BY cID, cName" ); -// execute query {% endhighlight %}@@ -682,7 +676,7 @@ val tableEnv = ... // see "Create a TableEnvironment" section // register "RevenueFrance" output table // compute revenue for all customers from France and emit to "RevenueFrance" -tableEnv.sqlUpdate(""" +tableEnv.executeSql(""" |INSERT INTO RevenueFrance |SELECT cID, cName, SUM(revenue) AS revSum |FROM Orders @@ -690,7 +684,6 @@ tableEnv.sqlUpdate(""" |GROUP BY cID, cName """.stripMargin) -// execute query {% endhighlight %}--**Note:** A detailed discussion about dynamic tables and their properties is given in the [Dynamic Tables](streaming/dynamic_tables.html) document. +**Note:** A detailed discussion about dynamic tables and their properties is given in the [Dynamic Tables](streaming/dynamic_tables.html) document. + +Attention **Once the Table is converted to a DataStream, please use the `StreamExecutionEnvironment.execute()` method to execute the DataStream program.** #### Convert a Table into a DataSet @@ -1084,6 +1079,8 @@ val dsTuple: DataSet[(String, Int)] = tableEnv.toDataSet[(String, Int)](table)-Table API and SQL queries are translated into [DataStream]({{ site.baseurl }}/dev/datastream_api.html) or [DataSet]({{ site.baseurl }}/dev/batch) programs depending on whether their input is a streaming or batch input. A query is internally represented as a logical query plan and is translated in two phases: - -1. Optimization of the logical plan -2. Translation into a DataStream or DataSet program -For streaming, a Table API or SQL query is translated when: ++Table API and SQL queries are translated into [DataStream]({{ site.baseurl }}/dev/datastream_api.html) programs whether their input is streaming or batch. A query is internally represented as a logical query plan and is translated in two phases: -* `TableEnvironment.execute()` is called. A `Table` (emitted to a `TableSink` through `Table.insertInto()`) or a SQL update query (specified through `TableEnvironment.sqlUpdate()`) will be buffered in `TableEnvironment` first. Each sink will be optimized independently. The execution graph contains multiple independent sub-DAGs. -* A `Table` is translated when it is converted into a `DataStream` (see [Integration with DataStream and DataSet API](#integration-with-datastream-and-dataset-api)). Once translated, it's a regular DataStream program and is executed when `StreamExecutionEnvironment.execute()` is called. +1. Optimization of the logical plan, +2. Translation into a DataStream program. -For batch, a Table API or SQL query is translated when: +a Table API or SQL query is translated when: -* a `Table` is emitted to a `TableSink`, i.e., when `Table.insertInto()` is called. -* a SQL update query is specified, i.e., when `TableEnvironment.sqlUpdate()` is called. -* a `Table` is converted into a `DataSet` (see [Integration with DataStream and DataSet API](#integration-with-datastream-and-dataset-api)). +* `TableEnvironment.executeSql()` is called. This method is used for executing a given statement, and the sql query is translated immediately once this method is called. +* `Table.executeInsert()` is called. This method is used for inserting the table content to the given sink path, and the Table API is translated immediately once this method is called. +* `Table.execute()` is called. This method is used for collecting the table content to local client, and the Table API is translated immediately once this method is called. +* `StatementSet.execute()` is called. A `Table` (emitted to a sink through `StatementSet.addInsert()`) or an INSERT statement (specified through `StatementSet.addInsertSql()`) will be buffered in `StatementSet` first. They are translated once `StatementSet.execute()` is called. All sinks will be optimized into one DAG. +* A `Table` is translated when it is converted into a `DataStream` (see [Integration with DataStream and DataSet API](#integration-with-datastream-and-dataset-api)). Once translated, it's a regular DataStream program and is executed when `StreamExecutionEnvironment.execute()` is called. -Once translated, a Table API or SQL query is handled like a regular DataSet program and is executed when `ExecutionEnvironment.execute()` is called. +Attention **Since 1.11 version, `sqlUpdate()` method and `insertInto()` method are deprecated. If the Table program is built from these two methods, we must use `StreamTableEnvironment.execute()` method instead of `StreamExecutionEnvironment.execute()` method to execute it.**--Table API and SQL queries are translated into [DataStream]({{ site.baseurl }}/dev/datastream_api.html) programs whether their input is streaming or batch. A query is internally represented as a logical query plan and is translated in two phases: +{% top %} @@ -879,7 +872,7 @@ This interaction can be achieved by converting a `DataStream` or `DataSet` into ### Implicit Conversion for Scala -The Scala Table API features implicit conversions for the `DataSet`, `DataStream`, and `Table` classes. These conversions are enabled by importing the package `org.apache.flink.table.api.scala._` in addition to `org.apache.flink.api.scala._` for the Scala DataStream API. +The Scala Table API features implicit conversions for the `DataSet`, `DataStream`, and `Table` classes. These conversions are enabled by importing the package `org.apache.flink.table.api.bridge.scala._` in addition to `org.apache.flink.api.scala._` for the Scala DataStream API. ### Create a View from a DataStream or DataSet @@ -1039,7 +1032,9 @@ val retractStream: DataStream[(Boolean, Row)] = tableEnv.toRetractStream[Row](ta+Table API and SQL queries are translated into [DataStream]({{ site.baseurl }}/dev/datastream_api.html) or [DataSet]({{ site.baseurl }}/dev/batch) programs depending on whether their input is a streaming or batch input. A query is internally represented as a logical query plan and is translated in two phases: -1. Optimization of the logical plan, -2. Translation into a DataStream program. +1. Optimization of the logical plan +2. Translation into a DataStream or DataSet program -a Table API or SQL query is translated when: +A Table API or SQL query is translated when: -* `TableEnvironment.execute()` is called. A `Table` (emitted to a `TableSink` through `Table.insertInto()`) or a SQL update query (specified through `TableEnvironment.sqlUpdate()`) will be buffered in `TableEnvironment` first. All sinks will be optimized into one DAG. -* A `Table` is translated when it is converted into a `DataStream` (see [Integration with DataStream and DataSet API](#integration-with-datastream-and-dataset-api)). Once translated, it's a regular DataStream program and is executed when `StreamExecutionEnvironment.execute()` is called. +* `TableEnvironment.executeSql()` is called. This method is used for executing a given statement, and the sql query is translated immediately once this method is called. +* `Table.executeInsert()` is called. This method is used for inserting the table content to the given sink path, and the Table API is translated immediately once this method is called. +* `Table.execute()` is called. This method is used for collecting the table content to local client, and the Table API is translated immediately once this method is called. +* `StatementSet.execute()` is called. A `Table` (emitted to a sink through `StatementSet.addInsert()`) or an INSERT statement (specified through `StatementSet.addInsertSql()`) will be buffered in `StatementSet` first. They are translated once `StatementSet.execute()` is called. Each sink will be optimized independently. The execution graph contains multiple independent sub-DAGs. +* For streaming, a `Table` is translated when it is converted into a `DataStream` (see [Integration with DataStream and DataSet API](#integration-with-datastream-and-dataset-api)). Once translated, it's a regular DataStream program and is executed when `StreamExecutionEnvironment.execute()` is called. For batch, a `Table` is translated when it is converted into a `DataSet` (see [Integration with DataStream and DataSet API](#integration-with-datastream-and-dataset-api)). Once translated, it's a regular DataSet program and is executed when `ExecutionEnvironment.execute()` is called. +Attention **Since 1.11 version, `sqlUpdate()` method and `insertInto()` method are deprecated. For streaming, if the Table program is built from these two methods, we must use `StreamTableEnvironment.execute()` method instead of `StreamExecutionEnvironment.execute()` method to execute it. For batch, if the Table program is built from these two methods, we must use `BatchTableEnvironment.execute()` method instead of `ExecutionEnvironment.execute()` method to execute it.**+--### Explaining a Table +Explaining a Table +------------------ The Table API provides a mechanism to explain the logical and optimized query plans to compute a `Table`. -This is done through the `TableEnvironment.explain(table)` method or `TableEnvironment.explain()` method. `explain(table)` returns the plan of a given `Table`. `explain()` returns the result of a multiple sinks plan and is mainly used for the Blink planner. It returns a String describing three plans: +This is done through the `Table.explain()` method or `StatementSet.explain()` method. `Table.explain()`returns the plan of a `Table`. `StatementSet.explain()` returns the plan of multiple sinks. It returns a String describing three plans: 1. the Abstract Syntax Tree of the relational query, i.e., the unoptimized logical query plan, 2. the optimized logical query plan, and 3. the physical execution plan. -The following code shows an example and the corresponding output for given `Table` using `explain(table)`: +`TableEnvironment.explainSql()` and `TableEnvironment.executeSql()` support execute a `EXPLAIN` statement to get the plans, Please refer to [EXPLAIN]({{ site.baseurl }}/dev/table/sql/explain.html) page. + +The following code shows an example and the corresponding output for given `Table` using `Table.explain()` method:- -Apache Flink leverages Apache Calcite to optimize and translate queries. The optimization currently performed include projection and filter push-down, subquery decorrelation, and other kinds of query rewriting. Old planner does not yet optimize the order of joins, but executes them in the same order as defined in the query (order of Tables in the `FROM` clause and/or order of join predicates in the `WHERE` clause). - -It is possible to tweak the set of optimization rules which are applied in different phases by providing a `CalciteConfig` object. This can be created via a builder by calling `CalciteConfig.createBuilder())` and is provided to the TableEnvironment by calling `tableEnv.getConfig.setPlannerConfig(calciteConfig)`. - --- Apache Flink leverages and extends Apache Calcite to perform sophisticated query optimization. This includes a series of rule and cost-based optimizations such as: @@ -1434,21 +1422,29 @@ This includes a series of rule and cost-based optimizations such as: The optimizer makes intelligent decisions, based not only on the plan but also rich statistics available from the data sources and fine-grain costs for each operator such as io, cpu, network, and memory. Advanced users may provide custom optimizations via a `CalciteConfig` object that can be provided to the table environment by calling `TableEnvironment#getConfig#setPlannerConfig`. +++Apache Flink leverages Apache Calcite to optimize and translate queries. The optimization currently performed include projection and filter push-down, subquery decorrelation, and other kinds of query rewriting. Old planner does not yet optimize the order of joins, but executes them in the same order as defined in the query (order of Tables in the `FROM` clause and/or order of join predicates in the `WHERE` clause). + +It is possible to tweak the set of optimization rules which are applied in different phases by providing a `CalciteConfig` object. This can be created via a builder by calling `CalciteConfig.createBuilder())` and is provided to the TableEnvironment by calling `tableEnv.getConfig.setPlannerConfig(calciteConfig)`.@@ -1496,51 +1491,16 @@ table2 = t_env.from_elements([(1, "hello")], ["count", "word"]) table = table1 \ .where("LIKE(word, 'F%')") \ .union_all(table2) +print(table.explain()) -explanation = t_env.explain(table) -print(explanation) {% endhighlight %}@@ -1459,14 +1455,14 @@ StreamTableEnvironment tEnv = StreamTableEnvironment.create(env); DataStream@@ -1480,9 +1476,8 @@ val table2 = env.fromElements((1, "hello")).toTable(tEnv, $"count", $"word") val table = table1 .where($"word".like("F%")) .unionAll(table2) +println(table.explain()) -val explanation: String = tEnv.explain(table) -println(explanation) {% endhighlight %}-@@ -419,7 +421,7 @@ tableEnvironment.sqlUpdate("CREATE [TEMPORARY] TABLE MyTable (...) WITH (...)") 用户可以指定一个 catalog 和数据库作为 "当前catalog" 和"当前数据库"。有了这些，那么刚刚提到的三元标识符的前两个部分就可以被省略了。如果前两部分的标识符没有指定， 那么会使用当前的 catalog 和当前数据库。用户也可以通过 Table API 或 SQL 切换当前的 catalog 和当前的数据库。 -标识符遵循 SQL 标准，因此使用时需要用反引号（`` ` ``）进行转义。此外，所有 SQL 保留关键字都必须转义。 +标识符遵循 SQL 标准，因此使用时需要用反引号（`` ` ``）进行转义。+The result of the above exmaple is +-The following code shows an example and the corresponding output for multiple-sinks plan using `explain()`: +The following code shows an example and the corresponding output for multiple-sinks plan using `StatementSet.explain()` method:{% highlight text %} -== Abstract Syntax Tree == -LogicalUnion(all=[true]) - LogicalFilter(condition=[LIKE($1, _UTF-16LE'F%')]) - FlinkLogicalDataStreamScan(id=[1], fields=[count, word]) - FlinkLogicalDataStreamScan(id=[2], fields=[count, word]) - -== Optimized Logical Plan == -DataStreamUnion(all=[true], union all=[count, word]) - DataStreamCalc(select=[count, word], where=[LIKE(word, _UTF-16LE'F%')]) - DataStreamScan(id=[1], fields=[count, word]) - DataStreamScan(id=[2], fields=[count, word]) - -== Physical Execution Plan == -Stage 1 : Data Source - content : collect elements with CollectionInputFormat -Stage 2 : Data Source - content : collect elements with CollectionInputFormat - - Stage 3 : Operator - content : from: (count, word) - ship_strategy : REBALANCE - - Stage 4 : Operator - content : where: (LIKE(word, _UTF-16LE'F%')), select: (count, word) - ship_strategy : FORWARD - - Stage 5 : Operator - content : from: (count, word) - ship_strategy : REBALANCE -{% endhighlight %} -- --{% highlight text %} == Abstract Syntax Tree == LogicalUnion(all=[true]) LogicalFilter(condition=[LIKE($1, _UTF-16LE'F%')]) @@ -1571,62 +1531,11 @@ Stage 2 : Data Source Stage 5 : Operator content : from: (count, word) ship_strategy : REBALANCE -{% endhighlight %} ---{% highlight text %} -== Abstract Syntax Tree == -LogicalUnion(all=[true]) - LogicalFilter(condition=[LIKE($1, _UTF-16LE'F%')]) - FlinkLogicalDataStreamScan(id=[3], fields=[count, word]) - FlinkLogicalDataStreamScan(id=[6], fields=[count, word]) - -== Optimized Logical Plan == -DataStreamUnion(all=[true], union all=[count, word]) - DataStreamCalc(select=[count, word], where=[LIKE(word, _UTF-16LE'F%')]) - DataStreamScan(id=[3], fields=[count, word]) - DataStreamScan(id=[6], fields=[count, word]) - -== Physical Execution Plan == -Stage 1 : Data Source - content : collect elements with CollectionInputFormat - - Stage 2 : Operator - content : Flat Map - ship_strategy : FORWARD - - Stage 3 : Operator - content : Map - ship_strategy : FORWARD - -Stage 4 : Data Source - content : collect elements with CollectionInputFormat - - Stage 5 : Operator - content : Flat Map - ship_strategy : FORWARD - - Stage 6 : Operator - content : Map - ship_strategy : FORWARD - - Stage 7 : Operator - content : Map - ship_strategy : FORWARD - - Stage 8 : Operator - content : where: (LIKE(word, _UTF-16LE'F%')), select: (count, word) - ship_strategy : FORWARD - - Stage 9 : Operator - content : Map - ship_strategy : FORWARD {% endhighlight %}-diff --git a/docs/dev/table/common.zh.md b/docs/dev/table/common.zh.md index bd36b1e1af2d43..0371bb84f7af45 100644 --- a/docs/dev/table/common.zh.md +++ b/docs/dev/table/common.zh.md @@ -32,12 +32,11 @@ Table API 和 SQL 集成在同一套 API 中。这套 API 的核心概念是`Tab 1. Blink 将批处理作业视作流处理的一种特例。严格来说，`Table` 和 `DataSet` 之间不支持相互转换，并且批处理作业也不会转换成 `DataSet` 程序而是转换成 `DataStream` 程序，流处理作业也一样。 2. Blink 计划器不支持 `BatchTableSource`，而是使用有界的 `StreamTableSource` 来替代。 -3. Blink 计划器仅支持全新的 `Catalog` 不支持被弃用的 `ExternalCatalog`。 -4. 旧计划器和 Blink 计划器中 `FilterableTableSource` 的实现是不兼容的。旧计划器会将 `PlannerExpression` 下推至 `FilterableTableSource`，而 Blink 计划器则是将 `Expression` 下推。 -5. 基于字符串的键值配置选项仅在 Blink 计划器中使用。（详情参见 [配置]({{ site.baseurl }}/zh/dev/table/config.html) ） -6. `PlannerConfig` 在两种计划器中的实现（`CalciteConfig`）是不同的。 -7. Blink 计划器会将多sink（multiple-sinks）优化成一张有向无环图（DAG）（仅支持 `TableEnvironment`，不支持 `StreamTableEnvironment`）。旧计划器总是将每个sink都优化成一个新的有向无环图，且所有图相互独立。 -8. 旧计划器目前不支持 catalog 统计数据，而 Blink 支持。 +3. 旧计划器和 Blink 计划器中 `FilterableTableSource` 的实现是不兼容的。旧计划器会将 `PlannerExpression` 下推至 `FilterableTableSource`，而 Blink 计划器则是将 `Expression` 下推。 +4. 基于字符串的键值配置选项仅在 Blink 计划器中使用。（详情参见 [配置]({{ site.baseurl }}/zh/dev/table/config.html) ） +5. `PlannerConfig` 在两种计划器中的实现（`CalciteConfig`）是不同的。 +6. Blink 计划器会将多sink（multiple-sinks）优化成一张有向无环图（DAG），`TableEnvironment` 和 `StreamTableEnvironment` 都支持该特性。旧计划器总是将每个sink都优化成一个新的有向无环图，且所有图相互独立。 +7. 旧计划器目前不支持 catalog 统计数据，而 Blink 支持。 Table API 和 SQL 程序的结构 @@ -63,7 +62,8 @@ Table tapiResult = tableEnv.from("table1").select(...); Table sqlResult = tableEnv.sqlQuery("SELECT ... FROM table1 ... "); // emit a Table API result Table to a TableSink, same for SQL result -tapiResult.insertInto("outputTable"); +TableResult tableResult = tapiResult.executeInsert("outputTable"); +tableResult... // execute tableEnv.execute("java_job"); @@ -88,7 +88,8 @@ val tapiResult = tableEnv.from("table1").select(...) val sqlResult = tableEnv.sqlQuery("SELECT ... FROM table1 ...") // emit a Table API result Table to a TableSink, same for SQL result -tapiResult.insertInto("outputTable") +TableResult tableResult = tapiResult.executeInsert("outputTable"); +tableResult... // execute tableEnv.execute("scala_job") @@ -114,7 +115,8 @@ tapi_result = table_env.from_path("table1").select(...) sql_result = table_env.sql_query("SELECT ... FROM table1 ...") # emit a Table API result Table to a TableSink, same for SQL result -tapi_result.insert_into("outputTable") +table_result = tapi_result.execute_insert("outputTable") +table_result... # execute table_env.execute("python_job") @@ -157,7 +159,7 @@ table_env.execute("python_job") // ********************** import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; import org.apache.flink.table.api.EnvironmentSettings; -import org.apache.flink.table.api.java.StreamTableEnvironment; +import org.apache.flink.table.api.bridge.java.StreamTableEnvironment; EnvironmentSettings fsSettings = EnvironmentSettings.newInstance().useOldPlanner().inStreamingMode().build(); StreamExecutionEnvironment fsEnv = StreamExecutionEnvironment.getExecutionEnvironment(); @@ -168,7 +170,7 @@ StreamTableEnvironment fsTableEnv = StreamTableEnvironment.create(fsEnv, fsSetti // FLINK BATCH QUERY // ****************** import org.apache.flink.api.java.ExecutionEnvironment; -import org.apache.flink.table.api.java.BatchTableEnvironment; +import org.apache.flink.table.api.bridge.java.BatchTableEnvironment; ExecutionEnvironment fbEnv = ExecutionEnvironment.getExecutionEnvironment(); BatchTableEnvironment fbTableEnv = BatchTableEnvironment.create(fbEnv); @@ -178,7 +180,7 @@ BatchTableEnvironment fbTableEnv = BatchTableEnvironment.create(fbEnv); // ********************** import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; import org.apache.flink.table.api.EnvironmentSettings; -import org.apache.flink.table.api.java.StreamTableEnvironment; +import org.apache.flink.table.api.bridge.java.StreamTableEnvironment; StreamExecutionEnvironment bsEnv = StreamExecutionEnvironment.getExecutionEnvironment(); EnvironmentSettings bsSettings = EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build(); @@ -205,7 +207,7 @@ TableEnvironment bbTableEnv = TableEnvironment.create(bbSettings); // ********************** import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment import org.apache.flink.table.api.EnvironmentSettings -import org.apache.flink.table.api.scala.StreamTableEnvironment +import org.apache.flink.table.api.bridge.scala.StreamTableEnvironment val fsSettings = EnvironmentSettings.newInstance().useOldPlanner().inStreamingMode().build() val fsEnv = StreamExecutionEnvironment.getExecutionEnvironment @@ -216,7 +218,7 @@ val fsTableEnv = StreamTableEnvironment.create(fsEnv, fsSettings) // FLINK BATCH QUERY // ****************** import org.apache.flink.api.scala.ExecutionEnvironment -import org.apache.flink.table.api.scala.BatchTableEnvironment +import org.apache.flink.table.api.bridge.scala.BatchTableEnvironment val fbEnv = ExecutionEnvironment.getExecutionEnvironment val fbTableEnv = BatchTableEnvironment.create(fbEnv) @@ -226,7 +228,7 @@ val fbTableEnv = BatchTableEnvironment.create(fbEnv) // ********************** import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment import org.apache.flink.table.api.EnvironmentSettings -import org.apache.flink.table.api.scala.StreamTableEnvironment +import org.apache.flink.table.api.bridge.scala.StreamTableEnvironment val bsEnv = StreamExecutionEnvironment.getExecutionEnvironment val bsSettings = EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build() @@ -406,7 +408,7 @@ table_environment \@@ -1655,14 +1564,16 @@ tEnv.connect(new FileSystem("/sink/path2")) .withFormat(new Csv().deriveSchema()) .withSchema(schema) .createTemporaryTable("MySink2"); + +StatementSet stmtSet = tEnv.createStatementSet(); Table table1 = tEnv.from("MySource1").where($("word").like("F%")); -table1.insertInto("MySink1"); +stmtSet.addInsert("MySink1", table1); Table table2 = table1.unionAll(tEnv.from("MySource2")); -table2.insertInto("MySink2"); +stmtSet.addInsert("MySink2", table2); -String explanation = tEnv.explain(false); +String explanation = stmtSet.explain(); System.out.println(explanation); {% endhighlight %} @@ -1693,14 +1604,16 @@ tEnv.connect(new FileSystem("/sink/path2")) .withFormat(new Csv().deriveSchema()) .withSchema(schema) .createTemporaryTable("MySink2") + +val stmtSet = tEnv.createStatementSet() val table1 = tEnv.from("MySource1").where($"word".like("F%")) -table1.insertInto("MySink1") +stmtSet.addInsert("MySink1", table1) val table2 = table1.unionAll(tEnv.from("MySource2")) -table2.insertInto("MySink2") +stmtSet.addInsert("MySink2", table2) -val explanation = tEnv.explain(false) +val explanation = stmtSet.explain() println(explanation) {% endhighlight %} @@ -1731,15 +1644,18 @@ t_env.connect(FileSystem().path("/sink/path2"))) .with_format(Csv().deriveSchema()) .with_schema(schema) .create_temporary_table("MySink2") + +stmt_set = t_env.create_statement_set() table1 = t_env.from_path("MySource1").where("LIKE(word, 'F%')") -table1.insert_into("MySink1") +stmt_set.add_insert("MySink1", table1) table2 = table1.union_all(t_env.from_path("MySource2")) -table2.insert_into("MySink2") +stmt_set.add_insert("MySink2", table2) -explanation = t_env.explain() +explanation = stmt_set.explain() print(explanation) + {% endhighlight %}{% highlight sql %} -tableEnvironment.sqlUpdate("CREATE [TEMPORARY] TABLE MyTable (...) WITH (...)") +tableEnvironment.executeSql("CREATE [TEMPORARY] TABLE MyTable (...) WITH (...)") {% endhighlight %}@@ -683,7 +685,7 @@ table_env = ... # see "Create a TableEnvironment" section # register "RevenueFrance" output table # compute revenue for all customers from France and emit to "RevenueFrance" -table_env.sql_update( +table_env.execute_sql( "INSERT INTO RevenueFrance " "SELECT cID, cName, SUM(revenue) AS revSum " "FROM Orders " @@ -691,7 +693,6 @@ table_env.sql_update( "GROUP BY cID, cName" ) -# execute query {% endhighlight %}@@ -531,7 +533,9 @@ val revenue = orders // execute query {% endhighlight %} -**Note:** The Scala Table API uses Scala Symbols, which start with a single tick (`'`) to reference the attributes of a `Table`. The Table API uses Scala implicits. Make sure to import `org.apache.flink.api.scala._` and `org.apache.flink.table.api.scala._` in order to use Scala implicit conversions. +**Note:** The Scala Table API uses Scala String interpolation that starts with a dollar sign (`$`) to reference the attributes of a `Table`. The Table API uses Scala implicits. Make sure to import +* `org.apache.flink.table.api._` - for implicit expression conversions +* `org.apache.flink.api.scala._` and `org.apache.flink.table.api.bridge.scala._` if you want to convert from/to DataStream.@@ -561,7 +565,7 @@ revenue = orders \ Flink SQL 是基于实现了SQL标准的 [Apache Calcite](https://calcite.apache.org) 的。SQL 查询由常规字符串指定。 -文档 [SQL]({{ site.baseurl }}/zh/dev/table/sql.html) 描述了Flink对流处理和批处理表的SQL支持。 +文档 [SQL]({{ site.baseurl }}/zh/dev/table/sql/index.html) 描述了Flink对流处理和批处理表的SQL支持。 下面的示例演示了如何指定查询并将结果作为 `Table` 对象返回。 @@ -640,7 +644,7 @@ TableEnvironment tableEnv = ...; // see "Create a TableEnvironment" section // register "RevenueFrance" output table // compute revenue for all customers from France and emit to "RevenueFrance" -tableEnv.sqlUpdate( +tableEnv.executeSql( "INSERT INTO RevenueFrance " + "SELECT cID, cName, SUM(revenue) AS revSum " + "FROM Orders " + @@ -648,7 +652,6 @@ tableEnv.sqlUpdate( "GROUP BY cID, cName" ); -// execute query {% endhighlight %}@@ -661,7 +664,7 @@ val tableEnv = ... // see "Create a TableEnvironment" section // register "RevenueFrance" output table // compute revenue for all customers from France and emit to "RevenueFrance" -tableEnv.sqlUpdate(""" +tableEnv.executeSql(""" |INSERT INTO RevenueFrance |SELECT cID, cName, SUM(revenue) AS revSum |FROM Orders @@ -669,7 +672,6 @@ tableEnv.sqlUpdate(""" |GROUP BY cID, cName """.stripMargin) -// execute query {% endhighlight %}-+注意 **一旦 Table 被转化为 DataSet，必须使用 ExecutionEnvironment 的 execute 方法执行该 DataSet 作业。** + {% top %} ### 数据类型到 Table Schema 的映射 @@ -1273,7 +1278,7 @@ val table: Table = tableEnv.fromDataStream(stream, $"age" as "myAge", $"name" as #### POJO 类型 （Java 和 Scala） -Flink 支持 POJO 类型作为复合类型。确定 POJO 类型的规则记录在[这里]({{ site.baseurl }}/zh/dev/api_concepts.html#pojos). +Flink 支持 POJO 类型作为复合类型。确定 POJO 类型的规则记录在[这里]({{ site.baseurl }}{% link dev/types_serialization.zh.md %}#pojos). 在不指定字段名称的情况下将 POJO 类型的 `DataStream` 或 `DataSet` 转换成 `Table` 时，将使用原始 POJO 类型字段的名称。名称映射需要原始名称，并且不能按位置进行。字段可以使用别名（带有 `as` 关键字）来重命名，重新排序和投影。 @@ -1386,16 +1391,7 @@ val table: Table = tableEnv.fromDataStream(stream, $"name" as "myName") -------------------Table API 和 SQL 查询会被翻译成 [DataStream]({{ site.baseurl }}/zh/dev/datastream_api.html) 或者 [DataSet]({{ site.baseurl }}/zh/dev/batch) 程序， 这取决于它们的输入数据源是流式的还是批式的。查询在内部表示为逻辑查询计划，并被翻译成两个阶段： - -1. 优化逻辑执行计划 -2. 翻译成 DataStream 或 DataSet 程序 -对于 Streaming 而言，Table API 或者 SQL 查询在下列情况下会被翻译： - -* 当 `TableEnvironment.execute()` 被调用时。`Table` （通过 `Table.insertInto()` 输出给 `TableSink`）和 SQL （通过调用 `TableEnvironment.sqlUpdate()`）会先被缓存到 `TableEnvironment` 中，每个 sink 会被单独优化。执行计划将包括多个独立的有向无环子图。 -* `Table` 被转换成 `DataStream` 时（参阅[与 DataStream 和 DataSet API 结合](#integration-with-datastream-and-dataset-api)）。转换完成后，它就成为一个普通的 DataStream 程序，并且会在调用 `StreamExecutionEnvironment.execute()` 的时候被执行。 ++不论输入数据源是流式的还是批式的，Table API 和 SQL 查询都会被转换成 [DataStream]({{ site.baseurl }}/zh/dev/datastream_api.html) 程序。查询在内部表示为逻辑查询计划，并被翻译成两个阶段： -对于 Batch 而言，Table API 或者 SQL 查询在下列情况下会被翻译： +1. 优化逻辑执行计划 +2. 翻译成 DataStream 程序 -* `Table` 被输出给 `TableSink`，即当调用 `Table.insertInto()` 时。 -* SQL 更新语句执行时，即，当调用 `TableEnvironment.sqlUpdate()` 时。 -* `Table` 被转换成 `DataSet` 时（参阅[与 DataStream 和 DataSet API 结合](#integration-with-datastream-and-dataset-api)）。 +Table API 或者 SQL 查询在下列情况下会被翻译： -翻译完成后，Table API 或者 SQL 查询会被当做普通的 DataSet 程序对待并且会在调用 `ExecutionEnvironment.execute()` 的时候被执行。 +* 当 `TableEnvironment.executeSql()` 被调用时。该方法是用来执行一个 SQL 语句，一旦该方法被调用， SQL 语句立即被翻译。 +* 当 `Table.executeInsert()` 被调用时。该方法是用来将一个表的内容插入到目标表中，一旦该方法被调用， TABLE API 程序立即被翻译。 +* 当 `Table.execute()` 被调用时。该方法是用来将一个表的内容收集到本地，一旦该方法被调用， TABLE API 程序立即被翻译。 +* 当 `StatementSet.execute()` 被调用时。`Table` （通过 `StatementSet.addInsert()` 输出给某个 `Sink`）和 INSERT 语句 （通过调用 `StatementSet.addInsertSql()`）会先被缓存到 `StatementSet` 中，`StatementSet.execute()` 方法被调用时，所有的 sink 会被优化成一张有向无环图。 +* 当 `Table` 被转换成 `DataStream` 时（参阅[与 DataStream 和 DataSet API 结合](#integration-with-datastream-and-dataset-api)）。转换完成后，它就成为一个普通的 DataStream 程序，并会在调用 `StreamExecutionEnvironment.execute()` 时被执行。 +注意 **从 1.11 版本开始，`sqlUpdate` 方法 和 `insertInto` 方法被废弃，从这两个方法构建的 Table 程序必须通过 `StreamTableEnvironment.execute()` 方法执行，而不能通过 `StreamExecutionEnvironment.execute()` 方法来执行。**--不论输入数据源是流式的还是批式的，Table API 和 SQL 查询都会被转换成 [DataStream]({{ site.baseurl }}/zh/dev/datastream_api.html) 程序。查询在内部表示为逻辑查询计划，并被翻译成两个阶段： +{% top %} @@ -858,7 +859,7 @@ Table API 和 SQL 可以被很容易地集成并嵌入到 [DataStream]({{ site.b ### Scala 隐式转换 -Scala Table API 含有对 `DataSet`、`DataStream` 和 `Table` 类的隐式转换。 通过为 Scala DataStream API 导入 `org.apache.flink.table.api.scala._` 包以及 `org.apache.flink.api.scala._` 包，可以启用这些转换。 +Scala Table API 含有对 `DataSet`、`DataStream` 和 `Table` 类的隐式转换。 通过为 Scala DataStream API 导入 `org.apache.flink.table.api.bridge.scala._` 包以及 `org.apache.flink.api.scala._` 包，可以启用这些转换。 ### 通过 DataSet 或 DataStream 创建`视图` @@ -1022,6 +1023,8 @@ val retractStream: DataStream[(Boolean, Row)] = tableEnv.toRetractStream[Row](ta **注意：** 文档[动态表](streaming/dynamic_tables.html)给出了有关动态表及其属性的详细讨论。 +注意 **一旦 Table 被转化为 DataStream，必须使用 StreamExecutionEnvironment 的 execute 方法执行该 DataStream 作业。** + #### 将表转换成 DataSet 将 `Table` 转换成 `DataSet` 的过程如下： @@ -1065,6 +1068,8 @@ val dsTuple: DataSet[(String, Int)] = tableEnv.toDataSet[(String, Int)](table)+Table API 和 SQL 查询会被翻译成 [DataStream]({{ site.baseurl }}/zh/dev/datastream_api.html) 或者 [DataSet]({{ site.baseurl }}/zh/dev/batch) 程序， 这取决于它们的输入数据源是流式的还是批式的。查询在内部表示为逻辑查询计划，并被翻译成两个阶段： 1. 优化逻辑执行计划 -2. 翻译成 DataStream 程序 +2. 翻译成 DataStream 或 DataSet 程序 Table API 或者 SQL 查询在下列情况下会被翻译： -* 当 `TableEnvironment.execute()` 被调用时。`Table` （通过 `Table.insertInto()` 输出给 `TableSink`）和 SQL （通过调用 `TableEnvironment.sqlUpdate()`）会先被缓存到 `TableEnvironment` 中，所有的 sink 会被优化成一张有向无环图。 -* `Table` 被转换成 `DataStream` 时（参阅[与 DataStream 和 DataSet API 结合](#integration-with-datastream-and-dataset-api)）。转换完成后，它就成为一个普通的 DataStream 程序，并且会在调用 `StreamExecutionEnvironment.execute()` 的时候被执行。 +* 当 `TableEnvironment.executeSql()` 被调用时。该方法是用来执行一个 SQL 语句，一旦该方法被调用， SQL 语句立即被翻译。 +* 当 `Table.executeInsert()` 被调用时。该方法是用来将一个表的内容插入到目标表中，一旦该方法被调用， TABLE API 程序立即被翻译。 +* 当 `Table.execute()` 被调用时。该方法是用来将一个表的内容收集到本地，一旦该方法被调用， TABLE API 程序立即被翻译。 +* 当 `StatementSet.execute()` 被调用时。`Table` （通过 `StatementSet.addInsert()` 输出给某个 `Sink`）和 INSERT 语句 （通过调用 `StatementSet.addInsertSql()`）会先被缓存到 `StatementSet` 中，`StatementSet.execute()` 方法被调用时，所有的 sink 会被优化成一张有向无环图。 +* 对于 Streaming 而言，当`Table` 被转换成 `DataStream` 时（参阅[与 DataStream 和 DataSet API 结合](#integration-with-datastream-and-dataset-api)）触发翻译。转换完成后，它就成为一个普通的 DataStream 程序，并会在调用 `StreamExecutionEnvironment.execute()` 时被执行。对于 Batch 而言，`Table` 被转换成 `DataSet` 时（参阅[与 DataStream 和 DataSet API 结合](#integration-with-datastream-and-dataset-api)）触发翻译。转换完成后，它就成为一个普通的 DataSet 程序，并会在调用 `ExecutionEnvironment.execute()` 时被执行。 + +注意 **从 1.11 版本开始，`sqlUpdate` 方法 和 `insertInto` 方法被废弃。对于 Streaming 而言，如果一个 Table 程序是从这两个方法构建出来的，必须通过 `StreamTableEnvironment.execute()` 方法执行，而不能通过 `StreamExecutionEnvironment.execute()` 方法执行；对于 Batch 而言，如果一个 Table 程序是从这两个方法构建出来的，必须通过 `BatchTableEnvironment.execute()` 方法执行，而不能通过 `ExecutionEnvironment.execute()` 方法执行。**+--### 解释表 +解释表 +------------------ Table API 提供了一种机制来解释计算 `Table` 的逻辑和优化查询计划。 -这是通过 `TableEnvironment.explain(table)` 或者 `TableEnvironment.explain()` 完成的。`explain(table)` 返回给定 `Table` 的计划。 `explain()` 返回多 sink 计划的结果并且主要用于 Blink 计划器。它返回一个描述三种计划的字符串： +这是通过 `Table.explain()` 方法或者 `StatementSet.explain()` 方法来完成的。`Table.explain()` 返回一个 Table 的计划。`StatementSet.explain()` 返回多 sink 计划的结果。它返回一个描述三种计划的字符串： 1. 关系查询的抽象语法树（the Abstract Syntax Tree），即未优化的逻辑查询计划， 2. 优化的逻辑查询计划，以及 3. 物理执行计划。 -以下代码展示了一个示例以及对给定 `Table` 使用 `explain（table）` 的相应输出： +可以用 `TableEnvironment.explainSql()` 方法和 `TableEnvironment.executeSql()` 方法支持执行一个 `EXPLAIN` 语句获取逻辑和优化查询计划，请参阅 [EXPLAIN]({{ site.baseurl }}/zh/dev/table/sql/explain.html) 页面. + +以下代码展示了一个示例以及对给定 `Table` 使用 `Table.explain()` 方法的相应输出：- -Apache Flink 利用 Apache Calcite 来优化和翻译查询。当前执行的优化包括投影和过滤器下推，子查询消除以及其他类型的查询重写。原版计划程序尚未优化 join 的顺序，而是按照查询中定义的顺序执行它们（FROM 子句中的表顺序和/或 WHERE 子句中的 join 谓词顺序）。 - -通过提供一个 `CalciteConfig` 对象，可以调整在不同阶段应用的优化规则集合。这个对象可以通过调用构造器 `CalciteConfig.createBuilder()` 创建，并通过调用 `tableEnv.getConfig.setPlannerConfig(calciteConfig)` 提供给 TableEnvironment。 - --- Apache Flink 使用并扩展了 Apache Calcite 来执行复杂的查询优化。 这包括一系列基于规则和成本的优化，例如： @@ -1415,21 +1411,29 @@ Apache Flink 使用并扩展了 Apache Calcite 来执行复杂的查询优化。 优化器不仅基于计划，而且还基于可从数据源获得的丰富统计信息以及每个算子（例如 io，cpu，网络和内存）的细粒度成本来做出明智的决策。 高级用户可以通过 `CalciteConfig` 对象提供自定义优化，可以通过调用 `TableEnvironment＃getConfig＃setPlannerConfig` 将其提供给 TableEnvironment。 +++Apache Flink 利用 Apache Calcite 来优化和翻译查询。当前执行的优化包括投影和过滤器下推，子查询消除以及其他类型的查询重写。原版计划程序尚未优化 join 的顺序，而是按照查询中定义的顺序执行它们（FROM 子句中的表顺序和/或 WHERE 子句中的 join 谓词顺序）。 + +通过提供一个 `CalciteConfig` 对象，可以调整在不同阶段应用的优化规则集合。这个对象可以通过调用构造器 `CalciteConfig.createBuilder()` 创建，并通过调用 `tableEnv.getConfig.setPlannerConfig(calciteConfig)` 提供给 TableEnvironment。@@ -1477,50 +1480,14 @@ table2 = t_env.from_elements([(1, "hello")], ["count", "word"]) table = table1 \ .where("LIKE(word, 'F%')") \ .union_all(table2) +print(table.explain()) -explanation = t_env.explain(table) -print(explanation) {% endhighlight %}@@ -1440,14 +1444,14 @@ StreamTableEnvironment tEnv = StreamTableEnvironment.create(env); DataStream@@ -1461,9 +1465,8 @@ val table2 = env.fromElements((1, "hello")).toTable(tEnv, $"count", $"word") val table = table1 .where($"word".like("F%")) .unionAll(table2) +println(table.explain()) -val explanation: String = tEnv.explain(table) -println(explanation) {% endhighlight %}--{% highlight text %} -== Abstract Syntax Tree == -LogicalUnion(all=[true]) - LogicalFilter(condition=[LIKE($1, _UTF-16LE'F%')]) - FlinkLogicalDataStreamScan(id=[1], fields=[count, word]) - FlinkLogicalDataStreamScan(id=[2], fields=[count, word]) - -== Optimized Logical Plan == -DataStreamUnion(all=[true], union all=[count, word]) - DataStreamCalc(select=[count, word], where=[LIKE(word, _UTF-16LE'F%')]) - DataStreamScan(id=[1], fields=[count, word]) - DataStreamScan(id=[2], fields=[count, word]) - -== Physical Execution Plan == -Stage 1 : Data Source - content : collect elements with CollectionInputFormat - -Stage 2 : Data Source - content : collect elements with CollectionInputFormat - - Stage 3 : Operator - content : from: (count, word) - ship_strategy : REBALANCE - - Stage 4 : Operator - content : where: (LIKE(word, _UTF-16LE'F%')), select: (count, word) - ship_strategy : FORWARD - - Stage 5 : Operator - content : from: (count, word) - ship_strategy : REBALANCE -{% endhighlight %} -- -+上述例子的结果是： +- -以下代码展示了一个示例以及使用 `explain（）` 的多 sink 计划的相应输出： +以下代码展示了一个示例以及使用 `StatementSet.explain()` 的多 sink 计划的相应输出：{% highlight text %} == Abstract Syntax Tree == LogicalUnion(all=[true]) @@ -1555,59 +1522,7 @@ Stage 2 : Data Source {% endhighlight %}--{% highlight text %} -== Abstract Syntax Tree == -LogicalUnion(all=[true]) - LogicalFilter(condition=[LIKE($1, _UTF-16LE'F%')]) - FlinkLogicalDataStreamScan(id=[3], fields=[count, word]) - FlinkLogicalDataStreamScan(id=[6], fields=[count, word]) - -== Optimized Logical Plan == -DataStreamUnion(all=[true], union all=[count, word]) - DataStreamCalc(select=[count, word], where=[LIKE(word, _UTF-16LE'F%')]) - DataStreamScan(id=[3], fields=[count, word]) - DataStreamScan(id=[6], fields=[count, word]) - -== Physical Execution Plan == -Stage 1 : Data Source - content : collect elements with CollectionInputFormat - - Stage 2 : Operator - content : Flat Map - ship_strategy : FORWARD - - Stage 3 : Operator - content : Map - ship_strategy : FORWARD - -Stage 4 : Data Source - content : collect elements with CollectionInputFormat - - Stage 5 : Operator - content : Flat Map - ship_strategy : FORWARD - - Stage 6 : Operator - content : Map - ship_strategy : FORWARD - - Stage 7 : Operator - content : Map - ship_strategy : FORWARD - - Stage 8 : Operator - content : where: (LIKE(word, _UTF-16LE'F%')), select: (count, word) - ship_strategy : FORWARD - - Stage 9 : Operator - content : Map - ship_strategy : FORWARD -{% endhighlight %} --diff --git a/docs/dev/table/config.md b/docs/dev/table/config.md index dd312d6fc558ed..68a73bacf2eaea 100644 --- a/docs/dev/table/config.md +++ b/docs/dev/table/config.md @@ -104,3 +104,9 @@ The following options can be used to tune the performance of the query execution The following options can be used to adjust the behavior of the query optimizer to get a better execution plan. {% include generated/optimizer_config_configuration.html %} + +### Table Options + +The following options can be used to adjust the behavior of the table planner. + +{% include generated/table_config_configuration.html %} diff --git a/docs/dev/table/config.zh.md b/docs/dev/table/config.zh.md index 9d325e6035d983..3c41bdc825a8f4 100644 --- a/docs/dev/table/config.zh.md +++ b/docs/dev/table/config.zh.md @@ -96,3 +96,9 @@ configuration.set_string("table.exec.mini-batch.size", "5000"); 以下配置可以用于调整查询优化器的行为以获得更好的执行计划。 {% include generated/optimizer_config_configuration.html %} + +### Planner 配置 + +以下配置可以用于调整 planner 的行为。 + +{% include generated/table_config_configuration.html %} diff --git a/docs/dev/table/connect.md b/docs/dev/table/connect.md index 52a22b539b8de7..7aa151bf7de1e7 100644 --- a/docs/dev/table/connect.md +++ b/docs/dev/table/connect.md @@ -1,7 +1,5 @@ --- -title: "Table API Connectors" -nav-parent_id: connectors-root -nav-pos: 2 +title: "Table API Legacy Connectors" --- + +Sink +Sink + +* This will be replaced by the TOC +{:toc} + +The BlackHole connector allows for swallowing all input records. It is designed for: + +- high performance testing. +- UDF to output, not substantive sink. + +Just like /dev/null device on Unix-like operating systems. + +The BlackHole connector is built-in. + +How to create a BlackHole table +---------------- + +@@ -1637,13 +1552,15 @@ tEnv.connect(new FileSystem("/sink/path2")) .withSchema(schema) .createTemporaryTable("MySink2"); +StatementSet stmtSet = tEnv.createStatementSet(); + Table table1 = tEnv.from("MySource1").where($("word").like("F%")); -table1.insertInto("MySink1"); +stmtSet.addInsert("MySink1", table1); Table table2 = table1.unionAll(tEnv.from("MySource2")); -table2.insertInto("MySink2"); +stmtSet.addInsert("MySink2", table2); -String explanation = tEnv.explain(false); +String explanation = stmtSet.explain(); System.out.println(explanation); {% endhighlight %} @@ -1675,13 +1592,15 @@ tEnv.connect(new FileSystem("/sink/path2")) .withSchema(schema) .createTemporaryTable("MySink2") +val stmtSet = tEnv.createStatementSet() + val table1 = tEnv.from("MySource1").where($"word".like("F%")) -table1.insertInto("MySink1") +stmtSet.addInsert("MySink1", table1) val table2 = table1.unionAll(tEnv.from("MySource2")) -table2.insertInto("MySink2") +stmtSet.addInsert("MySink2", table2) -val explanation = tEnv.explain(false) +val explanation = stmtSet.explain() println(explanation) {% endhighlight %} @@ -1713,14 +1632,17 @@ t_env.connect(FileSystem().path("/sink/path2"))) .with_schema(schema) .create_temporary_table("MySink2") +stmt_set = t_env.create_statement_set() + table1 = t_env.from_path("MySource1").where("LIKE(word, 'F%')") -table1.insert_into("MySink1") +stmt_set.add_insert("MySink1", table1) table2 = table1.union_all(t_env.from_path("MySource2")) -table2.insert_into("MySink2") +stmt_set.add_insert("MySink2", table2) -explanation = t_env.explain() +explanation = stmt_set.explain() print(explanation) + {% endhighlight %}++ +Alternatively, it may be based on an existing schema using the [LIKE Clause]({% link dev/table/sql/create.md %}#create-table). + ++{% highlight sql %} +CREATE TABLE blackhole_table ( + f0 INT, + f1 INT, + f2 STRING, + f3 DOUBLE +) WITH ( + 'connector' = 'blackhole' +) +{% endhighlight %} ++++ +Connector Options +---------------- + ++{% highlight sql %} +CREATE TABLE blackhole_table WITH ('connector' = 'blackhole') +LIKE source_table (EXCLUDING ALL) +{% endhighlight %} +++ +
diff --git a/docs/dev/table/connectors/blackhole.zh.md b/docs/dev/table/connectors/blackhole.zh.md new file mode 100644 index 00000000000000..0727695afaad95 --- /dev/null +++ b/docs/dev/table/connectors/blackhole.zh.md @@ -0,0 +1,92 @@ +--- +title: "BlackHole SQL 连接器" +nav-title: BlackHole +nav-parent_id: sql-connectors +nav-pos: 12 +--- + + +Sink: Bounded +Sink: UnBounded + +* This will be replaced by the TOC +{:toc} + +BlackHole 连接器允许接收所有输入记录。它被设计用于： + +- 高性能测试。 +- UDF 输出，而不是实质性 sink。 + +就像类 Unix 操作系统上的 /dev/null。 + +BlackHole 连接器是内置的。 + +如何创建 BlackHole 表 +---------------- + ++ + + +Option +Required +Default +Type +Description ++ + +connector
required +(none) +String +Specify what connector to use, here should be 'blackhole'. +++ +或者，可以基于现有模式使用 [LIKE 子句]({% link dev/table/sql/create.zh.md %}#create-table) 创建。 + ++{% highlight sql %} +CREATE TABLE blackhole_table ( + f0 INT, + f1 INT, + f2 STRING, + f3 DOUBLE +) WITH ( + 'connector' = 'blackhole' +) +{% endhighlight %} ++++ +连接器选项 +---------------- + ++{% highlight sql %} +CREATE TABLE blackhole_table WITH ('connector' = 'blackhole') +LIKE source_table (EXCLUDING ALL) +{% endhighlight %} +++ +
diff --git a/docs/dev/table/connectors/datagen.md b/docs/dev/table/connectors/datagen.md new file mode 100644 index 00000000000000..ad1e6afe5432f1 --- /dev/null +++ b/docs/dev/table/connectors/datagen.md @@ -0,0 +1,152 @@ +--- +title: "DataGen SQL Connector" +nav-title: DataGen +nav-parent_id: sql-connectors +nav-pos: 10 +--- + + +Scan Source: Bounded +Scan Source: UnBounded + +* This will be replaced by the TOC +{:toc} + +The DataGen connector allows for reading by data generation rules. + +The DataGen connector can work with [Computed Column syntax]({% link dev/table/sql/create.md %}#create-table). +This allows you to generate records flexibly. + +The DataGen connector is built-in. + +Attention Complex types are not supported: Array, Map, Row. Please construct these types by computed column. + +How to create a DataGen table +---------------- + +The boundedness of table: when the generation of field data in the table is completed, the reading +is finished. So the boundedness of the table depends on the boundedness of fields. + +For each field, there are two ways to generate data: + +- Random generator is the default generator, you can specify random max and min values. For char/varchar/string, the length can be specified. It is a unbounded generator. +- Sequence generator, you can specify sequence start and end values. It is a bounded generator, when the sequence number reaches the end value, the reading ends. + ++ + + +选项 +是否必要 +默认值 +类型 +描述 ++ + +connector
必要 +(none) +String +指定需要使用的连接器，此处应为‘blackhole’。 +++ +Connector Options +---------------- + ++{% highlight sql %} +CREATE TABLE datagen ( + f_sequence INT, + f_random INT, + f_random_str STRING, + ts AS localtimestamp, + WATERMARK FOR ts AS ts +) WITH ( + 'connector' = 'datagen', + + -- optional options -- + + 'rows-per-second'='5', + + 'fields.f_sequence.kind'='sequence', + 'fields.f_sequence.start'='1', + 'fields.f_sequence.end'='1000', + + 'fields.f_random.min'='1', + 'fields.f_random.max'='1000', + + 'fields.f_random_str.length'='10' +) +{% endhighlight %} +++ +
diff --git a/docs/dev/table/connectors/datagen.zh.md b/docs/dev/table/connectors/datagen.zh.md new file mode 100644 index 00000000000000..92b59725e7c1a3 --- /dev/null +++ b/docs/dev/table/connectors/datagen.zh.md @@ -0,0 +1,152 @@ +--- +title: "DataGen SQL 连接器" +nav-title: DataGen +nav-parent_id: sql-connectors +nav-pos: 10 +--- + + +Scan Source: 有界 +Scan Source: 无界 + +* This will be replaced by the TOC +{:toc} + +DataGen 连接器允许按数据生成规则进行读取。 + +DataGen 连接器可以使用[计算列语法]({% link dev/table/sql/create.zh.md %}#create-table)。 +这使您可以灵活地生成记录。 + +DataGen 连接器是内置的。 + +注意 不支持复杂类型: Array，Map，Row。 请用计算列构造这些类型。 + +怎么创建一个 DataGen 的表 +---------------- + +表的有界性：当表中字段的数据全部生成完成后，source 就结束了。 因此，表的有界性取决于字段的有界性。 + +每个列，都有两种生成数据的方法： + +- 随机生成器是默认的生成器，您可以指定随机生成的最大和最小值。char、varchar、string （类型）可以指定长度。它是无界的生成器。 + +- 序列生成器，您可以指定序列的起始和结束值。它是有界的生成器，当序列数字达到结束值，读取结束。 ++ + + +Option +Required +Default +Type +Description ++ +connector
required +(none) +String +Specify what connector to use, here should be 'datagen'. ++ +rows-per-second
optional +10000 +Long +Rows per second to control the emit rate. ++ +fields.#.kind
optional +random +String +Generator of this '#' field. Can be 'sequence' or 'random'. ++ +fields.#.min
optional +(Minimum value of type) +(Type of field) +Minimum value of random generator, work for number types. ++ +fields.#.max
optional +(Maximum value of type) +(Type of field) +Maximum value of random generator, work for number types. ++ +fields.#.length
optional +100 +Integer +Length for string generating of random generator, work for char/varchar/string. ++ +fields.#.start
optional +(none) +(Type of field) +Start value of sequence generator. ++ + +fields.#.end
optional +(none) +(Type of field) +End value of sequence generator. +++ + +连接器参数 +---------------- + ++{% highlight sql %} +CREATE TABLE datagen ( + f_sequence INT, + f_random INT, + f_random_str STRING, + ts AS localtimestamp, + WATERMARK FOR ts AS ts +) WITH ( + 'connector' = 'datagen', + + -- optional options -- + + 'rows-per-second'='5', + + 'fields.f_sequence.kind'='sequence', + 'fields.f_sequence.start'='1', + 'fields.f_sequence.end'='1000', + + 'fields.f_random.min'='1', + 'fields.f_random.max'='1000', + + 'fields.f_random_str.length'='10' +) +{% endhighlight %} +++ +
diff --git a/docs/dev/table/connectors/elasticsearch.md b/docs/dev/table/connectors/elasticsearch.md new file mode 100644 index 00000000000000..c4b2a98b7650b4 --- /dev/null +++ b/docs/dev/table/connectors/elasticsearch.md @@ -0,0 +1,268 @@ +--- +title: "Elasticsearch SQL Connector" +nav-title: Elasticsearch +nav-parent_id: sql-connectors +nav-pos: 4 +--- + + +Sink: Batch +Sink: Streaming Append & Upsert Mode + +* This will be replaced by the TOC +{:toc} + +The Elasticsearch connector allows for writing into an index of the Elasticsearch engine. This document describes how to setup the Elasticsearch Connector to run SQL queries against Elasticsearch. + +The connector can operate in upsert mode for exchanging UPDATE/DELETE messages with the external system using the primary key defined on the DDL. + +If no primary key is defined on the DDL, the connector can only operate in append mode for exchanging INSERT only messages with external system. + +Dependencies +------------ + +In order to setup the Elasticsearch connector, the following table provides dependency information for both projects using a build automation tool (such as Maven or SBT) and SQL Client with SQL JAR bundles. + +| Elasticsearch Version | Maven dependency | SQL Client JAR | +| :---------------------- | :----------------------------------------------------------------- | :----------------------| +| 6.x | `flink-connector-elasticsearch6{{site.scala_version_suffix}}` | {% if site.is_stable %} [Download](https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-elasticsearch6{{site.scala_version_suffix}}/{{site.version}}/flink-sql-connector-elasticsearch6{{site.scala_version_suffix}}-{{site.version}}.jar) {% else %} Only available for [stable releases]({{ site.stable_baseurl }}/dev/table/connectors/elasticsearch.html) {% endif %}| +| 7.x and later versions | `flink-connector-elasticsearch7{{site.scala_version_suffix}}` | {% if site.is_stable %} [Download](https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-elasticsearch7{{site.scala_version_suffix}}/{{site.version}}/flink-sql-connector-elasticsearch7{{site.scala_version_suffix}}-{{site.version}}.jar) {% else %} Only available for [stable releases]({{ site.stable_baseurl }}/dev/table/connectors/elasticsearch.html) {% endif %}| + ++ + + +参数 +是否必选 +默认参数 +数据类型 +描述 ++ +connector
必须 +(none) +String +指定要使用的连接器，这里是 'datagen'。 ++ +rows-per-second
可选 +10000 +Long +每秒生成的行数，用以控制数据发出速率。 ++ +fields.#.kind
可选 +random +String +指定 '#' 字段的生成器。可以是 'sequence' 或 'random'。 ++ +fields.#.min
可选 +(Minimum value of type) +(Type of field) +随机生成器的最小值，适用于数字类型。 ++ +fields.#.max
可选 +(Maximum value of type) +(Type of field) +随机生成器的最大值，适用于数字类型。 ++ +fields.#.length
可选 +100 +Integer +随机生成器生成字符的长度，适用于 char、varchar、string。 ++ +fields.#.start
可选 +(none) +(Type of field) +序列生成器的起始值。 ++ + +fields.#.end
可选 +(none) +(Type of field) +序列生成器的结束值。 +
+Attention Elasticsearch connector works with JSON format which defines how to encode documents for the external system, therefore, it must be added as a [dependency]({% link dev/table/connectors/formats/index.md %}). + +How to create an Elasticsearch table +---------------- + +The example below shows how to create an Elasticsearch sink table: + +++ +Connector Options +---------------- + ++{% highlight sql %} +CREATE TABLE myUserTable ( + user_id STRING, + user_name STRING + uv BIGINT, + pv BIGINT, + PRIMARY KEY (user_id) NOT ENFORCED +) WITH ( + 'connector' = 'elasticsearch-7', + 'hosts' = 'http://localhost:9200', + 'index' = 'users' +); +{% endhighlight %} +++ +
+ +Features +---------------- + +### Key Handling + +Elasticsearch sink can work in either upsert mode or append mode, it depends on whether primary key is defined. +If primary key is defined, Elasticsearch sink works in upsert mode which can consume queries containing UPDATE/DELETE messages. +If primary key is not defined, Elasticsearch sink works in append mode which can only consume queries containing INSERT only messages. + +In Elasticsearch connector, the primary key is used to calculate the Elasticsearch document id, which is a string of up to 512 bytes. It cannot have whitespaces. +The Elasticsearch connector generates a document ID string for every row by concatenating all primary key fields in the order defined in the DDL using a key delimiter specified by `document-id.key-delimiter`. +Certain types are not allowed as primary key field as they do not have a good string representation, e.g. `BYTES`, `ROW`, `ARRAY`, `MAP`, etc. +If no primary key is specified, Elasticsearch will generate a document id automatically. + +See [CREATE TABLE DDL]({% link dev/table/sql/create.md %}#create-table) for more details about PRIMARY KEY syntax. + +### Dynamic Index + +Elasticsearch sink supports both static index and dynamic index. + +If you want to have a static index, the `index` option value should be a plain string, e.g. `'myusers'`, all the records will be consistently written into "myusers" index. + +If you want to have a dynamic index, you can use `{field_name}` to reference a field value in the record to dynamically generate a target index. +You can also use `'{field_name|date_format_string}'` to convert a field value of `TIMESTAMP/DATE/TIME` type into the format specified by the `date_format_string`. +The `date_format_string` is compatible with Java's [DateTimeFormatter](https://docs.oracle.com/javase/8/docs/api/index.html). +For example, if the option value is `'myusers-{log_ts|yyyy-MM-dd}'`, then a record with `log_ts` field value `2020-03-27 12:25:55` will be written into "myusers-2020-03-27" index. + + +Data Type Mapping +---------------- + +Elasticsearch stores document in a JSON string. So the data type mapping is between Flink data type and JSON data type. +Flink uses built-in `'json'` format for Elasticsearch connector. Please refer to [JSON Format]({% link dev/table/connectors/formats/json.md %}) page for more type mapping details. + +{% top %} \ No newline at end of file diff --git a/docs/dev/table/connectors/elasticsearch.zh.md b/docs/dev/table/connectors/elasticsearch.zh.md new file mode 100644 index 00000000000000..21c69468966054 --- /dev/null +++ b/docs/dev/table/connectors/elasticsearch.zh.md @@ -0,0 +1,268 @@ +--- +title: "Elasticsearch SQL Connector" +nav-title: Elasticsearch +nav-parent_id: sql-connectors +nav-pos: 4 +--- + + +Sink: Batch +Sink: Streaming Append & Upsert Mode + +* This will be replaced by the TOC +{:toc} + +The Elasticsearch connector allows for writing into an index of the Elasticsearch engine. This document describes how to setup the Elasticsearch Connector to run SQL queries against Elasticsearch. + +The connector can operate in upsert mode for exchanging UPDATE/DELETE messages with the external system using the primary key defined on the DDL. + +If no primary key is defined on the DDL, the connector can only operate in append mode for exchanging INSERT only messages with external system. + +Dependencies +------------ + +In order to setup the Elasticsearch connector, the following table provides dependency information for both projects using a build automation tool (such as Maven or SBT) and SQL Client with SQL JAR bundles. + +| Elasticsearch Version | Maven dependency | SQL Client JAR | +| :---------------------- | :----------------------------------------------------------------- | :----------------------| +| 6.x | `flink-connector-elasticsearch6{{site.scala_version_suffix}}` | {% if site.is_stable %} [Download](https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-elasticsearch6{{site.scala_version_suffix}}/{{site.version}}/flink-sql-connector-elasticsearch6{{site.scala_version_suffix}}-{{site.version}}.jar) {% else %} Only available for [stable releases]({{ site.stable_baseurl }}/zh/dev/table/connectors/elasticsearch.html) {% endif %}| +| 7.x and later versions | `flink-connector-elasticsearch7{{site.scala_version_suffix}}` | {% if site.is_stable %} [Download](https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-elasticsearch7{{site.scala_version_suffix}}/{{site.version}}/flink-sql-connector-elasticsearch7{{site.scala_version_suffix}}-{{site.version}}.jar) {% else %} Only available for [stable releases]({{ site.stable_baseurl }}/zh/dev/table/connectors/elasticsearch.html) {% endif %}| + ++ + + +Option +Required +Default +Type +Description ++ +connector
required +(none) +String +Specify what connector to use, valid values are: + +-
+
elasticsearch-6: connect to Elasticsearch 6.x cluster
+ elasticsearch-7: connect to Elasticsearch 7.x and later versions cluster
+
+ +hosts
required +(none) +String +One or more Elasticsearch hosts to connect to, e.g. +'http://host_name:9092;http://host_name:9093'.+ +index
required +(none) +String +Elasticsearch index for every record. Can be a static index (e.g. +'myIndex') or + a dynamic index (e.g.'index-{log_ts|yyyy-MM-dd}'). + See the following Dynamic Index section for more details.+ +document-type
required in 6.x +(none) +String +Elasticsearch document type. Not necessary anymore in +elasticsearch-7.+ +document-id.key-delimiter
optional +_ +String +Delimiter for composite keys ("_" by default), e.g., "$" would result in IDs "KEY1$KEY2$KEY3"." ++ +failure-handler
optional +fail +String +Failure handling strategy in case a request to Elasticsearch fails. Valid strategies are: + +-
+
fail: throws an exception if a request fails and thus causes a job failure.
+ ignore: ignores failures and drops the request.
+ retry_rejected: re-adds requests that have failed due to queue capacity saturation.
+ - custom class name: for failure handling with a ActionRequestFailureHandler subclass. +
+ +sink.flush-on-checkpoint
optional +true +Boolean +Flush on checkpoint or not. When disabled, a sink will not wait for all pending action requests + to be acknowledged by Elasticsearch on checkpoints. Thus, a sink does NOT provide any strong + guarantees for at-least-once delivery of action requests. + ++ +sink.bulk-flush.max-actions
optional +1000 +Integer +Maximum number of buffered actions per bulk request. + Can be set to +'0'to disable it. ++ +sink.bulk-flush.max-size
optional +2mb +MemorySize +Maximum size in memory of buffered actions per bulk request. Must be in MB granularity. + Can be set to +'0'to disable it. ++ +sink.bulk-flush.interval
optional +1s +Duration +The interval to flush buffered actions. + Can be set to +'0'to disable it. Note, both'sink.bulk-flush.max-size'and'sink.bulk-flush.max-actions'+ can be set to'0'with the flush interval set allowing for complete async processing of buffered actions. ++ +sink.bulk-flush.backoff.strategy
optional +DISABLED +String +Specify how to perform retries if any flush actions failed due to a temporary request error. Valid strategies are: + +-
+
DISABLED: no retry performed, i.e. fail after the first request error.
+ CONSTANT: wait for backoff delay between retries.
+ EXPONENTIAL: initially wait for backoff delay and increase exponentially between retries.
+
+ +sink.bulk-flush.backoff.max-retries
optional +8 +Integer +Maximum number of backoff retries. ++ +sink.bulk-flush.backoff.delay
optional +50ms +Duration +Delay between each backoff attempt. For +CONSTANTbackoff, this is simply the delay between each retry. ForEXPONENTIALbackoff, this is the initial base delay.+ +connection.max-retry-timeout
optional +(none) +Duration +Maximum timeout between retries. ++ +connection.path-prefix
optional +(none) +String +Prefix string to be added to every REST communication, e.g., +'/v1'+ + +format
optional +json +String +Elasticsearch connector supports to specify a format. The format must produce a valid json document. + By default uses built-in +'json'format. Please refer to JSON Format page for more details. +
+Attention Elasticsearch connector works with JSON format which defines how to encode documents for the external system, therefore, it must be added as a [dependency]({% link dev/table/connectors/formats/index.zh.md %}). + +How to create an Elasticsearch table +---------------- + +The example below shows how to create an Elasticsearch sink table: + +++ +Connector Options +---------------- + ++{% highlight sql %} +CREATE TABLE myUserTable ( + user_id STRING, + user_name STRING + uv BIGINT, + pv BIGINT, + PRIMARY KEY (user_id) NOT ENFORCED +) WITH ( + 'connector' = 'elasticsearch-7', + 'hosts' = 'http://localhost:9200', + 'index' = 'users' +); +{% endhighlight %} +++ +
+ +Features +---------------- + +### Key Handling + +Elasticsearch sink can work in either upsert mode or append mode, it depends on whether primary key is defined. +If primary key is defined, Elasticsearch sink works in upsert mode which can consume queries containing UPDATE/DELETE messages. +If primary key is not defined, Elasticsearch sink works in append mode which can only consume queries containing INSERT only messages. + +In Elasticsearch connector, the primary key is used to calculate the Elasticsearch document id, which is a string of up to 512 bytes. It cannot have whitespaces. +The Elasticsearch connector generates a document ID string for every row by concatenating all primary key fields in the order defined in the DDL using a key delimiter specified by `document-id.key-delimiter`. +Certain types are not allowed as primary key field as they do not have a good string representation, e.g. `BYTES`, `ROW`, `ARRAY`, `MAP`, etc. +If no primary key is specified, Elasticsearch will generate a document id automatically. + +See [CREATE TABLE DDL]({% link dev/table/sql/create.zh.md %}#create-table) for more details about PRIMARY KEY syntax. + +### Dynamic Index + +Elasticsearch sink supports both static index and dynamic index. + +If you want to have a static index, the `index` option value should be a plain string, e.g. `'myusers'`, all the records will be consistently written into "myusers" index. + +If you want to have a dynamic index, you can use `{field_name}` to reference a field value in the record to dynamically generate a target index. +You can also use `'{field_name|date_format_string}'` to convert a field value of `TIMESTAMP/DATE/TIME` type into the format specified by the `date_format_string`. +The `date_format_string` is compatible with Java's [DateTimeFormatter](https://docs.oracle.com/javase/8/docs/api/index.html). +For example, if the option value is `'myusers-{log_ts|yyyy-MM-dd}'`, then a record with `log_ts` field value `2020-03-27 12:25:55` will be written into "myusers-2020-03-27" index. + + +Data Type Mapping +---------------- + +Elasticsearch stores document in a JSON string. So the data type mapping is between Flink data type and JSON data type. +Flink uses built-in `'json'` format for Elasticsearch connector. Please refer to [JSON Format]({% link dev/table/connectors/formats/json.zh.md %}) page for more type mapping details. + +{% top %} diff --git a/docs/dev/table/connectors/filesystem.md b/docs/dev/table/connectors/filesystem.md new file mode 100644 index 00000000000000..1300bfa510a76f --- /dev/null +++ b/docs/dev/table/connectors/filesystem.md @@ -0,0 +1,374 @@ +--- +title: "FileSystem SQL Connector" +nav-title: FileSystem +nav-parent_id: sql-connectors +nav-pos: 5 +--- + + +This connector provides access to partitioned files in filesystems +supported by the [Flink FileSystem abstraction]({{ site.baseurl}}/ops/filesystems/index.html). + +* This will be replaced by the TOC +{:toc} + +The file system connector itself is included in Flink and does not require an additional dependency. +A corresponding format needs to be specified for reading and writing rows from and to a file system. + +The file system connector allows for reading and writing from a local or distributed filesystem. A filesystem table can be defined as: + ++ + + +Option +Required +Default +Type +Description ++ +connector
required +(none) +String +Specify what connector to use, valid values are: + +-
+
elasticsearch-6: connect to Elasticsearch 6.x cluster
+ elasticsearch-7: connect to Elasticsearch 7.x and later versions cluster
+
+ +hosts
required +(none) +String +One or more Elasticsearch hosts to connect to, e.g. +'http://host_name:9092;http://host_name:9093'.+ +index
required +(none) +String +Elasticsearch index for every record. Can be a static index (e.g. +'myIndex') or + a dynamic index (e.g.'index-{log_ts|yyyy-MM-dd}'). + See the following Dynamic Index section for more details.+ +document-type
required in 6.x +(none) +String +Elasticsearch document type. Not necessary anymore in +elasticsearch-7.+ +document-id.key-delimiter
optional +_ +String +Delimiter for composite keys ("_" by default), e.g., "$" would result in IDs "KEY1$KEY2$KEY3"." ++ +failure-handler
optional +fail +String +Failure handling strategy in case a request to Elasticsearch fails. Valid strategies are: + +-
+
fail: throws an exception if a request fails and thus causes a job failure.
+ ignore: ignores failures and drops the request.
+ retry_rejected: re-adds requests that have failed due to queue capacity saturation.
+ - custom class name: for failure handling with a ActionRequestFailureHandler subclass. +
+ +sink.flush-on-checkpoint
optional +true +Boolean +Flush on checkpoint or not. When disabled, a sink will not wait for all pending action requests + to be acknowledged by Elasticsearch on checkpoints. Thus, a sink does NOT provide any strong + guarantees for at-least-once delivery of action requests. + ++ +sink.bulk-flush.max-actions
optional +1000 +Integer +Maximum number of buffered actions per bulk request. + Can be set to +'0'to disable it. ++ +sink.bulk-flush.max-size
optional +2mb +MemorySize +Maximum size in memory of buffered actions per bulk request. Must be in MB granularity. + Can be set to +'0'to disable it. ++ +sink.bulk-flush.interval
optional +1s +Duration +The interval to flush buffered actions. + Can be set to +'0'to disable it. Note, both'sink.bulk-flush.max-size'and'sink.bulk-flush.max-actions'+ can be set to'0'with the flush interval set allowing for complete async processing of buffered actions. ++ +sink.bulk-flush.backoff.strategy
optional +DISABLED +String +Specify how to perform retries if any flush actions failed due to a temporary request error. Valid strategies are: + +-
+
DISABLED: no retry performed, i.e. fail after the first request error.
+ CONSTANT: wait for backoff delay between retries.
+ EXPONENTIAL: initially wait for backoff delay and increase exponentially between retries.
+
+ +sink.bulk-flush.backoff.max-retries
optional +8 +Integer +Maximum number of backoff retries. ++ +sink.bulk-flush.backoff.delay
optional +50ms +Duration +Delay between each backoff attempt. For +CONSTANTbackoff, this is simply the delay between each retry. ForEXPONENTIALbackoff, this is the initial base delay.+ +connection.max-retry-timeout
optional +(none) +Duration +Maximum timeout between retries. ++ +connection.path-prefix
optional +(none) +String +Prefix string to be added to every REST communication, e.g., +'/v1'+ + +format
optional +json +String +Elasticsearch connector supports to specify a format. The format must produce a valid json document. + By default uses built-in +'json'format. Please refer to JSON Format page for more details. +++ +Attention Make sure to include [Flink File System specific dependencies]({{ site.baseurl }}/ops/filesystems/index.html). + +Attention File system sources for streaming is still under development. In the future, the community will add support for common streaming use cases, i.e., partition and directory monitoring. + +Attention The behaviour of file system connector is much different from `previous legacy filesystem connector`: +the path parameter is specified for a directory not for a file and you can't get a human-readable file in the path that you declare. + +## Partition Files + +Flink's file system partition support uses the standard hive format. However, it does not require partitions to be pre-registered with a table catalog. Partitions are discovered and inferred based on directory structure. For example, a table partitioned based on the directory below would be inferred to contain `datetime` and `hour` partitions. + +``` +path +└── datetime=2019-08-25 + └── hour=11 + ├── part-0.parquet + ├── part-1.parquet + └── hour=12 + ├── part-0.parquet +└── datetime=2019-08-26 + └── hour=6 + ├── part-0.parquet +``` + +The file system table supports both partition inserting and overwrite inserting. See [INSERT Statement]({{ site.baseurl }}/dev/table/sql/insert.html). When you insert overwrite to a partitioned table, only the corresponding partition will be overwritten, not the entire table. + +## File Formats + +The file system connector supports multiple formats: + + - CSV: [RFC-4180](https://tools.ietf.org/html/rfc4180). Uncompressed. + - JSON: Note JSON format for file system connector is not a typical JSON file but uncompressed [newline delimited JSON](http://jsonlines.org/). + - Avro: [Apache Avro](http://avro.apache.org). Support compression by configuring `avro.codec`. + - Parquet: [Apache Parquet](http://parquet.apache.org). Compatible with Hive. + - Orc: [Apache Orc](http://orc.apache.org). Compatible with Hive. + +## Streaming Sink + +The file system connector supports streaming writes, based on Flink's [Streaming File Sink]({{ site.baseurl }}/dev/connectors/streamfile_sink.html) +to write records to file. Row-encoded Formats are csv and json. Bulk-encoded Formats are parquet, orc and avro. + +You can write SQL directly, insert the stream data into the non-partitioned table. +If it is a partitioned table, you can configure partition related operations. See [Partition Commit](filesystem.html#partition-commit) for details. + +### Rolling Policy + +Data within the partition directories are split into part files. Each partition will contain at least one part file for +each subtask of the sink that has received data for that partition. The in-progress part file will be closed and additional +part file will be created according to the configurable rolling policy. The policy rolls part files based on size, +a timeout that specifies the maximum duration for which a file can be open. + ++{% highlight sql %} +CREATE TABLE MyUserTable ( + column_name1 INT, + column_name2 STRING, + ... + part_name1 INT, + part_name2 STRING +) PARTITIONED BY (part_name1, part_name2) WITH ( + 'connector' = 'filesystem', -- required: specify the connector + 'path' = 'file:///path/to/whatever', -- required: path to a directory + 'format' = '...', -- required: file system connector requires to specify a format, + -- Please refer to Table Formats + -- section for more details + 'partition.default-name' = '...', -- optional: default partition name in case the dynamic partition + -- column value is null/empty string + + -- optional: the option to enable shuffle data by dynamic partition fields in sink phase, this can greatly + -- reduce the number of file for filesystem sink but may lead data skew, the default value is false. + 'sink.shuffle-by-partition.enable' = '...', + ... +) +{% endhighlight %} +++ +
+ +**NOTE:** For bulk formats (parquet, orc, avro), the rolling policy in combination with the checkpoint interval(pending files +become finished on the next checkpoint) control the size and number of these parts. + +**NOTE:** For row formats (csv, json), you can set the parameter `sink.rolling-policy.file-size` or `sink.rolling-policy.rollover-interval` in the connector properties and parameter `execution.checkpointing.interval` in flink-conf.yaml together +if you don't want to wait a long period before observe the data exists in file system. For other formats (avro, orc), you can just set parameter `execution.checkpointing.interval` in flink-conf.yaml. + +### Partition Commit + +After writing a partition, it is often necessary to notify downstream applications. For example, add the partition to a Hive metastore or writing a `_SUCCESS` file in the directory. The file system sink contains a partition commit feature that allows configuring custom policies. Commit actions are based on a combination of `triggers` and `policies`. + +- Trigger: The timing of the commit of the partition can be determined by the watermark with the time extracted from the partition, or by processing time. +- Policy: How to commit a partition, built-in policies support for the commit of success files and metastore, you can also implement your own policies, such as triggering hive's analysis to generate statistics, or merging small files, etc. + +**NOTE:** Partition Commit only works in dynamic partition inserting. + +#### Partition commit trigger + +To define when to commit a partition, providing partition commit trigger: + ++ + + +Key +Default +Type +Description ++ +sink.rolling-policy.file-size
128MB +MemorySize +The maximum part file size before rolling. ++ +sink.rolling-policy.rollover-interval
30 m +Duration +The maximum time duration a part file can stay open before rolling (by default 30 min to avoid to many small files). + The frequency at which this is checked is controlled by the 'sink.rolling-policy.check-interval' option. ++ + +sink.rolling-policy.check-interval
1 m +Duration +The interval for checking time based rolling policies. This controls the frequency to check whether a part file should rollover based on 'sink.rolling-policy.rollover-interval'. ++ +
+ +There are two types of trigger: +- The first is partition processing time. It neither requires partition time extraction nor watermark +generation. The trigger of partition commit according to partition creation time and current system time. This trigger +is more universal, but not so precise. For example, data delay or failover will lead to premature partition commit. +- The second is the trigger of partition commit according to the time that extracted from partition values and watermark. +This requires that your job has watermark generation, and the partition is divided according to time, such as +hourly partition or daily partition. + +If you want to let downstream see the partition as soon as possible, no matter whether its data is complete or not: +- 'sink.partition-commit.trigger'='process-time' (Default value) +- 'sink.partition-commit.delay'='0s' (Default value) +Once there is data in the partition, it will immediately commit. Note: the partition may be committed multiple times. + +If you want to let downstream see the partition only when its data is complete, and your job has watermark generation, and you can extract the time from partition values: +- 'sink.partition-commit.trigger'='partition-time' +- 'sink.partition-commit.delay'='1h' ('1h' if your partition is hourly partition, depends on your partition type) +This is the most accurate way to commit partition, and it will try to ensure that the committed partitions are as data complete as possible. + +If you want to let downstream see the partition only when its data is complete, but there is no watermark, or the time cannot be extracted from partition values: +- 'sink.partition-commit.trigger'='process-time' (Default value) +- 'sink.partition-commit.delay'='1h' ('1h' if your partition is hourly partition, depends on your partition type) +Try to commit partition accurately, but data delay or failover will lead to premature partition commit. + +Late data processing: The record will be written into its partition when a record is supposed to be +written into a partition that has already been committed, and then the committing of this partition +will be triggered again. + +#### Partition Time Extractor + +Time extractors define extracting time from partition values. + ++ + + +Key +Default +Type +Description ++ +sink.partition-commit.trigger
process-time +String +Trigger type for partition commit: 'process-time': based on the time of the machine, it neither requires partition time extraction nor watermark generation. Commit partition once the 'current system time' passes 'partition creation system time' plus 'delay'. 'partition-time': based on the time that extracted from partition values, it requires watermark generation. Commit partition once the 'watermark' passes 'time extracted from partition values' plus 'delay'. ++ + +sink.partition-commit.delay
0 s +Duration +The partition will not commit until the delay time. If it is a daily partition, should be '1 d', if it is a hourly partition, should be '1 h'. ++ +
+ +The default extractor is based on a timestamp pattern composed of your partition fields. You can also specify an implementation for fully custom partition extraction based on the `PartitionTimeExtractor` interface. + ++ + + +Key +Default +Type +Description ++ +partition.time-extractor.kind
default +String +Time extractor to extract time from partition values. Support default and custom. For default, can configure timestamp pattern. For custom, should configure extractor class. ++ +partition.time-extractor.class
(none) +String +The extractor class for implement PartitionTimeExtractor interface. ++ + +partition.time-extractor.timestamp-pattern
(none) +String +The 'default' construction way allows users to use partition fields to get a legal timestamp pattern. Default support 'yyyy-mm-dd hh:mm:ss' from first field. If timestamp should be extracted from a single partition field 'dt', can configure: '$dt'. If timestamp should be extracted from multiple partition fields, say 'year', 'month', 'day' and 'hour', can configure: '$year-$month-$day $hour:00:00'. If timestamp should be extracted from two partition fields 'dt' and 'hour', can configure: '$dt $hour:00:00'. +++ +#### Partition Commit Policy + +The partition commit policy defines what action is taken when partitions are committed. + +- The first is metastore, only hive table supports metastore policy, file system manages partitions through directory structure. +- The second is the success file, which will write an empty file in the directory corresponding to the partition. + ++{% highlight java %} + +public class HourPartTimeExtractor implements PartitionTimeExtractor { + @Override + public LocalDateTime extract(List+keys, List values) { + String dt = values.get(0); + String hour = values.get(1); + return Timestamp.valueOf(dt + " " + hour + ":00:00").toLocalDateTime(); + } +} + +{% endhighlight %} + + +
+ +You can extend the implementation of commit policy, The custom commit policy implementation like: + ++ + + +Key +Default +Type +Description ++ +sink.partition-commit.policy.kind
(none) +String +Policy to commit a partition is to notify the downstream application that the partition has finished writing, the partition is ready to be read. metastore: add partition to metastore. Only hive table supports metastore policy, file system manages partitions through directory structure. success-file: add '_success' file to directory. Both can be configured at the same time: 'metastore,success-file'. custom: use policy class to create a commit policy. Support to configure multiple policies: 'metastore,success-file'. ++ +sink.partition-commit.policy.class
(none) +String +The partition commit policy class for implement PartitionCommitPolicy interface. Only work in custom commit policy. ++ + +sink.partition-commit.success-file.name
(none) +String +The file name for success-file partition commit policy, default is '_SUCCESS'. +++ +## Full Example + +The below shows how the file system connector can be used to write a streaming query to write data from Kafka into a file system and runs a batch query to read that data back out. + +{% highlight sql %} + +CREATE TABLE kafka_table ( + user_id STRING, + order_amount DOUBLE, + log_ts TIMESTAMP(3), + WATERMARK FOR log_ts AS log_ts - INTERVAL '5' SECOND +) WITH (...); + +CREATE TABLE fs_table ( + user_id STRING, + order_amount DOUBLE, + dt STRING, + hour STRING +) PARTITIONED BY (dt, hour) WITH ( + 'connector'='filesystem', + 'path'='...', + 'format'='parquet', + 'sink.partition-commit.delay'='1 h', + 'sink.partition-commit.policy.kind'='success-file' +); + +-- streaming sql, insert into file system table +INSERT INTO TABLE fs_table SELECT user_id, order_amount, DATE_FORMAT(log_ts, 'yyyy-MM-dd'), DATE_FORMAT(log_ts, 'HH') FROM kafka_table; + +-- batch sql, select with partition pruning +SELECT * FROM fs_table WHERE dt='2020-05-20' and hour='12'; + +{% endhighlight %} + +{% top %} diff --git a/docs/dev/table/connectors/filesystem.zh.md b/docs/dev/table/connectors/filesystem.zh.md new file mode 100644 index 00000000000000..1300bfa510a76f --- /dev/null +++ b/docs/dev/table/connectors/filesystem.zh.md @@ -0,0 +1,374 @@ +--- +title: "FileSystem SQL Connector" +nav-title: FileSystem +nav-parent_id: sql-connectors +nav-pos: 5 +--- + + +This connector provides access to partitioned files in filesystems +supported by the [Flink FileSystem abstraction]({{ site.baseurl}}/ops/filesystems/index.html). + +* This will be replaced by the TOC +{:toc} + +The file system connector itself is included in Flink and does not require an additional dependency. +A corresponding format needs to be specified for reading and writing rows from and to a file system. + +The file system connector allows for reading and writing from a local or distributed filesystem. A filesystem table can be defined as: + ++{% highlight java %} + +public class AnalysisCommitPolicy implements PartitionCommitPolicy { + private HiveShell hiveShell; + @Override + public void commit(Context context) throws Exception { + if (hiveShell == null) { + hiveShell = createHiveShell(context.catalogName()); + } + hiveShell.execute(String.format("ALTER TABLE %s ADD IF NOT EXISTS PARTITION (%s = '%s') location '%s'", + context.tableName(), + context.partitionKeys().get(0), + context.partitionValues().get(0), + context.partitionPath())); + hiveShell.execute(String.format( + "ANALYZE TABLE %s PARTITION (%s = '%s') COMPUTE STATISTICS FOR COLUMNS", + context.tableName(), + context.partitionKeys().get(0), + context.partitionValues().get(0))); + } +} + +{% endhighlight %} ++++ +Attention Make sure to include [Flink File System specific dependencies]({{ site.baseurl }}/ops/filesystems/index.html). + +Attention File system sources for streaming is still under development. In the future, the community will add support for common streaming use cases, i.e., partition and directory monitoring. + +Attention The behaviour of file system connector is much different from `previous legacy filesystem connector`: +the path parameter is specified for a directory not for a file and you can't get a human-readable file in the path that you declare. + +## Partition Files + +Flink's file system partition support uses the standard hive format. However, it does not require partitions to be pre-registered with a table catalog. Partitions are discovered and inferred based on directory structure. For example, a table partitioned based on the directory below would be inferred to contain `datetime` and `hour` partitions. + +``` +path +└── datetime=2019-08-25 + └── hour=11 + ├── part-0.parquet + ├── part-1.parquet + └── hour=12 + ├── part-0.parquet +└── datetime=2019-08-26 + └── hour=6 + ├── part-0.parquet +``` + +The file system table supports both partition inserting and overwrite inserting. See [INSERT Statement]({{ site.baseurl }}/dev/table/sql/insert.html). When you insert overwrite to a partitioned table, only the corresponding partition will be overwritten, not the entire table. + +## File Formats + +The file system connector supports multiple formats: + + - CSV: [RFC-4180](https://tools.ietf.org/html/rfc4180). Uncompressed. + - JSON: Note JSON format for file system connector is not a typical JSON file but uncompressed [newline delimited JSON](http://jsonlines.org/). + - Avro: [Apache Avro](http://avro.apache.org). Support compression by configuring `avro.codec`. + - Parquet: [Apache Parquet](http://parquet.apache.org). Compatible with Hive. + - Orc: [Apache Orc](http://orc.apache.org). Compatible with Hive. + +## Streaming Sink + +The file system connector supports streaming writes, based on Flink's [Streaming File Sink]({{ site.baseurl }}/dev/connectors/streamfile_sink.html) +to write records to file. Row-encoded Formats are csv and json. Bulk-encoded Formats are parquet, orc and avro. + +You can write SQL directly, insert the stream data into the non-partitioned table. +If it is a partitioned table, you can configure partition related operations. See [Partition Commit](filesystem.html#partition-commit) for details. + +### Rolling Policy + +Data within the partition directories are split into part files. Each partition will contain at least one part file for +each subtask of the sink that has received data for that partition. The in-progress part file will be closed and additional +part file will be created according to the configurable rolling policy. The policy rolls part files based on size, +a timeout that specifies the maximum duration for which a file can be open. + ++{% highlight sql %} +CREATE TABLE MyUserTable ( + column_name1 INT, + column_name2 STRING, + ... + part_name1 INT, + part_name2 STRING +) PARTITIONED BY (part_name1, part_name2) WITH ( + 'connector' = 'filesystem', -- required: specify the connector + 'path' = 'file:///path/to/whatever', -- required: path to a directory + 'format' = '...', -- required: file system connector requires to specify a format, + -- Please refer to Table Formats + -- section for more details + 'partition.default-name' = '...', -- optional: default partition name in case the dynamic partition + -- column value is null/empty string + + -- optional: the option to enable shuffle data by dynamic partition fields in sink phase, this can greatly + -- reduce the number of file for filesystem sink but may lead data skew, the default value is false. + 'sink.shuffle-by-partition.enable' = '...', + ... +) +{% endhighlight %} +++ +
+ +**NOTE:** For bulk formats (parquet, orc, avro), the rolling policy in combination with the checkpoint interval(pending files +become finished on the next checkpoint) control the size and number of these parts. + +**NOTE:** For row formats (csv, json), you can set the parameter `sink.rolling-policy.file-size` or `sink.rolling-policy.rollover-interval` in the connector properties and parameter `execution.checkpointing.interval` in flink-conf.yaml together +if you don't want to wait a long period before observe the data exists in file system. For other formats (avro, orc), you can just set parameter `execution.checkpointing.interval` in flink-conf.yaml. + +### Partition Commit + +After writing a partition, it is often necessary to notify downstream applications. For example, add the partition to a Hive metastore or writing a `_SUCCESS` file in the directory. The file system sink contains a partition commit feature that allows configuring custom policies. Commit actions are based on a combination of `triggers` and `policies`. + +- Trigger: The timing of the commit of the partition can be determined by the watermark with the time extracted from the partition, or by processing time. +- Policy: How to commit a partition, built-in policies support for the commit of success files and metastore, you can also implement your own policies, such as triggering hive's analysis to generate statistics, or merging small files, etc. + +**NOTE:** Partition Commit only works in dynamic partition inserting. + +#### Partition commit trigger + +To define when to commit a partition, providing partition commit trigger: + ++ + + +Key +Default +Type +Description ++ +sink.rolling-policy.file-size
128MB +MemorySize +The maximum part file size before rolling. ++ +sink.rolling-policy.rollover-interval
30 m +Duration +The maximum time duration a part file can stay open before rolling (by default 30 min to avoid to many small files). + The frequency at which this is checked is controlled by the 'sink.rolling-policy.check-interval' option. ++ + +sink.rolling-policy.check-interval
1 m +Duration +The interval for checking time based rolling policies. This controls the frequency to check whether a part file should rollover based on 'sink.rolling-policy.rollover-interval'. ++ +
+ +There are two types of trigger: +- The first is partition processing time. It neither requires partition time extraction nor watermark +generation. The trigger of partition commit according to partition creation time and current system time. This trigger +is more universal, but not so precise. For example, data delay or failover will lead to premature partition commit. +- The second is the trigger of partition commit according to the time that extracted from partition values and watermark. +This requires that your job has watermark generation, and the partition is divided according to time, such as +hourly partition or daily partition. + +If you want to let downstream see the partition as soon as possible, no matter whether its data is complete or not: +- 'sink.partition-commit.trigger'='process-time' (Default value) +- 'sink.partition-commit.delay'='0s' (Default value) +Once there is data in the partition, it will immediately commit. Note: the partition may be committed multiple times. + +If you want to let downstream see the partition only when its data is complete, and your job has watermark generation, and you can extract the time from partition values: +- 'sink.partition-commit.trigger'='partition-time' +- 'sink.partition-commit.delay'='1h' ('1h' if your partition is hourly partition, depends on your partition type) +This is the most accurate way to commit partition, and it will try to ensure that the committed partitions are as data complete as possible. + +If you want to let downstream see the partition only when its data is complete, but there is no watermark, or the time cannot be extracted from partition values: +- 'sink.partition-commit.trigger'='process-time' (Default value) +- 'sink.partition-commit.delay'='1h' ('1h' if your partition is hourly partition, depends on your partition type) +Try to commit partition accurately, but data delay or failover will lead to premature partition commit. + +Late data processing: The record will be written into its partition when a record is supposed to be +written into a partition that has already been committed, and then the committing of this partition +will be triggered again. + +#### Partition Time Extractor + +Time extractors define extracting time from partition values. + ++ + + +Key +Default +Type +Description ++ +sink.partition-commit.trigger
process-time +String +Trigger type for partition commit: 'process-time': based on the time of the machine, it neither requires partition time extraction nor watermark generation. Commit partition once the 'current system time' passes 'partition creation system time' plus 'delay'. 'partition-time': based on the time that extracted from partition values, it requires watermark generation. Commit partition once the 'watermark' passes 'time extracted from partition values' plus 'delay'. ++ + +sink.partition-commit.delay
0 s +Duration +The partition will not commit until the delay time. If it is a daily partition, should be '1 d', if it is a hourly partition, should be '1 h'. ++ +
+ +The default extractor is based on a timestamp pattern composed of your partition fields. You can also specify an implementation for fully custom partition extraction based on the `PartitionTimeExtractor` interface. + ++ + + +Key +Default +Type +Description ++ +partition.time-extractor.kind
default +String +Time extractor to extract time from partition values. Support default and custom. For default, can configure timestamp pattern. For custom, should configure extractor class. ++ +partition.time-extractor.class
(none) +String +The extractor class for implement PartitionTimeExtractor interface. ++ + +partition.time-extractor.timestamp-pattern
(none) +String +The 'default' construction way allows users to use partition fields to get a legal timestamp pattern. Default support 'yyyy-mm-dd hh:mm:ss' from first field. If timestamp should be extracted from a single partition field 'dt', can configure: '$dt'. If timestamp should be extracted from multiple partition fields, say 'year', 'month', 'day' and 'hour', can configure: '$year-$month-$day $hour:00:00'. If timestamp should be extracted from two partition fields 'dt' and 'hour', can configure: '$dt $hour:00:00'. +++ +#### Partition Commit Policy + +The partition commit policy defines what action is taken when partitions are committed. + +- The first is metastore, only hive table supports metastore policy, file system manages partitions through directory structure. +- The second is the success file, which will write an empty file in the directory corresponding to the partition. + ++{% highlight java %} + +public class HourPartTimeExtractor implements PartitionTimeExtractor { + @Override + public LocalDateTime extract(List+keys, List values) { + String dt = values.get(0); + String hour = values.get(1); + return Timestamp.valueOf(dt + " " + hour + ":00:00").toLocalDateTime(); + } +} + +{% endhighlight %} + + +
+ +You can extend the implementation of commit policy, The custom commit policy implementation like: + ++ + + +Key +Default +Type +Description ++ +sink.partition-commit.policy.kind
(none) +String +Policy to commit a partition is to notify the downstream application that the partition has finished writing, the partition is ready to be read. metastore: add partition to metastore. Only hive table supports metastore policy, file system manages partitions through directory structure. success-file: add '_success' file to directory. Both can be configured at the same time: 'metastore,success-file'. custom: use policy class to create a commit policy. Support to configure multiple policies: 'metastore,success-file'. ++ +sink.partition-commit.policy.class
(none) +String +The partition commit policy class for implement PartitionCommitPolicy interface. Only work in custom commit policy. ++ + +sink.partition-commit.success-file.name
(none) +String +The file name for success-file partition commit policy, default is '_SUCCESS'. +++ +## Full Example + +The below shows how the file system connector can be used to write a streaming query to write data from Kafka into a file system and runs a batch query to read that data back out. + +{% highlight sql %} + +CREATE TABLE kafka_table ( + user_id STRING, + order_amount DOUBLE, + log_ts TIMESTAMP(3), + WATERMARK FOR log_ts AS log_ts - INTERVAL '5' SECOND +) WITH (...); + +CREATE TABLE fs_table ( + user_id STRING, + order_amount DOUBLE, + dt STRING, + hour STRING +) PARTITIONED BY (dt, hour) WITH ( + 'connector'='filesystem', + 'path'='...', + 'format'='parquet', + 'sink.partition-commit.delay'='1 h', + 'sink.partition-commit.policy.kind'='success-file' +); + +-- streaming sql, insert into file system table +INSERT INTO TABLE fs_table SELECT user_id, order_amount, DATE_FORMAT(log_ts, 'yyyy-MM-dd'), DATE_FORMAT(log_ts, 'HH') FROM kafka_table; + +-- batch sql, select with partition pruning +SELECT * FROM fs_table WHERE dt='2020-05-20' and hour='12'; + +{% endhighlight %} + +{% top %} diff --git a/docs/dev/table/connectors/formats/avro.md b/docs/dev/table/connectors/formats/avro.md new file mode 100644 index 00000000000000..017c96c27dbeeb --- /dev/null +++ b/docs/dev/table/connectors/formats/avro.md @@ -0,0 +1,218 @@ +--- +title: "Avro Format" +nav-title: Avro +nav-parent_id: sql-formats +nav-pos: 3 +--- + + +Format: Serialization Schema +Format: Deserialization Schema + +* This will be replaced by the TOC +{:toc} + +The [Apache Avro](https://avro.apache.org/) format allows to read and write Avro data based on an Avro schema. Currently, the Avro schema is derived from table schema. + +Dependencies +------------ + +In order to setup the Avro format, the following table provides dependency information for both projects using a build automation tool (such as Maven or SBT) and SQL Client with SQL JAR bundles. + ++{% highlight java %} + +public class AnalysisCommitPolicy implements PartitionCommitPolicy { + private HiveShell hiveShell; + @Override + public void commit(Context context) throws Exception { + if (hiveShell == null) { + hiveShell = createHiveShell(context.catalogName()); + } + hiveShell.execute(String.format("ALTER TABLE %s ADD IF NOT EXISTS PARTITION (%s = '%s') location '%s'", + context.tableName(), + context.partitionKeys().get(0), + context.partitionValues().get(0), + context.partitionPath())); + hiveShell.execute(String.format( + "ANALYZE TABLE %s PARTITION (%s = '%s') COMPUTE STATISTICS FOR COLUMNS", + context.tableName(), + context.partitionKeys().get(0), + context.partitionValues().get(0))); + } +} + +{% endhighlight %} ++++ +How to create a table with Avro format +---------------- + +Here is an example to create a table using Kafka connector and Avro format. + ++You can download flink-avro from [Download](https://repo.maven.apache.org/maven2/org/apache/flink/flink-avro/{{site.version}}/flink-avro-{{site.version}}-sql-jar.jar), +and requires additional [Hadoop dependency]({% link ops/deployment/hadoop.md %}) for cluster execution. +++{% highlight xml %} +++ +{% endhighlight %} +org.apache.flink +flink-avro +{{ site.version }} +++ +Format Options +---------------- + ++{% highlight sql %} +CREATE TABLE user_behavior ( + user_id BIGINT, + item_id BIGINT, + category_id BIGINT, + behavior STRING, + ts TIMESTAMP(3) +) WITH ( + 'connector' = 'kafka', + 'topic' = 'user_behavior', + 'properties.bootstrap.servers' = 'localhost:9092', + 'properties.group.id' = 'testGroup', + 'format' = 'avro' +) +{% endhighlight %} +++ +
+ +Data Type Mapping +---------------- + +Currently, the Avro schema is always derived from table schema. Explicitly defining an Avro schema is not supported yet. +So the following table lists the type mapping from Flink type to Avro type. + ++ + + +Option +Required +Default +Type +Description ++ +format
required +(none) +String +Specify what format to use, here should be +'avro'.+ + +avro.codec
optional +(none) +String +For Filesystem only, the compression codec for avro. No compression as default. The valid enumerations are: deflate, snappy, bzip2, xz. ++ +
+ +In addition to the types listed above, Flink supports reading/writing nullable types. Flink maps nullable types to Avro `union(something, null)`, where `something` is the Avro type converted from Flink type. + +You can refer to [Avro Specification](https://avro.apache.org/docs/current/spec.html) for more information about Avro types. diff --git a/docs/dev/table/connectors/formats/avro.zh.md b/docs/dev/table/connectors/formats/avro.zh.md new file mode 100644 index 00000000000000..89cbfc9ede3d47 --- /dev/null +++ b/docs/dev/table/connectors/formats/avro.zh.md @@ -0,0 +1,217 @@ +--- +title: "Avro Format" +nav-title: Avro +nav-parent_id: sql-formats +nav-pos: 3 +--- + + +Format: Serialization Schema +Format: Deserialization Schema + +* This will be replaced by the TOC +{:toc} + +[Apache Avro](https://avro.apache.org/) format 允许基于 Avro schema 读取和写入 Avro 数据。目前，Avro schema 从 table schema 推导而来。 + +依赖 +------------ + +为了设置 Avro format，下表提供了使用自动化构建工具（例如 Maven 或 SBT）和带有 SQL JAR 捆绑包的 SQL Client 的项目依赖信息。 + ++ + + +Flink SQL type +Avro type +Avro logical type ++ +CHAR / VARCHAR / STRING +string ++ + +BOOLEANboolean+ + +BINARY / VARBINARYbytes+ + +DECIMALfixeddecimal+ +TINYINTint+ + +SMALLINTint+ + +INTint+ + +BIGINTlong+ + +FLOATfloat+ + +DOUBLEdouble+ + +DATEintdate+ +TIMEinttime-millis+ +TIMESTAMPlongtimestamp-millis+ +ARRAYarray+ + +MAP
+ (key must be string/char/varchar type)map+ + +MULTISET
+ (element must be string/char/varchar type)map+ + + +ROWrecord+ ++ +如何使用 Avro format 创建表 +---------------- + +这是使用 Kafka 连接器和 Avro format 创建表的示例。 + ++您可以从 [这里](https://repo.maven.apache.org/maven2/org/apache/flink/flink-avro/{{site.version}}/flink-avro-{{site.version}}-sql-jar.jar) 下载 flink-avro，除此之外还需要额外的 [Hadoop 依赖]({% link ops/deployment/hadoop.zh.md %}) 用于集群执行。 + +++{% highlight xml %} +++ +{% endhighlight %} +org.apache.flink +flink-avro +{{ site.version }} +++ +Format 参数 +---------------- + ++{% highlight sql %} +CREATE TABLE user_behavior ( + user_id BIGINT, + item_id BIGINT, + category_id BIGINT, + behavior STRING, + ts TIMESTAMP(3) +) WITH ( + 'connector' = 'kafka', + 'topic' = 'user_behavior', + 'properties.bootstrap.servers' = 'localhost:9092', + 'properties.group.id' = 'testGroup', + 'format' = 'avro' +) +{% endhighlight %} +++ +
+ +数据类型映射 +---------------- + +目前，Avro schema 通常是从 table schema 中推导而来。尚不支持显式定义 Avro schema。因此，下表列出了从 Flink 类型到 Avro 类型的类型映射。 + ++ + + +参数 +是否必选 +默认值 +类型 +描述 ++ +format
必要 +(none) +String +指定使用什么 format，这里应该是 +'avro'。+ + +avro.codec
可选 +(none) +String +仅用于 filesystem，avro 压缩编解码器。默认不压缩。目前支持：deflate、snappy、bzip2、xz。 ++ +
+ +除了上面列出的类型，Flink 支持读取/写入 nullable 的类型。Flink 将 nullable 的类型映射到 Avro `union(something, null)`，其中 `something` 是从 Flink 类型转换的 Avro 类型。 + +您可以参考 [Avro 规范](https://avro.apache.org/docs/current/spec.html) 获取更多有关 Avro 类型的信息。 diff --git a/docs/dev/table/connectors/formats/canal.md b/docs/dev/table/connectors/formats/canal.md new file mode 100644 index 00000000000000..6f9ea44de9d5d8 --- /dev/null +++ b/docs/dev/table/connectors/formats/canal.md @@ -0,0 +1,191 @@ +--- +title: "Canal Format" +nav-title: Canal +nav-parent_id: sql-formats +nav-pos: 5 +--- + + +Changelog-Data-Capture Format +Format: Deserialization Schema + +* This will be replaced by the TOC +{:toc} + +[Canal](https://github.com/alibaba/canal/wiki) is a CDC (Changelog Data Capture) tool that can stream changes in real-time from MySQL into other systems. Canal provides a unified format schema for changelog and supports to serialize messages using JSON and [protobuf](https://developers.google.com/protocol-buffers) (protobuf is the default format for Canal). + +Flink supports to interpret Canal JSON messages as INSERT/UPDATE/DELETE messages into Flink SQL system. This is useful in many cases to leverage this feature, such as + - synchronizing incremental data from databases to other systems + - auditing logs + - real-time materialized views on databases + - temporal join changing history of a database table and so on. + +*Note: Support for interpreting Canal protobuf messages and emitting Canal messages is on the roadmap.* + +Dependencies +------------ + +In order to setup the Canal format, the following table provides dependency information for both projects using a build automation tool (such as Maven or SBT) and SQL Client with SQL JAR bundles. + +| Maven dependency | SQL Client JAR | +| :----------------- | :----------------------| +| `flink-json` | Built-in | + +*Note: please refer to [Canal documentation](https://github.com/alibaba/canal/wiki) about how to deploy Canal to synchronize changelog to message queues.* + + +How to use Canal format +---------------- + +Canal provides a unified format for changelog, here is a simple example for an update operation captured from a MySQL `products` table: + +```json +{ + "data": [ + { + "id": "111", + "name": "scooter", + "description": "Big 2-wheel scooter", + "weight": "5.18" + } + ], + "database": "inventory", + "es": 1589373560000, + "id": 9, + "isDdl": false, + "mysqlType": { + "id": "INTEGER", + "name": "VARCHAR(255)", + "description": "VARCHAR(512)", + "weight": "FLOAT" + }, + "old": [ + { + "weight": "5.15" + } + ], + "pkNames": [ + "id" + ], + "sql": "", + "sqlType": { + "id": 4, + "name": 12, + "description": 12, + "weight": 7 + }, + "table": "products", + "ts": 1589373560798, + "type": "UPDATE" +} +``` + +*Note: please refer to [Canal documentation](https://github.com/alibaba/canal/wiki) about the meaning of each fields.* + +The MySQL `products` table has 4 columns (`id`, `name`, `description` and `weight`). The above JSON message is an update change event on the `products` table where the `weight` value of the row with `id = 111` is changed from `5.18` to `5.15`. +Assuming the messages have been synchronized to Kafka topic `products_binlog`, then we can use the following DDL to consume this topic and interpret the change events. + ++ + + +Flink SQL 类型 +Avro 类型 +Avro 逻辑类型 ++ +CHAR / VARCHAR / STRING +string ++ + +BOOLEANboolean+ + +BINARY / VARBINARYbytes+ + +DECIMALfixeddecimal+ +TINYINTint+ + +SMALLINTint+ + +INTint+ + +BIGINTlong+ + +FLOATfloat+ + +DOUBLEdouble+ + +DATEintdate+ +TIMEinttime-millis+ +TIMESTAMPlongtimestamp-millis+ +ARRAYarray+ + +MAP
+ (key 必须是 string/char/varchar 类型)map+ + +MULTISET
+ (元素必须是 string/char/varchar 类型)map+ + + +ROWrecord+ ++ +After registering the topic as a Flink table, you can consume the Canal messages as a changelog source. + ++{% highlight sql %} +CREATE TABLE topic_products ( + -- schema is totally the same to the MySQL "products" table + id BIGINT, + name STRING, + description STRING, + weight DECIMAL(10, 2) +) WITH ( + 'connector' = 'kafka', + 'topic' = 'products_binlog', + 'properties.bootstrap.servers' = 'localhost:9092', + 'properties.group.id' = 'testGroup', + 'format' = 'canal-json' -- using canal-json as the format +) +{% endhighlight %} ++++ + +Format Options +---------------- + ++{% highlight sql %} +-- a real-time materialized view on the MySQL "products" +-- which calculates the latest average of weight for the same products +SELECT name, AVG(weight) FROM topic_products GROUP BY name; + +-- synchronize all the data and incremental changes of MySQL "products" table to +-- Elasticsearch "products" index for future searching +INSERT INTO elasticsearch_products +SELECT * FROM topic_products; +{% endhighlight %} +++ +
+ +Data Type Mapping +---------------- + +Currently, the Canal format uses JSON format for deserialization. Please refer to [JSON format documentation]({% link dev/table/connectors/formats/json.md %}#data-type-mapping) for more details about the data type mapping. + diff --git a/docs/dev/table/connectors/formats/canal.zh.md b/docs/dev/table/connectors/formats/canal.zh.md new file mode 100644 index 00000000000000..92c7180284dded --- /dev/null +++ b/docs/dev/table/connectors/formats/canal.zh.md @@ -0,0 +1,189 @@ +--- +title: "Canal Format" +nav-title: Canal +nav-parent_id: sql-formats +nav-pos: 5 +--- + + +Changelog-Data-Capture Format +Format: Deserialization Schema + +* This will be replaced by the TOC +{:toc} + +[Canal](https://github.com/alibaba/canal/wiki) 是一个 CDC（ChangeLog Data Capture，变更日志数据捕获）工具，可以实时地将 MySQL 变更传输到其他系统。Canal 为变更日志提供了统一的数据格式，并支持使用 JSON 或 [protobuf](https://developers.google.com/protocol-buffers) 序列化消息（Canal 默认使用 protobuf）。 + +Flink 支持将 Canal 的 JSON 消息解析为 INSERT / UPDATE / DELETE 消息到 Flink SQL 系统中。在很多情况下，利用这个特性非常的有用，例如 + - 将增量数据从数据库同步到其他系统 + - 日志审计 + - 数据库的实时物化视图 + - 关联维度数据库的变更历史，等等。 + +*注意：未来会支持 Canal protobuf 类型消息的解析以及输出 Canal 格式的消息。* + +依赖 +------------ + +为了设置 Canal 格式，下表提供了使用自动化构建工具（例如：Maven 或 SBT）项目和使用绑定 SQL JAR 包的 SQL Client 所需的依赖信息。 + +| Maven 依赖 | SQL Client JAR | +| :----------------- | :----------------------| +| `flink-json` | 内置 | + +*注意：有关如何部署 Canal 以将变更日志同步到消息队列，请参阅 [Canal 文档](https://github.com/alibaba/canal/wiki)。* + + +如何使用 Canal Format +---------------- + +Canal 为变更日志提供了统一的格式，下面是一个从 MySQL 库 `products` 表中捕获更新操作的简单示例： + +```json +{ + "data": [ + { + "id": "111", + "name": "scooter", + "description": "Big 2-wheel scooter", + "weight": "5.18" + } + ], + "database": "inventory", + "es": 1589373560000, + "id": 9, + "isDdl": false, + "mysqlType": { + "id": "INTEGER", + "name": "VARCHAR(255)", + "description": "VARCHAR(512)", + "weight": "FLOAT" + }, + "old": [ + { + "weight": "5.15" + } + ], + "pkNames": [ + "id" + ], + "sql": "", + "sqlType": { + "id": 4, + "name": 12, + "description": 12, + "weight": 7 + }, + "table": "products", + "ts": 1589373560798, + "type": "UPDATE" +} +``` + +*注意：有关各个字段的含义，请参阅 [Canal 文档](https://github.com/alibaba/canal/wiki)* + +MySQL `products` 表有4列（`id`，`name`，`description` 和 `weight`）。上面的 JSON 消息是 `products` 表上的一个更新事件，表示 `id = 111` 的行数据上 `weight` 字段值从`5.15`变更成为 `5.18`。假设消息已经同步到了一个 Kafka 主题：`products_binlog`，那么就可以使用以下DDL来从这个主题消费消息并解析变更事件。 + ++ + + +Option +Required +Default +Type +Description ++ +format
required +(none) +String +Specify what format to use, here should be +'canal-json'.+ +canal-json.ignore-parse-errors
optional +false +Boolean +Skip fields and rows with parse errors instead of failing. + Fields are set to null in case of errors. ++ + +canal-json.timestamp-format.standard
optional +'SQL'String +Specify the input and output timestamp format. Currently supported values are +'SQL'and'ISO-8601': +-
+
- Option
'SQL'will parse input timestamp in "yyyy-MM-dd HH:mm:ss.s{precision}" format, e.g '2020-12-30 12:13:14.123' and output timestamp in the same format.
+ - Option
'ISO-8601'will parse input timestamp in "yyyy-MM-ddTHH:mm:ss.s{precision}" format, e.g '2020-12-30T12:13:14.123' and output timestamp in the same format.
+
++ +将 Kafka 主题注册成 Flink 表之后，就可以将 Canal 消息用作变更日志源。 + ++{% highlight sql %} +CREATE TABLE topic_products ( + -- 元数据与 MySQL "products" 表完全相同 + id BIGINT, + name STRING, + description STRING, + weight DECIMAL(10, 2) +) WITH ( + 'connector' = 'kafka', + 'topic' = 'products_binlog', + 'properties.bootstrap.servers' = 'localhost:9092', + 'properties.group.id' = 'testGroup', + 'format' = 'canal-json' -- 使用 canal-json 格式 +) +{% endhighlight %} ++++ + +Format 参数 +---------------- + ++{% highlight sql %} +-- 关于MySQL "products" 表的实时物化视图 +-- 计算相同产品的最新平均重量 +SELECT name, AVG(weight) FROM topic_products GROUP BY name; + +-- 将 MySQL "products" 表的所有数据和增量更改同步到 +-- Elasticsearch "products" 索引以供将来搜索 +INSERT INTO elasticsearch_products +SELECT * FROM topic_products; +{% endhighlight %} +++ +
+ +数据类型映射 +---------------- + +目前，Canal Format 使用 JSON Format 进行反序列化。 有关数据类型映射的更多详细信息，请参阅 [JSON Format 文档]({% link dev/table/connectors/formats/json.zh.md %}#data-type-mapping)。 + diff --git a/docs/dev/table/connectors/formats/csv.md b/docs/dev/table/connectors/formats/csv.md new file mode 100644 index 00000000000000..b34a1e839a569c --- /dev/null +++ b/docs/dev/table/connectors/formats/csv.md @@ -0,0 +1,253 @@ +--- +title: "CSV Format" +nav-title: CSV +nav-parent_id: sql-formats +nav-pos: 1 +--- + + +Format: Serialization Schema +Format: Deserialization Schema + +* This will be replaced by the TOC +{:toc} + +The [CSV](https://en.wikipedia.org/wiki/Comma-separated_values) format allows to read and write CSV data based on an CSV schema. Currently, the CSV schema is derived from table schema. + +Dependencies +------------ + +In order to setup the CSV format, the following table provides dependency information for both projects using a build automation tool (such as Maven or SBT) and SQL Client with SQL JAR bundles. + +| Maven dependency | SQL Client JAR | +| :----------------- | :----------------------| +| `flink-csv` | Built-in | + +How to create a table with CSV format +---------------- + +Here is an example to create a table using Kafka connector and CSV format. + ++ + + +选项 +要求 +默认 +类型 +描述 ++ +format
必填 +(none) +String +指定要使用的格式，此处应为 +'canal-json'.+ +canal-json.ignore-parse-errors
选填 +false +Boolean +当解析异常时，是跳过当前字段或行，还是抛出错误失败（默认为 false，即抛出错误失败）。如果忽略字段的解析异常，则会将该字段值设置为 +null。+ + +canal-json.timestamp-format.standard
选填 +'SQL'String +指定输入和输出时间戳格式。 当前支持的值是 +'SQL'和'ISO-8601': +-
+
- 选项
'SQL'将解析 "yyyy-MM-dd HH:mm:ss.s{precision}" 格式的输入时间戳，例如 '2020-12-30 12:13:14.123'，并以相同格式输出时间戳。
+ - 选项
'ISO-8601'将解析 "yyyy-MM-ddTHH:mm:ss.s{precision}" 格式的输入时间戳，例如 '2020-12-30T12:13:14.123'，并以相同的格式输出时间戳。
+
++ +Format Options +---------------- + ++{% highlight sql %} +CREATE TABLE user_behavior ( + user_id BIGINT, + item_id BIGINT, + category_id BIGINT, + behavior STRING, + ts TIMESTAMP(3) +) WITH ( + 'connector' = 'kafka', + 'topic' = 'user_behavior', + 'properties.bootstrap.servers' = 'localhost:9092', + 'properties.group.id' = 'testGroup', + 'format' = 'csv', + 'csv.ignore-parse-errors' = 'true', + 'csv.allow-comments' = 'true' +) +{% endhighlight %} +++ +
+ +Data Type Mapping +---------------- + +Currently, the CSV schema is always derived from table schema. Explicitly defining an CSV schema is not supported yet. + +Flink CSV format uses [jackson databind API](https://github.com/FasterXML/jackson-databind) to parse and generate CSV string. + +The following table lists the type mapping from Flink type to CSV type. + ++ + + +Option +Required +Default +Type +Description ++ +format
required +(none) +String +Specify what format to use, here should be +'csv'.+ +csv.field-delimiter
optional +,String +Field delimiter character ( +','by default).+ +csv.line-delimiter
optional +\nString +Line delimiter, +\nby default. Note the\nand\rare invisible special characters, you have to use unicode to specify them in plain SQL. +-
+
- e.g.
'csv.line-delimiter' = U&'\\000D'specifies the to use carriage return\ras line delimiter.
+ - e.g.
'csv.line-delimiter' = U&'\\000A'specifies the to use line feed\nas line delimiter.
+
+ +csv.disable-quote-character
optional +false +Boolean +Disabled quote character for enclosing field values (false by default). + If true, option +'csv.quote-character'must be set.+ +csv.quote-character
optional +"String +Quote character for enclosing field values ( +"by default).+ +csv.allow-comments
optional +false +Boolean +Ignore comment lines that start with +'#'(disabled by default). + If enabled, make sure to also ignore parse errors to allow empty rows.+ +csv.ignore-parse-errors
optional +false +Boolean +Skip fields and rows with parse errors instead of failing. + Fields are set to null in case of errors. ++ +csv.array-element-delimiter
optional +;String +Array element delimiter string for separating + array and row element values ( +';'by default).+ +csv.escape-character
optional +(none) +String +Escape character for escaping values (disabled by default). ++ + +csv.null-literal
optional +(none) +String +Null literal string that is interpreted as a null value (disabled by default). ++ +
+ + + + + diff --git a/docs/dev/table/connectors/formats/csv.zh.md b/docs/dev/table/connectors/formats/csv.zh.md new file mode 100644 index 00000000000000..d63aab591e47a2 --- /dev/null +++ b/docs/dev/table/connectors/formats/csv.zh.md @@ -0,0 +1,247 @@ +--- +title: "CSV Format" +nav-title: CSV +nav-parent_id: sql-formats +nav-pos: 1 +--- + + +Format: Serialization Schema +Format: Deserialization Schema + +* This will be replaced by the TOC +{:toc} + +[CSV](https://zh.wikipedia.org/wiki/%E9%80%97%E5%8F%B7%E5%88%86%E9%9A%94%E5%80%BC) Format 允许我们基于 CSV schema 进行解析和生成 CSV 数据。 目前 CSV schema 是基于 table schema 推断而来的。 + +依赖 +------------ + + +为了建立CSV格式，下列的表格提供了为项目使用自动化工具（例如Maven或者SBT）以及SQL客户端使用SQL JAR包的依赖信息。 + +| Maven依赖 | SQL 客户端 JAR | +| :----------------- | :----------------------| +| `flink-csv` | 内置 | + +如何创建使用 CSV 格式的表 +---------------- + + +以下是一个使用 Kafka 连接器和 CSV 格式创建表的示例。 + ++ + + +Flink SQL type +CSV type ++ +CHAR / VARCHAR / STRINGstring+ +BOOLEANboolean+ +BINARY / VARBINARYstring with encoding: base64+ +DECIMALnumber+ +TINYINTnumber+ +SMALLINTnumber+ +INTnumber+ +BIGINTnumber+ +FLOATnumber+ +DOUBLEnumber+ +DATEstring with format: date+ +TIMEstring with format: time+ +TIMESTAMPstring with format: date-time+ +INTERVALnumber+ +ARRAYarray+ + +ROWobject++ +Format 参数 +---------------- + ++{% highlight sql %} +CREATE TABLE user_behavior ( + user_id BIGINT, + item_id BIGINT, + category_id BIGINT, + behavior STRING, + ts TIMESTAMP(3) +) WITH ( + 'connector' = 'kafka', + 'topic' = 'user_behavior', + 'properties.bootstrap.servers' = 'localhost:9092', + 'properties.group.id' = 'testGroup', + 'format' = 'csv', + 'csv.ignore-parse-errors' = 'true', + 'csv.allow-comments' = 'true' +) +{% endhighlight %} +++ +
+ +数据类型映射 +---------------- + +目前 CSV 的 schema 都是从 table schema 推断而来的。显式地定义 CSV schema 暂不支持。 +Flink 的 CSV Format 数据使用 [jackson databind API](https://github.com/FasterXML/jackson-databind) 去解析 CSV 字符串。 + +下面的表格列出了flink数据和CSV数据的对应关系。 + ++ + + +参数 +是否必选 +默认值 +类型 +描述 ++ +format
必选 +(none) +String +指定要使用的格式，这里应该是 +'csv'。+ +csv.field-delimiter
可选 +,String +字段分隔符 (默认 +',')。+ +csv.line-delimiter
可选 +\nString +行分隔符, 默认 +\n。注意\n和\r是不可见的特殊符号, 在显式的 SQL 语句中必须使用 unicode 编码。 +-
+
- 例如
'csv.line-delimiter' = U&'\\000D'使用换行符号\r作为行分隔符。
+ - 例如
'csv.line-delimiter' = U&'\\000A'使用换行符号\n作为行分隔符。
+
+ +csv.disable-quote-character
可选 +false +Boolean +关闭对引用的值使用引号 (默认是 false).如果允许，选项 +'csv.quote-character'必须被设置。+ +csv.quote-character
可选 +"String +用于围住字段值的引号字符 (默认 +").+ +csv.allow-comments
可选 +false +Boolean +是否允许忽略注释行（默认不允许），注释行以 +'#'作为起始字符。 + 如果允许注释行，请确保csv.ignore-parse-errors也开启了从而允许空行。 ++ +csv.ignore-parse-errors
可选 +false +Boolean +当解析异常时，是跳过当前字段或行，还是抛出错误失败（默认为 false，即抛出错误失败）。如果忽略字段的解析异常，则会将该字段值设置为 +null。+ +csv.array-element-delimiter
可选 +;String +分隔数组和行元素的字符串(默认 +';').+ +csv.escape-character
可选 +(none) +String +转义字符(默认关闭). ++ + +csv.null-literal
可选 +(none) +String +是否将 "null" 字符串转化为 null 值。 ++ +
diff --git a/docs/dev/table/connectors/formats/debezium.md b/docs/dev/table/connectors/formats/debezium.md new file mode 100644 index 00000000000000..98acf5f4621556 --- /dev/null +++ b/docs/dev/table/connectors/formats/debezium.md @@ -0,0 +1,207 @@ +--- +title: "Debezium Format" +nav-title: Debezium +nav-parent_id: sql-formats +nav-pos: 4 +--- + + +Changelog-Data-Capture Format +Format: Deserialization Schema + +* This will be replaced by the TOC +{:toc} + +[Debezium](https://debezium.io/) is a CDC (Changelog Data Capture) tool that can stream changes in real-time from MySQL, PostgreSQL, Oracle, Microsoft SQL Server and many other databases into Kafka. Debezium provides a unified format schema for changelog and supports to serialize messages using JSON and [Apache Avro](https://avro.apache.org/). + +Flink supports to interpret Debezium JSON messages as INSERT/UPDATE/DELETE messages into Flink SQL system. This is useful in many cases to leverage this feature, such as + - synchronizing incremental data from databases to other systems + - auditing logs + - real-time materialized views on databases + - temporal join changing history of a database table and so on. + +*Note: Support for interpreting Debezium Avro messages and emitting Debezium messages is on the roadmap.* + +Dependencies +------------ + +In order to setup the Debezium format, the following table provides dependency information for both projects using a build automation tool (such as Maven or SBT) and SQL Client with SQL JAR bundles. + +| Maven dependency | SQL Client JAR | +| :----------------- | :----------------------| +| `flink-json` | Built-in | + +*Note: please refer to [Debezium documentation](https://debezium.io/documentation/reference/1.1/index.html) about how to setup a Debezium Kafka Connect to synchronize changelog to Kafka topics.* + + +How to use Debezium format +---------------- + +Debezium provides a unified format for changelog, here is a simple example for an update operation captured from a MySQL `products` table: + +```json +{ + "before": { + "id": 111, + "name": "scooter", + "description": "Big 2-wheel scooter", + "weight": 5.18 + }, + "after": { + "id": 111, + "name": "scooter", + "description": "Big 2-wheel scooter", + "weight": 5.15 + }, + "source": {...}, + "op": "u", + "ts_ms": 1589362330904, + "transaction": null +} +``` + +*Note: please refer to [Debezium documentation](https://debezium.io/documentation/reference/1.1/connectors/mysql.html#mysql-connector-events_debezium) about the meaning of each fields.* + +The MySQL `products` table has 4 columns (`id`, `name`, `description` and `weight`). The above JSON message is an update change event on the `products` table where the `weight` value of the row with `id = 111` is changed from `5.18` to `5.15`. +Assuming this messages is synchronized to Kafka topic `products_binlog`, then we can use the following DDL to consume this topic and interpret the change events. + ++ + + +Flink SQL 类型 +CSV 类型 ++ +CHAR / VARCHAR / STRINGstring+ +BOOLEANboolean+ +BINARY / VARBINARYstring with encoding: base64+ +DECIMALnumber+ +TINYINTnumber+ +SMALLINTnumber+ +INTnumber+ +BIGINTnumber+ +FLOATnumber+ +DOUBLEnumber+ +DATEstring with format: date+ +TIMEstring with format: time+ +TIMESTAMPstring with format: date-time+ +INTERVALnumber+ +ARRAYarray+ + +ROWobject++ +In some cases, users may setup the Debezium Kafka Connect with the Kafka configuration `'value.converter.schemas.enable'` enabled to include schema in the message. Then the Debezium JSON message may look like this: + +```json +{ + "schema": {...}, + "payload": { + "before": { + "id": 111, + "name": "scooter", + "description": "Big 2-wheel scooter", + "weight": 5.18 + }, + "after": { + "id": 111, + "name": "scooter", + "description": "Big 2-wheel scooter", + "weight": 5.15 + }, + "source": {...}, + "op": "u", + "ts_ms": 1589362330904, + "transaction": null + } +} +``` + +In order to interpret such messages, you need to add the option `'debezium-json.schema-include' = 'true'` into above DDL WITH clause (`false` by default). Usually, this is not recommended to include schema because this makes the messages very verbose and reduces parsing performance. + +After registering the topic as a Flink table, then you can consume the Debezium messages as a changelog source. + ++{% highlight sql %} +CREATE TABLE topic_products ( + -- schema is totally the same to the MySQL "products" table + id BIGINT, + name STRING, + description STRING, + weight DECIMAL(10, 2) +) WITH ( + 'connector' = 'kafka', + 'topic' = 'products_binlog', + 'properties.bootstrap.servers' = 'localhost:9092', + 'properties.group.id' = 'testGroup', + 'format' = 'debezium-json' -- using debezium-json as the format +) +{% endhighlight %} ++++ + +Format Options +---------------- + ++{% highlight sql %} +-- a real-time materialized view on the MySQL "products" +-- which calculate the latest average of weight for the same products +SELECT name, AVG(weight) FROM topic_products GROUP BY name; + +-- synchronize all the data and incremental changes of MySQL "products" table to +-- Elasticsearch "products" index for future searching +INSERT INTO elasticsearch_products +SELECT * FROM topic_products; +{% endhighlight %} +++ +
+ +Data Type Mapping +---------------- + +Currently, the Debezium format uses JSON format for deserialization. Please refer to [JSON format documentation]({% link dev/table/connectors/formats/json.md %}#data-type-mapping) for more details about the data type mapping. + diff --git a/docs/dev/table/connectors/formats/debezium.zh.md b/docs/dev/table/connectors/formats/debezium.zh.md new file mode 100644 index 00000000000000..8ffdbd21200614 --- /dev/null +++ b/docs/dev/table/connectors/formats/debezium.zh.md @@ -0,0 +1,205 @@ +--- +title: "Debezium Format" +nav-title: Debezium +nav-parent_id: sql-formats +nav-pos: 4 +--- + + +Changelog-Data-Capture Format +Format: Deserialization Schema + +* This will be replaced by the TOC +{:toc} + +[Debezium](https://debezium.io/) 是一个 CDC（Changelog Data Capture，变更数据捕获）的工具，可以把来自 MySQL、PostgreSQL、Oracle、Microsoft SQL Server 和许多其他数据库的更改实时流式传输到 Kafka 中。 Debezium 为变更日志提供了统一的格式结构，并支持使用 JSON 和 Apache Avro 序列化消息。 + +Flink 支持将 Debezium JSON 消息解析为 INSERT / UPDATE / DELETE 消息到 Flink SQL 系统中。在很多情况下，利用这个特性非常的有用，例如 + - 将增量数据从数据库同步到其他系统 + - 日志审计 + - 数据库的实时物化视图 + - 关联维度数据库的变更历史，等等。 + +*注意: 支持解析 Debezium Avro 消息和输出 Debezium 消息已经规划在路线图上了。* + +依赖 +------------ + +为了设置 Debezium Format，下表提供了使用构建自动化工具（例如 Maven 或 SBT）和带有 SQL JAR 包的 SQL Client 的两个项目的依赖项信息。 + +| Maven 依赖 | SQL Client JAR | +| :----------------- | :----------------------| +| `flink-json` | 内置 | + +*注意: 请参考 [Debezium 文档](https://debezium.io/documentation/reference/1.1/index.html)，了解如何设置 Debezium Kafka Connect 用来将变更日志同步到 Kafka 主题。* + + +如何使用 Debezium Format +---------------- + + +Debezium 为变更日志提供了统一的格式，这是一个从 MySQL product 表捕获的更新操作的简单示例: + +```json +{ + "before": { + "id": 111, + "name": "scooter", + "description": "Big 2-wheel scooter", + "weight": 5.18 + }, + "after": { + "id": 111, + "name": "scooter", + "description": "Big 2-wheel scooter", + "weight": 5.15 + }, + "source": {...}, + "op": "u", + "ts_ms": 1589362330904, + "transaction": null +} +``` + +*注意: 请参考 [Debezium 文档](https://debezium.io/documentation/reference/1.1/connectors/mysql.html#mysql-connector-events_debezium)，了解每个字段的含义。* + +MySQL 产品表有4列（`id`、`name`、`description`、`weight`）。上面的 JSON 消息是 `products` 表上的一条更新事件，其中 `id = 111` 的行的 `weight` 值从 `5.18` 更改为 `5.15`。假设此消息已同步到 Kafka 主题 `products_binlog`，则可以使用以下 DDL 来使用此主题并解析更改事件。 + ++ + + +Option +Required +Default +Type +Description ++ +format
required +(none) +String +Specify what format to use, here should be +'debezium-json'.+ +debezium-json.schema-include
optional +false +Boolean +When setting up a Debezium Kafka Connect, users may enable a Kafka configuration +'value.converter.schemas.enable'to include schema in the message. + This option indicates whether the Debezium JSON message includes the schema or not.+ +debezium-json.ignore-parse-errors
optional +false +Boolean +Skip fields and rows with parse errors instead of failing. + Fields are set to null in case of errors. ++ + +debezium-json.timestamp-format.standard
optional +'SQL'String +Specify the input and output timestamp format. Currently supported values are +'SQL'and'ISO-8601': +-
+
- Option
'SQL'will parse input timestamp in "yyyy-MM-dd HH:mm:ss.s{precision}" format, e.g '2020-12-30 12:13:14.123' and output timestamp in the same format.
+ - Option
'ISO-8601'will parse input timestamp in "yyyy-MM-ddTHH:mm:ss.s{precision}" format, e.g '2020-12-30T12:13:14.123' and output timestamp in the same format.
+
++ +在某些情况下，用户在设置 Debezium Kafka Connect 时，可能会开启 Kafka 的配置 `'value.converter.schemas.enable'`，用来在消息体中包含 schema 信息。然后，Debezium JSON 消息可能如下所示: + +```json +{ + "schema": {...}, + "payload": { + "before": { + "id": 111, + "name": "scooter", + "description": "Big 2-wheel scooter", + "weight": 5.18 + }, + "after": { + "id": 111, + "name": "scooter", + "description": "Big 2-wheel scooter", + "weight": 5.15 + }, + "source": {...}, + "op": "u", + "ts_ms": 1589362330904, + "transaction": null + } +} +``` + +为了解析这一类信息，你需要在上述 DDL WITH 子句中添加选项 `'debezium-json.schema-include' = 'true'`（默认为 false）。通常情况下，建议不要包含 schema 的描述，因为这样会使消息变得非常冗长，并降低解析性能。 + +在将主题注册为 Flink 表之后，可以将 Debezium 消息用作变更日志源。 + ++{% highlight sql %} +CREATE TABLE topic_products ( + -- schema 与 MySQL 的 products 表完全相同 + id BIGINT, + name STRING, + description STRING, + weight DECIMAL(10, 2) +) WITH ( + 'connector' = 'kafka', + 'topic' = 'products_binlog', + 'properties.bootstrap.servers' = 'localhost:9092', + 'properties.group.id' = 'testGroup', + 'format' = 'debezium-json' -- 使用 debezium-json 作为 format +) +{% endhighlight %} ++++ + +Format 参数 +---------------- + ++{% highlight sql %} +-- MySQL "products" 的实时物化视图 +-- 计算相同产品的最新平均重量 +SELECT name, AVG(weight) FROM topic_products GROUP BY name; + +-- 将 MySQL "products" 表的所有数据和增量更改同步到 +-- Elasticsearch "products" 索引，供将来查找 +INSERT INTO elasticsearch_products +SELECT * FROM topic_products; +{% endhighlight %} +++ +
+ +数据类型映射 +---------------- + +目前，Debezium Format 使用 JSON Format 进行反序列化。有关数据类型映射的更多详细信息，请参考 [JSON Format 文档]({% link dev/table/connectors/formats/json.zh.md %}#data-type-mapping)。 + diff --git a/docs/dev/table/connectors/formats/index.md b/docs/dev/table/connectors/formats/index.md new file mode 100644 index 00000000000000..66f9bc87bc262c --- /dev/null +++ b/docs/dev/table/connectors/formats/index.md @@ -0,0 +1,72 @@ +--- +title: "Formats" +nav-id: sql-formats +nav-parent_id: sql-connectors +nav-pos: 1 +nav-show_overview: true +--- + + +Flink provides a set of table formats that can be used with table connectors. A table format is a storage format defines how to map binary data onto table columns. + +Flink supports the following formats: + ++ + + +参数 +是否必选 +默认值 +类型 +描述 ++ +format
必选 +(none) +String +指定要使用的格式，此处应为 +'debezium-json'。+ +debezium-json.schema-include
可选 +false +Boolean +设置 Debezium Kafka Connect 时，用户可以启用 Kafka 配置 +'value.converter.schemas.enable'以在消息中包含 schema。此选项表明 Debezium JSON 消息是否包含 schema。+ +debezium-json.ignore-parse-errors
可选 +false +Boolean +当解析异常时，是跳过当前字段或行，还是抛出错误失败（默认为 false，即抛出错误失败）。如果忽略字段的解析异常，则会将该字段值设置为 +null。+ + +debezium-json.timestamp-format.standard
可选 +'SQL'String +声明输入和输出的时间戳格式。当前支持的格式为 +'SQL'以及'ISO-8601'： +-
+
- 可选参数
'SQL'将会以 "yyyy-MM-dd HH:mm:ss.s{precision}" 的格式解析时间戳, 例如 '2020-12-30 12:13:14.123'，且会以相同的格式输出。
+ - 可选参数
'ISO-8601'将会以 "yyyy-MM-ddTHH:mm:ss.s{precision}" 的格式解析输入时间戳, 例如 '2020-12-30T12:13:14.123' ，且会以相同的格式输出。
+
+ +
\ No newline at end of file diff --git a/docs/dev/table/connectors/formats/index.zh.md b/docs/dev/table/connectors/formats/index.zh.md new file mode 100644 index 00000000000000..a86add9d00afb4 --- /dev/null +++ b/docs/dev/table/connectors/formats/index.zh.md @@ -0,0 +1,72 @@ +--- +title: "Formats" +nav-id: sql-formats +nav-parent_id: sql-connectors +nav-pos: 1 +nav-show_overview: true +--- + + +Flink 提供了一套与表连接器（table connector）一起使用的表格式（table format）。表格式是一种存储格式，定义了如何把二进制数据映射到表的列上。 + +Flink 支持以下格式： + ++ + + +Formats +Supported Connectors ++ +CSV +Apache Kafka, + Filesystem ++ +JSON +Apache Kafka, + Filesystem, + Elasticsearch ++ +Apache Avro +Apache Kafka, + Filesystem ++ +Debezium CDC +Apache Kafka ++ +Canal CDC +Apache Kafka ++ +Apache Parquet +Filesystem ++ + +Apache ORC +Filesystem ++ +
diff --git a/docs/dev/table/connectors/formats/json.md b/docs/dev/table/connectors/formats/json.md new file mode 100644 index 00000000000000..181f48e291186c --- /dev/null +++ b/docs/dev/table/connectors/formats/json.md @@ -0,0 +1,212 @@ +--- +title: "JSON Format" +nav-title: JSON +nav-parent_id: sql-formats +nav-pos: 2 +--- + + +Format: Serialization Schema +Format: Deserialization Schema + +* This will be replaced by the TOC +{:toc} + +The [JSON](https://www.json.org/json-en.html) format allows to read and write JSON data based on an JSON schema. Currently, the JSON schema is derived from table schema. + +Dependencies +------------ + +In order to setup the JSON format, the following table provides dependency information for both projects using a build automation tool (such as Maven or SBT) and SQL Client with SQL JAR bundles. + +| Maven dependency | SQL Client JAR | +| :----------------- | :----------------------| +| `flink-json` | Built-in | + +How to create a table with JSON format +---------------- + +Here is an example to create a table using Kafka connector and JSON format. + ++ + + +格式 +支持的连接器 ++ +CSV +Apache Kafka, + Filesystem ++ +JSON +Apache Kafka, + Filesystem, + Elasticsearch ++ +Apache Avro +Apache Kafka, + Filesystem ++ +Debezium CDC +Apache Kafka ++ +Canal CDC +Apache Kafka ++ +Apache Parquet +Filesystem ++ + +Apache ORC +Filesystem +++ +Format Options +---------------- + ++{% highlight sql %} +CREATE TABLE user_behavior ( + user_id BIGINT, + item_id BIGINT, + category_id BIGINT, + behavior STRING, + ts TIMESTAMP(3) +) WITH ( + 'connector' = 'kafka', + 'topic' = 'user_behavior', + 'properties.bootstrap.servers' = 'localhost:9092', + 'properties.group.id' = 'testGroup', + 'format' = 'json', + 'json.fail-on-missing-field' = 'false', + 'json.ignore-parse-errors' = 'true' +) +{% endhighlight %} +++ +
+ +Data Type Mapping +---------------- + +Currently, the JSON schema is always derived from table schema. Explicitly defining an JSON schema is not supported yet. + +Flink JSON format uses [jackson databind API](https://github.com/FasterXML/jackson-databind) to parse and generate JSON string. + +The following table lists the type mapping from Flink type to JSON type. + ++ + + +Option +Required +Default +Type +Description ++ +format
required +(none) +String +Specify what format to use, here should be +'json'.+ +json.fail-on-missing-field
optional +false +Boolean +Whether to fail if a field is missing or not. ++ +json.ignore-parse-errors
optional +false +Boolean +Skip fields and rows with parse errors instead of failing. + Fields are set to null in case of errors. ++ + +json.timestamp-format.standard
optional +'SQL'String +Specify the input and output timestamp format. Currently supported values are +'SQL'and'ISO-8601': +-
+
- Option
'SQL'will parse input timestamp in "yyyy-MM-dd HH:mm:ss.s{precision}" format, e.g '2020-12-30 12:13:14.123' and output timestamp in the same format.
+ - Option
'ISO-8601'will parse input timestamp in "yyyy-MM-ddTHH:mm:ss.s{precision}" format, e.g '2020-12-30T12:13:14.123' and output timestamp in the same format.
+
+ +
+ + + + + diff --git a/docs/dev/table/connectors/formats/json.zh.md b/docs/dev/table/connectors/formats/json.zh.md new file mode 100644 index 00000000000000..cf8f2863f921c1 --- /dev/null +++ b/docs/dev/table/connectors/formats/json.zh.md @@ -0,0 +1,211 @@ +--- +title: "JSON Format" +nav-title: JSON +nav-parent_id: sql-formats +nav-pos: 2 +--- + + +Format: Serialization Schema +Format: Deserialization Schema + +* This will be replaced by the TOC +{:toc} + +[JSON](https://www.json.org/json-en.html) Format 能读写 JSON 格式的数据。当前，JSON schema 是从 table schema 中自动推导而得的。 + +依赖 +------------ + +为了使用 JSON format，下表列出了利用自动化构建工具（例如，Maven 或者 SBT ）构建项目以及 SQL Client 所需要的依赖。 + +| Maven 依赖 | SQL Client JAR | +| :----------------- | :----------------------| +| `flink-json` | 内置 | + +如何创建一张基于 JSON Format 的表 +---------------- + +以下是一个利用 Kafka 以及 JSON Format 构建表的例子。 + ++ + + +Flink SQL type +JSON type ++ +CHAR / VARCHAR / STRINGstring+ +BOOLEANboolean+ +BINARY / VARBINARYstring with encoding: base64+ +DECIMALnumber+ +TINYINTnumber+ +SMALLINTnumber+ +INTnumber+ +BIGINTnumber+ +FLOATnumber+ +DOUBLEnumber+ +DATEstring with format: date+ +TIMEstring with format: time+ +TIMESTAMPstring with format: date-time+ +INTERVALnumber+ +ARRAYarray+ +MAP / MULTISETobject+ + +ROWobject++ +Format 参数 +---------------- + ++{% highlight sql %} +CREATE TABLE user_behavior ( + user_id BIGINT, + item_id BIGINT, + category_id BIGINT, + behavior STRING, + ts TIMESTAMP(3) +) WITH ( + 'connector' = 'kafka', + 'topic' = 'user_behavior', + 'properties.bootstrap.servers' = 'localhost:9092', + 'properties.group.id' = 'testGroup', + 'format' = 'json', + 'json.fail-on-missing-field' = 'false', + 'json.ignore-parse-errors' = 'true' +) +{% endhighlight %} +++ +
+ +数据类型映射关系 +---------------- + +当前，JSON schema 将会自动从 table schema 之中自动推导得到。不支持显式地定义 JSON schema。 + +在 Flink 中，JSON Format 使用 [jackson databind API](https://github.com/FasterXML/jackson-databind) 去解析和生成 JSON。 + +下表列出了 Flink 中的数据类型与 JSON 中的数据类型的映射关系。 + ++ + + +参数 +是否必须 +默认值 +类型 +描述 ++ +format
必选 +(none) +String +声明使用的格式，这里应为 +'json'。+ +json.fail-on-missing-field
可选 +false +Boolean +当解析字段缺失时，是跳过当前字段或行，还是抛出错误失败（默认为 false，即抛出错误失败）。 ++ +json.ignore-parse-errors
可选 +false +Boolean +当解析异常时，是跳过当前字段或行，还是抛出错误失败（默认为 false，即抛出错误失败）。如果忽略字段的解析异常，则会将该字段值设置为 +null。+ + +json.timestamp-format.standard
可选 +'SQL'String +声明输入和输出的时间戳格式。当前支持的格式为 +'SQL'以及'ISO-8601'： +-
+
- 可选参数
'SQL'将会以 "yyyy-MM-dd HH:mm:ss.s{precision}" 的格式解析时间戳, 例如 '2020-12-30 12:13:14.123'，且会以相同的格式输出。
+ - 可选参数
'ISO-8601'将会以 "yyyy-MM-ddTHH:mm:ss.s{precision}" 的格式解析输入时间戳, 例如 '2020-12-30T12:13:14.123' ，且会以相同的格式输出。
+
+ +
+ + + + + diff --git a/docs/dev/table/connectors/formats/orc.md b/docs/dev/table/connectors/formats/orc.md new file mode 100644 index 00000000000000..4c878a43467194 --- /dev/null +++ b/docs/dev/table/connectors/formats/orc.md @@ -0,0 +1,183 @@ +--- +title: "Orc Format" +nav-title: Orc +nav-parent_id: sql-formats +nav-pos: 6 +--- + + +Format: Serialization Schema +Format: Deserialization Schema + +* This will be replaced by the TOC +{:toc} + +The [Apache Orc](https://orc.apache.org/) format allows to read and write Orc data. + +Dependencies +------------ + +In order to setup the Orc format, the following table provides dependency information for both +projects using a build automation tool (such as Maven or SBT) and SQL Client with SQL JAR bundles. + +| Maven dependency | SQL Client JAR | +| :----------------- | :----------------------| +| flink-orc{{site.scala_version_suffix}} |{% if site.is_stable %}[Download](https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-orc{{site.scala_version_suffix}}/{{site.version}}/flink-sql-orc{{site.scala_version_suffix}}-{{site.version}}.jar) {% else %} Only available for stable releases {% endif %}| + +How to create a table with Orc format +---------------- + +Here is an example to create a table using Filesystem connector and Orc format. + ++ + + +Flink SQL 类型 +JSON 类型 ++ +CHAR / VARCHAR / STRINGstring+ +BOOLEANboolean+ +BINARY / VARBINARYstring with encoding: base64+ +DECIMALnumber+ +TINYINTnumber+ +SMALLINTnumber+ +INTnumber+ +BIGINTnumber+ +FLOATnumber+ +DOUBLEnumber+ +DATEstring with format: date+ +TIMEstring with format: time+ +TIMESTAMPstring with format: date-time+ +INTERVALnumber+ +ARRAYarray+ +MAP / MULTISETobject+ + +ROWobject++ +Format Options +---------------- + ++{% highlight sql %} +CREATE TABLE user_behavior ( + user_id BIGINT, + item_id BIGINT, + category_id BIGINT, + behavior STRING, + ts TIMESTAMP(3), + dt STRING +) PARTITIONED BY (dt) WITH ( + 'connector' = 'filesystem', + 'path' = '/tmp/user_behavior', + 'format' = 'orc' +) +{% endhighlight %} +++ +
+ +Orc format also supports table properties from [Table properties](https://orc.apache.org/docs/hive-config.html#table-properties). +For example, you can configure `orc.compress=SNAPPY` to enable snappy compression. + +Data Type Mapping +---------------- + +Orc format type mapping is compatible with Apache Hive. +The following table lists the type mapping from Flink type to Orc type. + ++ + + +Option +Required +Default +Type +Description ++ + +format
required +(none) +String +Specify what format to use, here should be 'orc'. ++ +
+ +Attention Composite data type: Array, Map and Row are not supported. diff --git a/docs/dev/table/connectors/formats/orc.zh.md b/docs/dev/table/connectors/formats/orc.zh.md new file mode 100644 index 00000000000000..03b73f8822eb71 --- /dev/null +++ b/docs/dev/table/connectors/formats/orc.zh.md @@ -0,0 +1,180 @@ +--- +title: "Orc Format" +nav-title: Orc +nav-parent_id: sql-formats +nav-pos: 6 +--- + + +Format: Serialization Schema +Format: Deserialization Schema + +* This will be replaced by the TOC {:toc} +{:toc} + +[Apache Orc](https://orc.apache.org/) Format 允许读写 ORC 数据。 + +依赖 +------------ + +为了建立Orc格式，下列的表格提供了为项目使用自动化工具（例如Maven或者SBT）以及SQL客户端使用SQL JAR包的依赖信息。 + +| Maven 依赖 | SQL 客户端 JAR | +| :----------------- | :----------------------| +| flink-orc{{site.scala_version_suffix}} |{% if site.is_stable %}[Download](https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-orc{{site.scala_version_suffix}}/{{site.version}}/flink-sql-orc{{site.scala_version_suffix}}-{{site.version}}.jar) {% else %} Only available for stable releases {% endif %}| + + +如何用 Orc 格式创建一个表格 +---------------- + +下面是一个用 Filesystem connector 和 Orc format 创建表格的例子 + ++ + + +Flink Data Type +Orc physical type +Orc logical type ++ +CHAR +bytes +CHAR ++ +VARCHAR +bytes +VARCHAR ++ +STRING +bytes +STRING ++ +BOOLEAN +long +BOOLEAN ++ +BYTES +bytes +BINARY ++ +DECIMAL +decimal +DECIMAL ++ +TINYINT +long +BYTE ++ +SMALLINT +long +SHORT ++ +INT +long +INT ++ +BIGINT +long +LONG ++ +FLOAT +double +FLOAT ++ +DOUBLE +double +DOUBLE ++ +DATE +long +DATE ++ + +TIMESTAMP +timestamp +TIMESTAMP +++ +Format 参数 +---------------- + ++{% highlight sql %} +CREATE TABLE user_behavior ( + user_id BIGINT, + item_id BIGINT, + category_id BIGINT, + behavior STRING, + ts TIMESTAMP(3), + dt STRING +) PARTITIONED BY (dt) WITH ( + 'connector' = 'filesystem', + 'path' = '/tmp/user_behavior', + 'format' = 'orc' +) +{% endhighlight %} +++ +
+ +Orc 格式也支持来源于 [Table properties](https://orc.apache.org/docs/hive-config.html#table-properties) 的表属性。 举个例子，你可以设置 `orc.compress=SNAPPY` 来允许spappy压缩。 + +数据类型映射 +---------------- + +Orc 格式类型的映射和 Apache Hive 是兼容的。下面的表格列出了 Flink 类型的数据和 Orc 类型的数据的映射关系。 ++ + + +参数 +是否必选 +默认值 +类型 +描述 ++ + +format
必选 +(none) +String +指定要使用的格式，这里应该是 'orc'。 ++ +
+ +注意 复合数据类型: 数组、 映射和行类型暂不支持。 diff --git a/docs/dev/table/connectors/formats/parquet.md b/docs/dev/table/connectors/formats/parquet.md new file mode 100644 index 00000000000000..c6f94cde8f0a43 --- /dev/null +++ b/docs/dev/table/connectors/formats/parquet.md @@ -0,0 +1,189 @@ +--- +title: "Parquet Format" +nav-title: Parquet +nav-parent_id: sql-formats +nav-pos: 5 +--- + + +Format: Serialization Schema +Format: Deserialization Schema + +* This will be replaced by the TOC +{:toc} + +The [Apache Parquet](https://parquet.apache.org/) format allows to read and write Parquet data. + +Dependencies +------------ + +In order to setup the Parquet format, the following table provides dependency information for both +projects using a build automation tool (such as Maven or SBT) and SQL Client with SQL JAR bundles. + +| Maven dependency | SQL Client JAR | +| :----------------- | :----------------------| +| flink-parquet{{site.scala_version_suffix}} |{% if site.is_stable %}[Download](https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-parquet{{site.scala_version_suffix}}/{{site.version}}/flink-sql-parquet{{site.scala_version_suffix}}-{{site.version}}.jar) {% else %} Only available for stable releases {% endif %}| + +How to create a table with Parquet format +---------------- + +Here is an example to create a table using Filesystem connector and Parquet format. + ++ + + +Flink 数据类型 +Orc 物理类型 +Orc 逻辑类型 ++ +CHAR +bytes +CHAR ++ +VARCHAR +bytes +VARCHAR ++ +STRING +bytes +STRING ++ +BOOLEAN +long +BOOLEAN ++ +BYTES +bytes +BINARY ++ +DECIMAL +decimal +DECIMAL ++ +TINYINT +long +BYTE ++ +SMALLINT +long +SHORT ++ +INT +long +INT ++ +BIGINT +long +LONG ++ +FLOAT +double +FLOAT ++ +DOUBLE +double +DOUBLE ++ +DATE +long +DATE ++ + +TIMESTAMP +timestamp +TIMESTAMP +++ +Format Options +---------------- + ++{% highlight sql %} +CREATE TABLE user_behavior ( + user_id BIGINT, + item_id BIGINT, + category_id BIGINT, + behavior STRING, + ts TIMESTAMP(3), + dt STRING +) PARTITIONED BY (dt) WITH ( + 'connector' = 'filesystem', + 'path' = '/tmp/user_behavior', + 'format' = 'parquet' +) +{% endhighlight %} +++ +
+ +Parquet format also supports configuration from [ParquetOutputFormat](https://www.javadoc.io/doc/org.apache.parquet/parquet-hadoop/1.10.0/org/apache/parquet/hadoop/ParquetOutputFormat.html). +For example, you can configure `parquet.compression=GZIP` to enable gzip compression. + +Data Type Mapping +---------------- + +Currently, Parquet format type mapping is compatible with Apache Hive, but different with Apache Spark: + +- Timestamp: mapping timestamp type to int96 whatever the precision is. +- Decimal: mapping decimal type to fixed length byte array according to the precision. + +The following table lists the type mapping from Flink type to Parquet type. + ++ + + +Option +Required +Default +Type +Description ++ +format
required +(none) +String +Specify what format to use, here should be 'parquet'. ++ + +parquet.utc-timezone
optional +false +Boolean +Use UTC timezone or local timezone to the conversion between epoch time and LocalDateTime. Hive 0.x/1.x/2.x use local timezone. But Hive 3.x use UTC timezone. ++ +
+ +Attention Composite data type: Array, Map and Row are not supported. diff --git a/docs/dev/table/connectors/formats/parquet.zh.md b/docs/dev/table/connectors/formats/parquet.zh.md new file mode 100644 index 00000000000000..e6a4876ca7ed59 --- /dev/null +++ b/docs/dev/table/connectors/formats/parquet.zh.md @@ -0,0 +1,189 @@ +--- +title: "Parquet Format" +nav-title: Parquet +nav-parent_id: sql-formats +nav-pos: 5 +--- + + +Format: Serialization Schema +Format: Deserialization Schema + +* This will be replaced by the TOC +{:toc} + +The [Apache Parquet](https://parquet.apache.org/) format allows to read and write Parquet data. + +Dependencies +------------ + +In order to setup the Parquet format, the following table provides dependency information for both +projects using a build automation tool (such as Maven or SBT) and SQL Client with SQL JAR bundles. + +| Maven dependency | SQL Client JAR | +| :----------------- | :----------------------| +| flink-parquet{{site.scala_version_suffix}} |{% if site.is_stable %}[Download](https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-parquet{{site.scala_version_suffix}}/{{site.version}}/flink-sql-parquet{{site.scala_version_suffix}}-{{site.version}}.jar) {% else %} Only available for stable releases {% endif %}| + +How to create a table with Parquet format +---------------- + +Here is an example to create a table using Filesystem connector and Parquet format. + ++ + + +Flink Data Type +Parquet type +Parquet logical type ++ +CHAR / VARCHAR / STRING +BINARY +UTF8 ++ +BOOLEAN +BOOLEAN ++ + +BINARY / VARBINARY +BINARY ++ + +DECIMAL +FIXED_LEN_BYTE_ARRAY +DECIMAL ++ +TINYINT +INT32 +INT_8 ++ +SMALLINT +INT32 +INT_16 ++ +INT +INT32 ++ + +BIGINT +INT64 ++ + +FLOAT +FLOAT ++ + +DOUBLE +DOUBLE ++ + +DATE +INT32 +DATE ++ +TIME +INT32 +TIME_MILLIS ++ + +TIMESTAMP +INT96 ++ ++ +Format Options +---------------- + ++{% highlight sql %} +CREATE TABLE user_behavior ( + user_id BIGINT, + item_id BIGINT, + category_id BIGINT, + behavior STRING, + ts TIMESTAMP(3), + dt STRING +) PARTITIONED BY (dt) WITH ( + 'connector' = 'filesystem', + 'path' = '/tmp/user_behavior', + 'format' = 'parquet' +) +{% endhighlight %} +++ +
+ +Parquet format also supports configuration from [ParquetOutputFormat](https://www.javadoc.io/doc/org.apache.parquet/parquet-hadoop/1.10.0/org/apache/parquet/hadoop/ParquetOutputFormat.html). +For example, you can configure `parquet.compression=GZIP` to enable gzip compression. + +Data Type Mapping +---------------- + +Currently, Parquet format type mapping is compatible with Apache Hive, but different with Apache Spark: + +- Timestamp: mapping timestamp type to int96 whatever the precision is. +- Decimal: mapping decimal type to fixed length byte array according to the precision. + +The following table lists the type mapping from Flink type to Parquet type. + ++ + + +Option +Required +Default +Type +Description ++ +format
required +(none) +String +Specify what format to use, here should be 'parquet'. ++ + +parquet.utc-timezone
optional +false +Boolean +Use UTC timezone or local timezone to the conversion between epoch time and LocalDateTime. Hive 0.x/1.x/2.x use local timezone. But Hive 3.x use UTC timezone. ++ +
+ +Attention Composite data type: Array, Map and Row are not supported. diff --git a/docs/dev/table/connectors/hbase.md b/docs/dev/table/connectors/hbase.md new file mode 100644 index 00000000000000..2fab3ebf9baff1 --- /dev/null +++ b/docs/dev/table/connectors/hbase.md @@ -0,0 +1,303 @@ +--- +title: "HBase SQL Connector" +nav-title: HBase +nav-parent_id: sql-connectors +nav-pos: 6 +--- + + +Scan Source: Bounded +Lookup Source: Sync Mode +Sink: Batch +Sink: Streaming Upsert Mode + +* This will be replaced by the TOC +{:toc} + +The HBase connector allows for reading from and writing to an HBase cluster. This document describes how to setup the HBase Connector to run SQL queries against HBase. + +HBase always works in upsert mode for exchange changelog messages with the external system using a primary key defined on the DDL. The primary key must be defined on the HBase rowkey field (rowkey field must be declared). If the PRIMARY KEY clause is not declared, the HBase connector will take rowkey as the primary key by default. + +Dependencies +------------ + +In order to setup the HBase connector, the following table provide dependency information for both projects using a build automation tool (such as Maven or SBT) and SQL Client with SQL JAR bundles. + +| HBase Version | Maven dependency | SQL Client JAR | +| :------------------ | :-------------------------------------------------------- | :----------------------| +| 1.4.x | `flink-connector-hbase{{site.scala_version_suffix}}` | Unsupported | + +*Note: To use HBase connector in SQL Client or Flink cluster, it's highly recommended to add HBase dependency jars to Hadoop classpath. Flink will load all jars under Hadoop classpath automatically, please refer to [HBase, MapReduce, and the CLASSPATH](https://hbase.apache.org/book.html#hbase.mapreduce.classpath) about how to add HBase dependency jars to Hadoop classpath.* + + +How to use HBase table +---------------- + +All the column families in HBase table must be declared as ROW type, the field name maps to the column family name, and the nested field names map to the column qualifier names. There is no need to declare all the families and qualifiers in the schema, users can declare what’s used in the query. Except the ROW type fields, the single atomic type field (e.g. STRING, BIGINT) will be recognized as HBase rowkey. The rowkey field can be arbitrary name, but should be quoted using backticks if it is a reserved keyword. + ++ + + +Flink Data Type +Parquet type +Parquet logical type ++ +CHAR / VARCHAR / STRING +BINARY +UTF8 ++ +BOOLEAN +BOOLEAN ++ + +BINARY / VARBINARY +BINARY ++ + +DECIMAL +FIXED_LEN_BYTE_ARRAY +DECIMAL ++ +TINYINT +INT32 +INT_8 ++ +SMALLINT +INT32 +INT_16 ++ +INT +INT32 ++ + +BIGINT +INT64 ++ + +FLOAT +FLOAT ++ + +DOUBLE +DOUBLE ++ + +DATE +INT32 +DATE ++ +TIME +INT32 +TIME_MILLIS ++ + +TIMESTAMP +INT96 ++ ++ +Connector Options +---------------- + ++{% highlight sql %} +-- register the HBase table 'mytable' in Flink SQL +CREATE TABLE hTable ( + rowkey INT, + family1 ROW+, + family2 ROW , + family3 ROW , + PRIMARY KEY (rowkey) NOT ENFORCED +) WITH ( + 'connector' = 'hbase-1.4', + 'table-name' = 'mytable', + 'zookeeper.quorum' = 'localhost:2181' +); + +-- use ROW(...) construction function construct column families and write data into the HBase table. +-- assuming the schema of "T" is [rowkey, f1q1, f2q2, f2q3, f3q4, f3q5, f3q6] +INSERT INTO hTable +SELECT rowkey, ROW(f1q1), ROW(f2q2, f2q3), ROW(f3q4, f3q5, f3q6) FROM T; + +-- scan data from the HBase table +SELECT rowkey, family1, family3.q4, family3.q6 FROM hTable; + +-- temporal join the HBase table as a dimension table +SELECT * FROM myTopic +LEFT JOIN hTable FOR SYSTEM_TIME AS OF myTopic.proctime +ON myTopic.key = hTable.rowkey; +{% endhighlight %} + + +
+ + + +Data Type Mapping +---------------- + +HBase stores all data as byte arrays. The data needs to be serialized and deserialized during read and write operation + +When serializing and de-serializing, Flink HBase connector uses utility class `org.apache.hadoop.hbase.util.Bytes` provided by HBase (Hadoop) to convert Flink Data Types to and from byte arrays. + +Flink HBase connector encodes `null` values to empty bytes, and decode empty bytes to `null` values for all data types except string type. For string type, the null literal is determined by `null-string-literal` option. + +The data type mappings are as follows: + ++ + + +Option +Required +Default +Type +Description ++ +connector
required +(none) +String +Specify what connector to use, here should be +'hbase-1.4'.+ +table-name
required +(none) +String +The name of HBase table to connect. ++ +zookeeper.quorum
required +(none) +String +The HBase Zookeeper quorum. ++ +zookeeper.znode.parent
optional +/hbase +String +The root dir in Zookeeper for HBase cluster. ++ +null-string-literal
optional +null +String +Representation for null values for string fields. HBase source and sink encodes/decodes empty bytes as null values for all types except string type. ++ +sink.buffer-flush.max-size
optional +2mb +MemorySize +Writing option, maximum size in memory of buffered rows for each writing request. + This can improve performance for writing data to HBase database, but may increase the latency. + Can be set to +'0'to disable it. ++ +sink.buffer-flush.max-rows
optional +1000 +Integer +Writing option, maximum number of rows to buffer for each writing request. + This can improve performance for writing data to HBase database, but may increase the latency. + Can be set to +'0'to disable it. ++ + +sink.buffer-flush.interval
optional +1s +Duration +Writing option, the interval to flush any buffered rows. + This can improve performance for writing data to HBase database, but may increase the latency. + Can be set to +'0'to disable it. Note, both'sink.buffer-flush.max-size'and'sink.buffer-flush.max-rows'+ can be set to'0'with the flush interval set allowing for complete async processing of buffered actions. ++ +
+ +{% top %} \ No newline at end of file diff --git a/docs/dev/table/connectors/hbase.zh.md b/docs/dev/table/connectors/hbase.zh.md new file mode 100644 index 00000000000000..7a738fb823041d --- /dev/null +++ b/docs/dev/table/connectors/hbase.zh.md @@ -0,0 +1,296 @@ +--- +title: "HBase SQL 连接器" +nav-title: HBase +nav-parent_id: sql-connectors +nav-pos: 6 +--- + + +Scan Source: Bounded +Lookup Source: Sync Mode +Sink: Batch +Sink: Streaming Upsert Mode + +* This will be replaced by the TOC +{:toc} + +HBase 连接器支持读取和写入 HBase 集群。本文档介绍如何使用 HBase 连接器基于 HBase 进行 SQL 查询。 + +HBase 连接器在 upsert 模式下运行，可以使用 DDL 中定义的主键与外部系统交换更新操作消息。但是主键只能基于 HBase 的 rowkey 字段定义。如果没有声明主键，HBase 连接器默认取 rowkey 作为主键。 + +依赖 +------------ + +安装 HBase 连接器的依赖条件如下表，包括自动构建工具（比如 Maven 或者 SBT）和 SQL 客户端： + +| HBase 版本 | Maven 依赖 | SQL 客户端 JAR | +| :------------------ | :-------------------------------------------------------- | :----------------------| +| 1.4.x | `flink-connector-hbase{{site.scala_version_suffix}}` | 不支持| + +*注意: 在 SQL Client 或者 Flink 集群中使用 HBase 连接器时，推荐将 HBase 依赖的 jar 包添加到 Hadoop 的 classpath。Flink 会自动加载 Hadoop classpath 下的所有 jar, 请参阅 [HBase, MapReduce 和 CLASSPATH](https://hbase.apache.org/book.html#hbase.mapreduce.classpath) 了解如何将 HBase 依赖的 jar 包添加到 Hadoop 的 classpath。* + + +如何使用 HBase 表 +---------------- + +所有 HBase 表的列簇必须定义为 ROW 类型，字段名对应列簇名（column family），嵌套的字段名对应列限定符名（column qualifier）。用户只需在表结构中声明查询中使用的的列簇和列限定符。除了 ROW 类型的列，剩下的原子数据类型字段（比如，STRING, BIGINT）将被识别为 HBase 的 rowkey，一张表中只能声明一个 rowkey。rowkey 字段的名字可以是任意的，如果是保留关键字，需要用反引号。 + ++ + + +Flink SQL type +HBase conversion ++ +CHAR / VARCHAR / STRING+{% highlight java %} +byte[] toBytes(String s) +String toString(byte[] b) +{% endhighlight %} + ++ +BOOLEAN+{% highlight java %} +byte[] toBytes(boolean b) +boolean toBoolean(byte[] b) +{% endhighlight %} + ++ +BINARY / VARBINARYReturns +byte[]as is.+ +DECIMAL+{% highlight java %} +byte[] toBytes(BigDecimal v) +BigDecimal toBigDecimal(byte[] b) +{% endhighlight %} + ++ +TINYINT+{% highlight java %} +new byte[] { val } +bytes[0] // returns first and only byte from bytes +{% endhighlight %} + ++ +SMALLINT+{% highlight java %} +byte[] toBytes(short val) +short toShort(byte[] bytes) +{% endhighlight %} + ++ +INT+{% highlight java %} +byte[] toBytes(int val) +int toInt(byte[] bytes) +{% endhighlight %} + ++ +BIGINT+{% highlight java %} +byte[] toBytes(long val) +long toLong(byte[] bytes) +{% endhighlight %} + ++ +FLOAT+{% highlight java %} +byte[] toBytes(float val) +float toFloat(byte[] bytes) +{% endhighlight %} + ++ +DOUBLE+{% highlight java %} +byte[] toBytes(double val) +double toDouble(byte[] bytes) +{% endhighlight %} + ++ +DATEStores the number of days since epoch as int value. ++ +TIMEStores the number of milliseconds of the day as int value. ++ +TIMESTAMPStores the milliseconds since epoch as long value. ++ +ARRAYNot supported ++ +MAP / MULTISETNot supported ++ + +ROWNot supported +++ +连接器参数 +---------------- + ++{% highlight sql %} +-- 在 Flink SQL 中注册 HBase 表 "mytable" +CREATE TABLE hTable ( + rowkey INT, + family1 ROW+, + family2 ROW , + family3 ROW , + PRIMARY KEY (rowkey) NOT ENFORCED +) WITH ( + 'connector' = 'hbase-1.4', + 'table-name' = 'mytable', + 'zookeeper.quorum' = 'localhost:2181' +); + +-- 用 ROW(...) 构造函数构造列簇，并往 HBase 表写数据。 +-- 假设 "T" 的表结构是 [rowkey, f1q1, f2q2, f2q3, f3q4, f3q5, f3q6] +INSERT INTO hTable +SELECT rowkey, ROW(f1q1), ROW(f2q2, f2q3), ROW(f3q4, f3q5, f3q6) FROM T; + +-- 从 HBase 表扫描数据 +SELECT rowkey, family1, family3.q4, family3.q6 FROM hTable; + +-- temporal join HBase 表，将 HBase 表作为维表 +SELECT * FROM myTopic +LEFT JOIN hTable FOR SYSTEM_TIME AS OF myTopic.proctime +ON myTopic.key = hTable.rowkey; +{% endhighlight %} + + +
+ + + +数据类型映射表 +---------------- + +HBase 以字节数组存储所有数据。在读和写过程中要序列化和反序列化数据。 + +Flink 的 HBase 连接器利用 HBase（Hadoop) 的工具类 `org.apache.hadoop.hbase.util.Bytes` 进行字节数组和 Flink 数据类型转换。 + +Flink 的 HBase 连接器将所有数据类型（除字符串外）`null` 值编码成空字节。对于字符串类型，`null` 值的字面值由`null-string-literal`选项值决定。 + +数据类型映射表如下： + ++ + + +参数 +是否必选 +默认参数 +数据类型 +描述 ++ +connector
必选 +(none) +String +指定使用的连接器，这里写“hbase-1.4”。 ++ +table-name
必选 +(none) +String +连接的 HBase 表名。 ++ +zookeeper.quorum
必选 +(none) +String +HBase Zookeeper quorum 信息。 ++ +zookeeper.znode.parent
可选 +/hbase +String +HBase 集群的 Zookeeper 根目录。 ++ +null-string-literal
可选 +null +String +当字符串值为 +null时的存储形式，默认存成 "null" 字符串。HBase 的 source 和 sink 的编解码将所有数据类型（除字符串外）将null值以空字节来存储。+ +sink.buffer-flush.max-size
可选 +2mb +MemorySize +写入的参数选项。每次写入请求缓存行的最大大小。它能提升写入 HBase 数据库的性能，但是也可能增加延迟。设置为 "0" 关闭此选项。 + ++ +sink.buffer-flush.max-rows
可选 +1000 +Integer +写入的参数选项。 每次写入请求缓存的最大行数。它能提升写入 HBase 数据库的性能，但是也可能增加延迟。设置为 "0" 关闭此选项。 + ++ + +sink.buffer-flush.interval
可选 +1s +Duration +写入的参数选项。刷写缓存行的间隔。它能提升写入 HBase 数据库的性能，但是也可能增加延迟。设置为 "0" 关闭此选项。注意："sink.buffer-flush.max-size" 和 "sink.buffer-flush.max-rows" 同时设置为 "0"，刷写选项整个异步处理缓存行为。 + ++ +
+ +{% top %} diff --git a/docs/dev/table/connectors/index.md b/docs/dev/table/connectors/index.md new file mode 100644 index 00000000000000..c7fbf6fb1a11a7 --- /dev/null +++ b/docs/dev/table/connectors/index.md @@ -0,0 +1,268 @@ +--- +title: "Table & SQL Connectors" +nav-id: sql-connectors +nav-parent_id: connectors-root +nav-pos: 2 +nav-show_overview: true +--- + + + +Flink's Table API & SQL programs can be connected to other external systems for reading and writing both batch and streaming tables. A table source provides access to data which is stored in external systems (such as a database, key-value store, message queue, or file system). A table sink emits a table to an external storage system. Depending on the type of source and sink, they support different formats such as CSV, Avro, Parquet, or ORC. + +This page describes how to register table sources and table sinks in Flink using the natively supported connectors. After a source or sink has been registered, it can be accessed by Table API & SQL statements. + +NOTE If you want to implement your own *custom* table source or sink, have a look at the [user-defined sources & sinks page]({% link dev/table/sourceSinks.md %}). + +Attention Flink Table & SQL introduces a new set of connector options since 1.11.0, if you are using the legacy connector options, please refer to the [legacy documentation]({% link dev/table/connect.md %}). + +* This will be replaced by the TOC +{:toc} + +Supported Connectors +------------ + +Flink natively support various connectors. The following tables list all available connectors. + ++ + + +Flink 数据类型 +HBase 转换 ++ +CHAR / VARCHAR / STRING+{% highlight java %} +byte[] toBytes(String s) +String toString(byte[] b) +{% endhighlight %} + ++ +BOOLEAN+{% highlight java %} +byte[] toBytes(boolean b) +boolean toBoolean(byte[] b) +{% endhighlight %} + ++ +BINARY / VARBINARY返回 +byte[]。+ +DECIMAL+{% highlight java %} +byte[] toBytes(BigDecimal v) +BigDecimal toBigDecimal(byte[] b) +{% endhighlight %} + ++ +TINYINT+{% highlight java %} +new byte[] { val } +bytes[0] // returns first and only byte from bytes +{% endhighlight %} + ++ +SMALLINT+{% highlight java %} +byte[] toBytes(short val) +short toShort(byte[] bytes) +{% endhighlight %} + ++ +INT+{% highlight java %} +byte[] toBytes(int val) +int toInt(byte[] bytes) +{% endhighlight %} + ++ +BIGINT+{% highlight java %} +byte[] toBytes(long val) +long toLong(byte[] bytes) +{% endhighlight %} + ++ +FLOAT+{% highlight java %} +byte[] toBytes(float val) +float toFloat(byte[] bytes) +{% endhighlight %} + ++ +DOUBLE+{% highlight java %} +byte[] toBytes(double val) +double toDouble(byte[] bytes) +{% endhighlight %} + ++ +DATE从 1970-01-01 00:00:00 UTC 开始的天数，int 值。 ++ +TIME从 1970-01-01 00:00:00 UTC 开始天的毫秒数，int 值。 ++ +TIMESTAMP从 1970-01-01 00:00:00 UTC 开始的毫秒数，long 值。 ++ +ARRAY不支持 ++ +MAP / MULTISET不支持 ++ + +ROW不支持 ++ +
+ +{% top %} + +How to use connectors +-------- + +Flink supports to use SQL CREATE TABLE statement to register a table. One can define the table name, the table schema, and the table options for connecting to an external system. + +The following code shows a full example of how to connect to Kafka for reading Json records. + ++ + + +Name +Version +Source +Sink ++ +Filesystem ++ Bounded and Unbounded Scan, Lookup +Streaming Sink, Batch Sink ++ +Elasticsearch +6.x & 7.x +Not supported +Streaming Sink, Batch Sink ++ +Apache Kafka +0.10+ +Unbounded Scan +Streaming Sink, Batch Sink ++ +JDBC ++ Bounded Scan, Lookup +Streaming Sink, Batch Sink ++ + +Apache HBase +1.4.x +Bounded Scan, Lookup +Streaming Sink, Batch Sink +++ +In this way the desired connection properties are converted into string-based key-value pairs. So-called [table factories]({% link dev/table/sourceSinks.md %}#define-a-tablefactory) create configured table sources, table sinks, and corresponding formats from the key-value pairs. All table factories that can be found via Java's [Service Provider Interfaces (SPI)](https://docs.oracle.com/javase/tutorial/sound/SPI-intro.html) are taken into account when searching for exactly-one matching table factory. + +If no factory can be found or multiple factories match for the given properties, an exception will be thrown with additional information about considered factories and supported properties. + +{% top %} + +Schema Mapping +------------ + +The body clause of a SQL `CREATE TABLE` statement defines the names and types of columns, constraints and watermarks. Flink doesn't hold the data, thus the schema definition only declares how to map types from an external system to Flink’s representation. The mapping may not be mapped by names, it depends on the implementation of formats and connectors. For example, a MySQL database table is mapped by field names (not case sensitive), and a CSV filesystem is mapped by field order (field names can be arbitrary). This will be explained in every connectors. + +The following example shows a simple schema without time attributes and one-to-one field mapping of input/output to table columns. + ++{% highlight sql %} +CREATE TABLE MyUserTable ( + -- declare the schema of the table + `user` BIGINT, + message STRING, + ts TIMESTAMP, + proctime AS PROCTIME(), -- use computed column to define proctime attribute + WATERMARK FOR ts AS ts - INTERVAL '5' SECOND -- use WATERMARK statement to define rowtime attribute +) WITH ( + -- declare the external system to connect to + 'connector' = 'kafka', + 'topic' = 'topic_name', + 'scan.startup.mode' = 'earliest-offset', + 'properties.bootstrap.servers' = 'localhost:9092', + 'format' = 'json' -- declare a format for this system +) +{% endhighlight %} ++++ +### Primary Key + +Primary key constraints tell that a column or a set of columns of a table are unique and they do not contain nulls. Primary key uniquely identifies a row in a table. + +The primary key of a source table is a metadata information for optimization. The primary key of a sink table is usually used by the sink implementation for upserting. + +SQL standard specifies that a constraint can either be ENFORCED or NOT ENFORCED. This controls if the constraint checks are performed on the incoming/outgoing data. Flink does not own the data the only mode we want to support is the NOT ENFORCED mode. Its up to the user to ensure that the query enforces key integrity. + ++{% highlight sql %} +CREATE TABLE MyTable ( + MyField1 INT, + MyField2 STRING, + MyField3 BOOLEAN +) WITH ( + ... +) +{% endhighlight %} ++++ +### Time Attributes + +Time attributes are essential when working with unbounded streaming tables. Therefore both proctime and rowtime attributes can be defined as part of the schema. + +For more information about time handling in Flink and especially event-time, we recommend the general [event-time section]({% link dev/table/streaming/time_attributes.md %}). + +#### Proctime Attributes + +In order to declare a proctime attribute in the schema, you can use [Computed Column syntax]({% link dev/table/sql/create.md %}#create-table) to declare a computed column which is generated from `PROCTIME()` builtin function. +The computed column is a virtual column which is not stored in the physical data. + ++{% highlight sql %} +CREATE TABLE MyTable ( + MyField1 INT, + MyField2 STRING, + MyField3 BOOLEAN, + PRIMARY KEY (MyField1, MyField2) NOT ENFORCED -- defines a primary key on columns +) WITH ( + ... +) +{% endhighlight %} ++++ +#### Rowtime Attributes + +In order to control the event-time behavior for tables, Flink provides predefined timestamp extractors and watermark strategies. + +Please refer to [CREATE TABLE statements]({% link dev/table/sql/create.md %}#create-table) for more information about defining time attributes in DDL. + +The following timestamp extractors are supported: + ++{% highlight sql %} +CREATE TABLE MyTable ( + MyField1 INT, + MyField2 STRING, + MyField3 BOOLEAN + MyField4 AS PROCTIME() -- declares a proctime attribute +) WITH ( + ... +) +{% endhighlight %} ++++ +The following watermark strategies are supported: + ++{% highlight sql %} +-- use the existing TIMESTAMP(3) field in schema as the rowtime attribute +CREATE TABLE MyTable ( + ts_field TIMESTAMP(3), + WATERMARK FOR ts_field AS ... +) WITH ( + ... +) + +-- use system functions or UDFs or expressions to extract the expected TIMESTAMP(3) rowtime field +CREATE TABLE MyTable ( + log_ts STRING, + ts_field AS TO_TIMESTAMP(log_ts), + WATERMARK FOR ts_field AS ... +) WITH ( + ... +) +{% endhighlight %} ++++ +Make sure to always declare both timestamps and watermarks. Watermarks are required for triggering time-based operations. + +### SQL Types + +Please see the [Data Types]({% link dev/table/types.md %}) page about how to declare a type in SQL. + +{% top %} \ No newline at end of file diff --git a/docs/dev/table/connectors/index.zh.md b/docs/dev/table/connectors/index.zh.md new file mode 100644 index 00000000000000..6e5ece58cdefe1 --- /dev/null +++ b/docs/dev/table/connectors/index.zh.md @@ -0,0 +1,268 @@ +--- +title: "Table & SQL Connectors" +nav-id: sql-connectors +nav-parent_id: connectors-root +nav-pos: 2 +nav-show_overview: true +--- + + + +Flink's Table API & SQL programs can be connected to other external systems for reading and writing both batch and streaming tables. A table source provides access to data which is stored in external systems (such as a database, key-value store, message queue, or file system). A table sink emits a table to an external storage system. Depending on the type of source and sink, they support different formats such as CSV, Avro, Parquet, or ORC. + +This page describes how to register table sources and table sinks in Flink using the natively supported connectors. After a source or sink has been registered, it can be accessed by Table API & SQL statements. + +NOTE If you want to implement your own *custom* table source or sink, have a look at the [user-defined sources & sinks page]({% link dev/table/sourceSinks.zh.md %}). + +Attention Flink Table & SQL introduces a new set of connector options since 1.11.0, if you are using the legacy connector options, please refer to the [legacy documentation]({% link dev/table/connect.zh.md %}). + +* This will be replaced by the TOC +{:toc} + +Supported Connectors +------------ + +Flink natively support various connectors. The following tables list all available connectors. + ++{% highlight sql %} +-- Sets a watermark strategy for strictly ascending rowtime attributes. Emits a watermark of the +-- maximum observed timestamp so far. Rows that have a timestamp smaller to the max timestamp +-- are not late. +CREATE TABLE MyTable ( + ts_field TIMESTAMP(3), + WATERMARK FOR ts_field AS ts_field +) WITH ( + ... +) + +-- Sets a watermark strategy for ascending rowtime attributes. Emits a watermark of the maximum +-- observed timestamp so far minus 1. Rows that have a timestamp equal to the max timestamp +-- are not late. +CREATE TABLE MyTable ( + ts_field TIMESTAMP(3), + WATERMARK FOR ts_field AS ts_field - INTERVAL '0.001' SECOND +) WITH ( + ... +) + +-- Sets a watermark strategy for rowtime attributes which are out-of-order by a bounded time interval. +-- Emits watermarks which are the maximum observed timestamp minus the specified delay, e.g. 2 seconds. +CREATE TABLE MyTable ( + ts_field TIMESTAMP(3), + WATERMARK FOR ts_field AS ts_field - INTERVAL '2' SECOND +) WITH ( + ... +) +{% endhighlight %} +++ +
+ +{% top %} + +How to use connectors +-------- + +Flink supports to use SQL CREATE TABLE statement to register a table. One can define the table name, the table schema, and the table options for connecting to an external system. + +The following code shows a full example of how to connect to Kafka for reading Json records. + ++ + + +Name +Version +Source +Sink ++ +Filesystem ++ Bounded and Unbounded Scan, Lookup +Streaming Sink, Batch Sink ++ +Elasticsearch +6.x & 7.x +Not supported +Streaming Sink, Batch Sink ++ +Apache Kafka +0.10+ +Unbounded Scan +Streaming Sink, Batch Sink ++ +JDBC ++ Bounded Scan, Lookup +Streaming Sink, Batch Sink ++ + +Apache HBase +1.4.x +Bounded Scan, Lookup +Streaming Sink, Batch Sink +++ +In this way the desired connection properties are converted into string-based key-value pairs. So-called [table factories]({% link dev/table/sourceSinks.zh.md %}#define-a-tablefactory) create configured table sources, table sinks, and corresponding formats from the key-value pairs. All table factories that can be found via Java's [Service Provider Interfaces (SPI)](https://docs.oracle.com/javase/tutorial/sound/SPI-intro.html) are taken into account when searching for exactly-one matching table factory. + +If no factory can be found or multiple factories match for the given properties, an exception will be thrown with additional information about considered factories and supported properties. + +{% top %} + +Schema Mapping +------------ + +The body clause of a SQL `CREATE TABLE` statement defines the names and types of columns, constraints and watermarks. Flink doesn't hold the data, thus the schema definition only declares how to map types from an external system to Flink’s representation. The mapping may not be mapped by names, it depends on the implementation of formats and connectors. For example, a MySQL database table is mapped by field names (not case sensitive), and a CSV filesystem is mapped by field order (field names can be arbitrary). This will be explained in every connectors. + +The following example shows a simple schema without time attributes and one-to-one field mapping of input/output to table columns. + ++{% highlight sql %} +CREATE TABLE MyUserTable ( + -- declare the schema of the table + `user` BIGINT, + message STRING, + ts TIMESTAMP, + proctime AS PROCTIME(), -- use computed column to define proctime attribute + WATERMARK FOR ts AS ts - INTERVAL '5' SECOND -- use WATERMARK statement to define rowtime attribute +) WITH ( + -- declare the external system to connect to + 'connector' = 'kafka', + 'topic' = 'topic_name', + 'scan.startup.mode' = 'earliest-offset', + 'properties.bootstrap.servers' = 'localhost:9092', + 'format' = 'json' -- declare a format for this system +) +{% endhighlight %} ++++ +### Primary Key + +Primary key constraints tell that a column or a set of columns of a table are unique and they do not contain nulls. Primary key uniquely identifies a row in a table. + +The primary key of a source table is a metadata information for optimization. The primary key of a sink table is usually used by the sink implementation for upserting. + +SQL standard specifies that a constraint can either be ENFORCED or NOT ENFORCED. This controls if the constraint checks are performed on the incoming/outgoing data. Flink does not own the data the only mode we want to support is the NOT ENFORCED mode. Its up to the user to ensure that the query enforces key integrity. + ++{% highlight sql %} +CREATE TABLE MyTable ( + MyField1 INT, + MyField2 STRING, + MyField3 BOOLEAN +) WITH ( + ... +) +{% endhighlight %} ++++ +### Time Attributes + +Time attributes are essential when working with unbounded streaming tables. Therefore both proctime and rowtime attributes can be defined as part of the schema. + +For more information about time handling in Flink and especially event-time, we recommend the general [event-time section]({% link dev/table/streaming/time_attributes.zh.md %}). + +#### Proctime Attributes + +In order to declare a proctime attribute in the schema, you can use [Computed Column syntax]({% link dev/table/sql/create.zh.md %}#create-table) to declare a computed column which is generated from `PROCTIME()` builtin function. +The computed column is a virtual column which is not stored in the physical data. + ++{% highlight sql %} +CREATE TABLE MyTable ( + MyField1 INT, + MyField2 STRING, + MyField3 BOOLEAN, + PRIMARY KEY (MyField1, MyField2) NOT ENFORCED -- defines a primary key on columns +) WITH ( + ... +) +{% endhighlight %} ++++ +#### Rowtime Attributes + +In order to control the event-time behavior for tables, Flink provides predefined timestamp extractors and watermark strategies. + +Please refer to [CREATE TABLE statements]({% link dev/table/sql/create.zh.md %}#create-table) for more information about defining time attributes in DDL. + +The following timestamp extractors are supported: + ++{% highlight sql %} +CREATE TABLE MyTable ( + MyField1 INT, + MyField2 STRING, + MyField3 BOOLEAN + MyField4 AS PROCTIME() -- declares a proctime attribute +) WITH ( + ... +) +{% endhighlight %} ++++ +The following watermark strategies are supported: + ++{% highlight sql %} +-- use the existing TIMESTAMP(3) field in schema as the rowtime attribute +CREATE TABLE MyTable ( + ts_field TIMESTAMP(3), + WATERMARK FOR ts_field AS ... +) WITH ( + ... +) + +-- use system functions or UDFs or expressions to extract the expected TIMESTAMP(3) rowtime field +CREATE TABLE MyTable ( + log_ts STRING, + ts_field AS TO_TIMESTAMP(log_ts), + WATERMARK FOR ts_field AS ... +) WITH ( + ... +) +{% endhighlight %} ++++ +Make sure to always declare both timestamps and watermarks. Watermarks are required for triggering time-based operations. + +### SQL Types + +Please see the [Data Types]({% link dev/table/types.zh.md %}) page about how to declare a type in SQL. + +{% top %} diff --git a/docs/dev/table/connectors/jdbc.md b/docs/dev/table/connectors/jdbc.md new file mode 100644 index 00000000000000..addf1c834e2236 --- /dev/null +++ b/docs/dev/table/connectors/jdbc.md @@ -0,0 +1,562 @@ +--- +title: "JDBC SQL Connector" +nav-title: JDBC +nav-parent_id: sql-connectors +nav-pos: 3 +--- + + +Scan Source: Bounded +Lookup Source: Sync Mode +Sink: Batch +Sink: Streaming Append & Upsert Mode + +* This will be replaced by the TOC +{:toc} + +The JDBC connector allows for reading data from and writing data into any relational databases with a JDBC driver. This document describes how to setup the JDBC connector to run SQL queries against relational databases. + +The JDBC sink operate in upsert mode for exchange UPDATE/DELETE messages with the external system if a primary key is defined on the DDL, otherwise, it operates in append mode and doesn't support to consume UPDATE/DELETE messages. + +Dependencies +------------ + +In order to setup the JDBC connector, the following table provides dependency information for both projects using a build automation tool (such as Maven or SBT) and SQL Client with SQL JAR bundles. + +| Maven dependency | SQL Client JAR | +| :------------------------------------------------- | :-------------------------------------------------------- | +| `flink-connector-jdbc{{site.scala_version_suffix}}`| {% if site.is_stable %} [Download](https://repo.maven.apache.org/maven2/org/apache/flink/flink-connector-jdbc{{site.scala_version_suffix}}/{{site.version}}/flink-connector-jdbc{{site.scala_version_suffix}}-{{site.version}}.jar) {% else %} Only available for [stable releases]({{ site.stable_baseurl }}/dev/table/connectors/jdbc.html) {% endif %}| + ++{% highlight sql %} +-- Sets a watermark strategy for strictly ascending rowtime attributes. Emits a watermark of the +-- maximum observed timestamp so far. Rows that have a timestamp smaller to the max timestamp +-- are not late. +CREATE TABLE MyTable ( + ts_field TIMESTAMP(3), + WATERMARK FOR ts_field AS ts_field +) WITH ( + ... +) + +-- Sets a watermark strategy for ascending rowtime attributes. Emits a watermark of the maximum +-- observed timestamp so far minus 1. Rows that have a timestamp equal to the max timestamp +-- are not late. +CREATE TABLE MyTable ( + ts_field TIMESTAMP(3), + WATERMARK FOR ts_field AS ts_field - INTERVAL '0.001' SECOND +) WITH ( + ... +) + +-- Sets a watermark strategy for rowtime attributes which are out-of-order by a bounded time interval. +-- Emits watermarks which are the maximum observed timestamp minus the specified delay, e.g. 2 seconds. +CREATE TABLE MyTable ( + ts_field TIMESTAMP(3), + WATERMARK FOR ts_field AS ts_field - INTERVAL '2' SECOND +) WITH ( + ... +) +{% endhighlight %} ++
+A driver dependency is also required to connect to a specified database. Here are drivers currently supported: + +| Driver | Group Id | Artifact Id | JAR | +| :-----------| :------------------| :----------------------| :----------------| +| MySQL | `mysql` | `mysql-connector-java` | [Download](https://repo.maven.apache.org/maven2/mysql/mysql-connector-java/) | +| PostgreSQL | `org.postgresql` | `postgresql` | [Download](https://jdbc.postgresql.org/download.html) | +| Derby | `org.apache.derby` | `derby` | [Download](http://db.apache.org/derby/derby_downloads.html) | + +
+JDBC connector and drivers are not currently part of Flink's binary distribution. See how to link with them for cluster execution [here]({% link dev/project-configuration.md %}). + + +How to create a JDBC table +---------------- + +The JDBC table can be defined as following: + +++ +Connector Options +---------------- + ++{% highlight sql %} +-- register a MySQL table 'users' in Flink SQL +CREATE TABLE MyUserTable ( + id BIGINT, + name STRING, + age INT, + status BOOLEAN, + PRIMARY KEY (id) NOT ENFORCED +) WITH ( + 'connector' = 'jdbc', + 'url' = 'jdbc:mysql://localhost:3306/mydatabase', + 'table-name' = 'users' +); + +-- write data into the JDBC table from the other table "T" +INSERT INTO MyUserTable +SELECT id, name, age, status FROM T; + +-- scan data from the JDBC table +SELECT id, name, age, status FROM MyUserTable; + +-- temporal join the JDBC table as a dimension table +SELECT * FROM myTopic +LEFT JOIN MyUserTable FOR SYSTEM_TIME AS OF myTopic.proctime +ON myTopic.key = MyUserTable.id; +{% endhighlight %} +++ +
+ +Features +-------- + +### Key handling + +Flink uses the primary key that defined in DDL when writing data to external databases. The connector operate in upsert mode if the primary key was defined, otherwise, the connector operate in append mode. + +In upsert mode, Flink will insert a new row or update the existing row according to the primary key, Flink can ensure the idempotence in this way. To guarantee the output result is as expected, it's recommended to define primary key for the table and make sure the primary key is one of the unique key sets or primary key of the underlying database table. In append mode, Flink will interpret all records as INSERT messages, the INSERT operation may fail if a primary key or unique constraint violation happens in the underlying database. + +See [CREATE TABLE DDL]({% link dev/table/sql/create.md %}#create-table) for more details about PRIMARY KEY syntax. + +### Partitioned Scan + +To accelerate reading data in parallel `Source` task instances, Flink provides partitioned scan feature for JDBC table. + +All the following scan partition options must all be specified if any of them is specified. They describe how to partition the table when reading in parallel from multiple tasks. +The `scan.partition.column` must be a numeric, date, or timestamp column from the table in question. Notice that `scan.partition.lower-bound` and `scan.partition.upper-bound` are just used to decide the partition stride, not for filtering the rows in table. So all rows in the table will be partitioned and returned. + +- `scan.partition.column`: The column name used for partitioning the input. +- `scan.partition.num`: The number of partitions. +- `scan.partition.lower-bound`: The smallest value of the first partition. +- `scan.partition.upper-bound`: The largest value of the last partition. + +### Lookup Cache + +JDBC connector can be used in temporal join as a lookup source (aka. dimension table). Currently, only sync lookup mode is supported. + +By default, lookup cache is not enabled. You can enable it by setting both `lookup.cache.max-rows` and `lookup.cache.ttl`. + +The lookup cache is used to improve performance of temporal join the JDBC connector. By default, lookup cache is not enabled, so all the requests are sent to external database. +When lookup cache is enabled, each process (i.e. TaskManager) will hold a cache. Flink will lookup the cache first, and only send requests to external database when cache missing, and update cache with the rows returned. +The oldest rows in cache will be expired when the cache hit to the max cached rows `lookup.cache.max-rows` or when the row exceeds the max time to live `lookup.cache.ttl`. +The cached rows might not be the latest, users can tune `lookup.cache.ttl` to a smaller value to have a better fresh data, but this may increase the number of requests send to database. So this is a balance between throughput and correctness. + +### Idempotent Writes + +JDBC sink will use upsert semantics rather than plain INSERT statements if primary key is defined in DDL. Upsert semantics refer to atomically adding a new row or updating the existing row if there is a unique constraint violation in the underlying database, which provides idempotence. + +If there are failures, the Flink job will recover and re-process from last successful checkpoint, which can lead to re-processing messages during recovery. The upsert mode is highly recommended as it helps avoid constraint violations or duplicate data if records need to be re-processed. + +Aside from failure recovery, the source topic may also naturally contain multiple records over time with the same primary key, making upserts desirable. + +As there is no standard syntax for upsert, the following table describes the database-specific DML that is used. + ++ + + +Option +Required +Default +Type +Description ++ +connector
required +(none) +String +Specify what connector to use, here should be +'jdbc'.+ +url
required +(none) +String +The JDBC database url. ++ +table-name
required +(none) +String +The name of JDBC table to connect. ++ +driver
optional +(none) +String +The class name of the JDBC driver to use to connect to this URL, if not set, it will automatically be derived from the URL. ++ +username
optional +(none) +String +The JDBC user name. +'username'and'password'must both be specified if any of them is specified.+ +password
optional +(none) +String +The JDBC password. ++ +scan.partition.column
optional +(none) +String +The column name used for partitioning the input. See the following Partitioned Scan section for more details. ++ +scan.partition.num
optional +(none) +Integer +The number of partitions. ++ +scan.partition.lower-bound
optional +(none) +Integer +The smallest value of the first partition. ++ +scan.partition.upper-bound
optional +(none) +Integer +The largest value of the last partition. ++ +scan.fetch-size
optional +0 +Integer +The number of rows that should be fetched from the database when reading per round trip. If the value specified is zero, then the hint is ignored. ++ +lookup.cache.max-rows
optional +(none) +Integer +The max number of rows of lookup cache, over this value, the oldest rows will be expired. + Lookup cache is disabled by default. See the following Lookup Cache section for more details. ++ +lookup.cache.ttl
optional +(none) +Integer +The max time to live for each rows in lookup cache, over this time, the oldest rows will be expired. + Lookup cache is disabled by default. See the following Lookup Cache section for more details. ++ +lookup.max-retries
optional +3 +Integer +The max retry times if lookup database failed. ++ +sink.buffer-flush.max-rows
optional +100 +Integer +The max size of buffered records before flush. Can be set to zero to disable it. ++ +sink.buffer-flush.interval
optional +1s +Duration +The flush interval mills, over this time, asynchronous threads will flush data. Can be set to +'0'to disable it. Note,'sink.buffer-flush.max-rows'can be set to'0'with the flush interval set allowing for complete async processing of buffered actions.+ + +sink.max-retries
optional +3 +Integer +The max retry times if writing records to database failed. ++ +
+ +### Postgres Database as a Catalog + +The `JdbcCatalog` enables users to connect Flink to relational databases over JDBC protocol. + +Currently, `PostgresCatalog` is the only implementation of JDBC Catalog at the moment, `PostgresCatalog` only supports limited `Catalog` methods include: + +{% highlight java %} +// The supported methods by Postgres Catalog. +PostgresCatalog.databaseExists(String databaseName) +PostgresCatalog.listDatabases() +PostgresCatalog.getDatabase(String databaseName) +PostgresCatalog.listTables(String databaseName) +PostgresCatalog.getTable(ObjectPath tablePath) +PostgresCatalog.tableExists(ObjectPath tablePath) +{% endhighlight %} + +Other `Catalog` methods is unsupported now. + +#### Usage of PostgresCatalog + +Please refer to [Dependencies](#dependencies) section for how to setup a JDBC connector and Postgres driver. + +Postgres catalog supports the following options: +- `name`: required, name of the catalog. +- `default-database`: required, default database to connect to. +- `username`: required, username of Postgres account. +- `password`: required, password of the account. +- `base-url`: required, should be of format `"jdbc:postgresql://+ + + +Database +Upsert Grammar ++ +MySQL +INSERT .. ON DUPLICATE KEY UPDATE .. ++ + +PostgreSQL +INSERT .. ON CONFLICT .. DO UPDATE SET .. +: "`, and should not contain database name here. + + ++ +#### PostgresSQL Metaspace Mapping + +PostgresSQL has an additional namespace as `schema` besides database. A Postgres instance can have multiple databases, each database can have multiple schemas with a default one named "public", each schema can have multiple tables. +In Flink, when querying tables registered by Postgres catalog, users can use either `schema_name.table_name` or just `table_name`. The `schema_name` is optional and defaults to "public". + +Therefor the metaspace mapping between Flink Catalog and Postgres is as following: + +| Flink Catalog Metaspace Structure | Postgres Metaspace Structure | +| :------------------------------------| :-----------------------------------| +| catalog name (defined in Flink only) | N/A | +| database name | database name | +| table name | [schema_name.]table_name | + +The full path of Postgres table in Flink should be ``"+{% highlight sql %} +CREATE CATALOG mypg WITH( + 'type' = 'jdbc', + 'default-database' = '...', + 'username' = '...', + 'password' = '...', + 'base-url' = '...' +); + +USE CATALOG mypg; +{% endhighlight %} +++{% highlight java %} + +EnvironmentSettings settings = EnvironmentSettings.newInstance().inStreamingMode().build(); +TableEnvironment tableEnv = TableEnvironment.create(settings); + +String name = "mypg"; +String defaultDatabase = "mydb"; +String username = "..."; +String password = "..."; +String baseUrl = "..." + +JdbcCatalog catalog = new JdbcCatalog(name, defaultDatabase, username, password, baseUrl); +tableEnv.registerCatalog("mypg", catalog); + +// set the JdbcCatalog as the current catalog of the session +tableEnv.useCatalog("mypg"); +{% endhighlight %} +++{% highlight scala %} + +val settings = EnvironmentSettings.newInstance().inStreamingMode().build() +val tableEnv = TableEnvironment.create(settings) + +val name = "mypg" +val defaultDatabase = "mydb" +val username = "..." +val password = "..." +val baseUrl = "..." + +val catalog = new JdbcCatalog(name, defaultDatabase, username, password, baseUrl) +tableEnv.registerCatalog("mypg", catalog) + +// set the JdbcCatalog as the current catalog of the session +tableEnv.useCatalog("mypg") +{% endhighlight %} +++{% highlight yaml %} + +execution: + planner: blink + ... + current-catalog: mypg # set the JdbcCatalog as the current catalog of the session + current-database: mydb + +catalogs: + - name: mypg + type: jdbc + default-database: mydb + username: ... + password: ... + base-url: ... +{% endhighlight %} ++. .` `"`` if schema is specified, note the ` ` should be escaped. + +Here are some examples to access Postgres tables: + +{% highlight sql %} +-- scan table 'test_table' of 'public' schema (i.e. the default schema), the schema name can be omitted +SELECT * FROM mypg.mydb.test_table; +SELECT * FROM mydb.test_table; +SELECT * FROM test_table; + +-- scan table 'test_table2' of 'custom_schema' schema, +-- the custom schema can not be omitted and must be escaped with table. +SELECT * FROM mypg.mydb.`custom_schema.test_table2` +SELECT * FROM mydb.`custom_schema.test_table2`; +SELECT * FROM `custom_schema.test_table2`; +{% endhighlight %} + +Data Type Mapping +---------------- +Flink supports connect to several databases which uses dialect like MySQL, PostgresSQL, Derby. The Derby dialect usually used for testing purpose. The field data type mappings from relational databases data types to Flink SQL data types are listed in the following table, the mapping table can help define JDBC table in Flink easily. + +Scan Source: Bounded +Lookup Source: Sync Mode +Sink: Batch +Sink: Streaming Append & Upsert Mode + +* This will be replaced by the TOC +{:toc} + +The JDBC connector allows for reading data from and writing data into any relational databases with a JDBC driver. This document describes how to setup the JDBC connector to run SQL queries against relational databases. + +The JDBC sink operate in upsert mode for exchange UPDATE/DELETE messages with the external system if a primary key is defined on the DDL, otherwise, it operates in append mode and doesn't support to consume UPDATE/DELETE messages. + +Dependencies +------------ + +In order to setup the JDBC connector, the following table provides dependency information for both projects using a build automation tool (such as Maven or SBT) and SQL Client with SQL JAR bundles. + +| Maven dependency | SQL Client JAR | +| :------------------------------------------------- | :-------------------------------------------------------- | +| `flink-connector-jdbc{{site.scala_version_suffix}}`| {% if site.is_stable %} [Download](https://repo.maven.apache.org/maven2/org/apache/flink/flink-connector-jdbc{{site.scala_version_suffix}}/{{site.version}}/flink-connector-jdbc{{site.scala_version_suffix}}-{{site.version}}.jar) {% else %} Only available for [stable releases]({{ site.stable_baseurl }}/zh/dev/table/connectors/jdbc.html) {% endif %}| + +
+A driver dependency is also required to connect to a specified database. Here are drivers currently supported: + +| Driver | Group Id | Artifact Id | JAR | +| :-----------| :------------------| :----------------------| :----------------| +| MySQL | `mysql` | `mysql-connector-java` | [Download](https://repo.maven.apache.org/maven2/mysql/mysql-connector-java/) | +| PostgreSQL | `org.postgresql` | `postgresql` | [Download](https://jdbc.postgresql.org/download.html) | +| Derby | `org.apache.derby` | `derby` | [Download](http://db.apache.org/derby/derby_downloads.html) | + +
+JDBC connector and drivers are not currently part of Flink's binary distribution. See how to link with them for cluster execution [here]({% link dev/project-configuration.zh.md %}). + + +How to create a JDBC table +---------------- + +The JDBC table can be defined as following: + +++ +Connector Options +---------------- + ++{% highlight sql %} +-- register a MySQL table 'users' in Flink SQL +CREATE TABLE MyUserTable ( + id BIGINT, + name STRING, + age INT, + status BOOLEAN, + PRIMARY KEY (id) NOT ENFORCED +) WITH ( + 'connector' = 'jdbc', + 'url' = 'jdbc:mysql://localhost:3306/mydatabase', + 'table-name' = 'users' +); + +-- write data into the JDBC table from the other table "T" +INSERT INTO MyUserTable +SELECT id, name, age, status FROM T; + +-- scan data from the JDBC table +SELECT id, name, age, status FROM MyUserTable; + +-- temporal join the JDBC table as a dimension table +SELECT * FROM myTopic +LEFT JOIN MyUserTable FOR SYSTEM_TIME AS OF myTopic.proctime +ON myTopic.key = MyUserTable.id; +{% endhighlight %} +++ +
+ +Features +-------- + +### Key handling + +Flink uses the primary key that defined in DDL when writing data to external databases. The connector operate in upsert mode if the primary key was defined, otherwise, the connector operate in append mode. + +In upsert mode, Flink will insert a new row or update the existing row according to the primary key, Flink can ensure the idempotence in this way. To guarantee the output result is as expected, it's recommended to define primary key for the table and make sure the primary key is one of the unique key sets or primary key of the underlying database table. In append mode, Flink will interpret all records as INSERT messages, the INSERT operation may fail if a primary key or unique constraint violation happens in the underlying database. + +See [CREATE TABLE DDL]({% link dev/table/sql/create.zh.md %}#create-table) for more details about PRIMARY KEY syntax. + +### Partitioned Scan + +To accelerate reading data in parallel `Source` task instances, Flink provides partitioned scan feature for JDBC table. + +All the following scan partition options must all be specified if any of them is specified. They describe how to partition the table when reading in parallel from multiple tasks. +The `scan.partition.column` must be a numeric, date, or timestamp column from the table in question. Notice that `scan.partition.lower-bound` and `scan.partition.upper-bound` are just used to decide the partition stride, not for filtering the rows in table. So all rows in the table will be partitioned and returned. + +- `scan.partition.column`: The column name used for partitioning the input. +- `scan.partition.num`: The number of partitions. +- `scan.partition.lower-bound`: The smallest value of the first partition. +- `scan.partition.upper-bound`: The largest value of the last partition. + +### Lookup Cache + +JDBC connector can be used in temporal join as a lookup source (aka. dimension table). Currently, only sync lookup mode is supported. + +By default, lookup cache is not enabled. You can enable it by setting both `lookup.cache.max-rows` and `lookup.cache.ttl`. + +The lookup cache is used to improve performance of temporal join the JDBC connector. By default, lookup cache is not enabled, so all the requests are sent to external database. +When lookup cache is enabled, each process (i.e. TaskManager) will hold a cache. Flink will lookup the cache first, and only send requests to external database when cache missing, and update cache with the rows returned. +The oldest rows in cache will be expired when the cache hit to the max cached rows `lookup.cache.max-rows` or when the row exceeds the max time to live `lookup.cache.ttl`. +The cached rows might not be the latest, users can tune `lookup.cache.ttl` to a smaller value to have a better fresh data, but this may increase the number of requests send to database. So this is a balance between throughput and correctness. + +### Idempotent Writes + +JDBC sink will use upsert semantics rather than plain INSERT statements if primary key is defined in DDL. Upsert semantics refer to atomically adding a new row or updating the existing row if there is a unique constraint violation in the underlying database, which provides idempotence. + +If there are failures, the Flink job will recover and re-process from last successful checkpoint, which can lead to re-processing messages during recovery. The upsert mode is highly recommended as it helps avoid constraint violations or duplicate data if records need to be re-processed. + +Aside from failure recovery, the source topic may also naturally contain multiple records over time with the same primary key, making upserts desirable. + +As there is no standard syntax for upsert, the following table describes the database-specific DML that is used. + ++ + + +Option +Required +Default +Type +Description ++ +connector
required +(none) +String +Specify what connector to use, here should be +'jdbc'.+ +url
required +(none) +String +The JDBC database url. ++ +table-name
required +(none) +String +The name of JDBC table to connect. ++ +driver
optional +(none) +String +The class name of the JDBC driver to use to connect to this URL, if not set, it will automatically be derived from the URL. ++ +username
optional +(none) +String +The JDBC user name. +'username'and'password'must both be specified if any of them is specified.+ +password
optional +(none) +String +The JDBC password. ++ +scan.partition.column
optional +(none) +String +The column name used for partitioning the input. See the following Partitioned Scan section for more details. ++ +scan.partition.num
optional +(none) +Integer +The number of partitions. ++ +scan.partition.lower-bound
optional +(none) +Integer +The smallest value of the first partition. ++ +scan.partition.upper-bound
optional +(none) +Integer +The largest value of the last partition. ++ +scan.fetch-size
optional +0 +Integer +The number of rows that should be fetched from the database when reading per round trip. If the value specified is zero, then the hint is ignored. ++ +lookup.cache.max-rows
optional +(none) +Integer +The max number of rows of lookup cache, over this value, the oldest rows will be expired. + Lookup cache is disabled by default. See the following Lookup Cache section for more details. ++ +lookup.cache.ttl
optional +(none) +Integer +The max time to live for each rows in lookup cache, over this time, the oldest rows will be expired. + Lookup cache is disabled by default. See the following Lookup Cache section for more details. ++ +lookup.max-retries
optional +3 +Integer +The max retry times if lookup database failed. ++ +sink.buffer-flush.max-rows
optional +100 +Integer +The max size of buffered records before flush. Can be set to zero to disable it. ++ +sink.buffer-flush.interval
optional +1s +Duration +The flush interval mills, over this time, asynchronous threads will flush data. Can be set to +'0'to disable it. Note,'sink.buffer-flush.max-rows'can be set to'0'with the flush interval set allowing for complete async processing of buffered actions.+ + +sink.max-retries
optional +3 +Integer +The max retry times if writing records to database failed. ++ +
+ +### Postgres Database as a Catalog + +The `JdbcCatalog` enables users to connect Flink to relational databases over JDBC protocol. + +Currently, `PostgresCatalog` is the only implementation of JDBC Catalog at the moment, `PostgresCatalog` only supports limited `Catalog` methods include: + +{% highlight java %} +// The supported methods by Postgres Catalog. +PostgresCatalog.databaseExists(String databaseName) +PostgresCatalog.listDatabases() +PostgresCatalog.getDatabase(String databaseName) +PostgresCatalog.listTables(String databaseName) +PostgresCatalog.getTable(ObjectPath tablePath) +PostgresCatalog.tableExists(ObjectPath tablePath) +{% endhighlight %} + +Other `Catalog` methods is unsupported now. + +#### Usage of PostgresCatalog + +Please refer to [Dependencies](#dependencies) section for how to setup a JDBC connector and Postgres driver. + +Postgres catalog supports the following options: +- `name`: required, name of the catalog. +- `default-database`: required, default database to connect to. +- `username`: required, username of Postgres account. +- `password`: required, password of the account. +- `base-url`: required, should be of format `"jdbc:postgresql://+ + + +Database +Upsert Grammar ++ +MySQL +INSERT .. ON DUPLICATE KEY UPDATE .. ++ + +PostgreSQL +INSERT .. ON CONFLICT .. DO UPDATE SET .. +: "`, and should not contain database name here. + + ++ +#### PostgresSQL Metaspace Mapping + +PostgresSQL has an additional namespace as `schema` besides database. A Postgres instance can have multiple databases, each database can have multiple schemas with a default one named "public", each schema can have multiple tables. +In Flink, when querying tables registered by Postgres catalog, users can use either `schema_name.table_name` or just `table_name`. The `schema_name` is optional and defaults to "public". + +Therefor the metaspace mapping between Flink Catalog and Postgres is as following: + +| Flink Catalog Metaspace Structure | Postgres Metaspace Structure | +| :------------------------------------| :-----------------------------------| +| catalog name (defined in Flink only) | N/A | +| database name | database name | +| table name | [schema_name.]table_name | + +The full path of Postgres table in Flink should be ``"+{% highlight sql %} +CREATE CATALOG mypg WITH( + 'type' = 'jdbc', + 'default-database' = '...', + 'username' = '...', + 'password' = '...', + 'base-url' = '...' +); + +USE CATALOG mypg; +{% endhighlight %} +++{% highlight java %} + +EnvironmentSettings settings = EnvironmentSettings.newInstance().inStreamingMode().build(); +TableEnvironment tableEnv = TableEnvironment.create(settings); + +String name = "mypg"; +String defaultDatabase = "mydb"; +String username = "..."; +String password = "..."; +String baseUrl = "..." + +JdbcCatalog catalog = new JdbcCatalog(name, defaultDatabase, username, password, baseUrl); +tableEnv.registerCatalog("mypg", catalog); + +// set the JdbcCatalog as the current catalog of the session +tableEnv.useCatalog("mypg"); +{% endhighlight %} +++{% highlight scala %} + +val settings = EnvironmentSettings.newInstance().inStreamingMode().build() +val tableEnv = TableEnvironment.create(settings) + +val name = "mypg" +val defaultDatabase = "mydb" +val username = "..." +val password = "..." +val baseUrl = "..." + +val catalog = new JdbcCatalog(name, defaultDatabase, username, password, baseUrl) +tableEnv.registerCatalog("mypg", catalog) + +// set the JdbcCatalog as the current catalog of the session +tableEnv.useCatalog("mypg") +{% endhighlight %} +++{% highlight yaml %} + +execution: + planner: blink + ... + current-catalog: mypg # set the JdbcCatalog as the current catalog of the session + current-database: mydb + +catalogs: + - name: mypg + type: jdbc + default-database: mydb + username: ... + password: ... + base-url: ... +{% endhighlight %} ++. .` `"`` if schema is specified, note the ` ` should be escaped. + +Here are some examples to access Postgres tables: + +{% highlight sql %} +-- scan table 'test_table' of 'public' schema (i.e. the default schema), the schema name can be omitted +SELECT * FROM mypg.mydb.test_table; +SELECT * FROM mydb.test_table; +SELECT * FROM test_table; + +-- scan table 'test_table2' of 'custom_schema' schema, +-- the custom schema can not be omitted and must be escaped with table. +SELECT * FROM mypg.mydb.`custom_schema.test_table2` +SELECT * FROM mydb.`custom_schema.test_table2`; +SELECT * FROM `custom_schema.test_table2`; +{% endhighlight %} + +Data Type Mapping +---------------- +Flink supports connect to several databases which uses dialect like MySQL, PostgresSQL, Derby. The Derby dialect usually used for testing purpose. The field data type mappings from relational databases data types to Flink SQL data types are listed in the following table, the mapping table can help define JDBC table in Flink easily. + +Scan Source: Unbounded +Sink: Streaming Append Mode + +* This will be replaced by the TOC +{:toc} + +The Kafka connector allows for reading data from and writing data into Kafka topics. + +Dependencies +------------ + +Apache Flink ships with multiple Kafka connectors: universal, 0.10, and 0.11. +This universal Kafka connector attempts to track the latest version of the Kafka client. +The version of the client it uses may change between Flink releases. +Modern Kafka clients are backwards compatible with broker versions 0.10.0 or later. +For most users the universal Kafka connector is the most appropriate. +However, for Kafka versions 0.11.x and 0.10.x, we recommend using the dedicated ``0.11`` and ``0.10`` connectors, respectively. +For details on Kafka compatibility, please refer to the official [Kafka documentation](https://kafka.apache.org/protocol.html#protocol_compatibility). + +| Kafka Version | Maven dependency | SQL Client JAR | +| :------------------ | :-------------------------------------------------------- | :----------------------| +| universal | `flink-connector-kafka{{site.scala_version_suffix}}` | {% if site.is_stable %} [Download](https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-kafka{{site.scala_version_suffix}}/{{site.version}}/flink-sql-connector-kafka{{site.scala_version_suffix}}-{{site.version}}.jar) {% else %} Only available for [stable releases]({{ site.stable_baseurl }}/dev/table/connectors/kafka.html) {% endif %} | +| 0.11.x | `flink-connector-kafka-0.11{{site.scala_version_suffix}}` | {% if site.is_stable %} [Download](https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-kafka-0.11{{site.scala_version_suffix}}/{{site.version}}/flink-sql-connector-kafka{{site.scala_version_suffix}}-{{site.version}}.jar) {% else %} Only available for [stable releases]({{ site.stable_baseurl }}/dev/table/connectors/kafka.html) {% endif %} | +| 0.10.x | `flink-connector-kafka-0.10{{site.scala_version_suffix}}` | {% if site.is_stable %} [Download](https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-kafka-0.10{{site.scala_version_suffix}}/{{site.version}}/flink-sql-connector-kafka{{site.scala_version_suffix}}-{{site.version}}.jar) {% else %} Only available for [stable releases]({{ site.stable_baseurl }}/dev/table/connectors/kafka.html) {% endif %} | + +The Kafka connectors are not currently part of the binary distribution. +See how to link with them for cluster execution [here]({% link dev/project-configuration.md %}). + +How to create a Kafka table +---------------- + +The example below shows how to create a Kafka table: + + ++ +Connector Options +---------------- + ++{% highlight sql %} +CREATE TABLE kafkaTable ( + user_id BIGINT, + item_id BIGINT, + category_id BIGINT, + behavior STRING, + ts TIMESTAMP(3) +) WITH ( + 'connector' = 'kafka', + 'topic' = 'user_behavior', + 'properties.bootstrap.servers' = 'localhost:9092', + 'properties.group.id' = 'testGroup', + 'format' = 'csv', + 'scan.startup.mode' = 'earliest-offset' +) +{% endhighlight %} +++ +
+ +Features +---------------- + +### Start Reading Position + +The config option `scan.startup.mode` specifies the startup mode for Kafka consumer. The valid enumerations are: ++ + + +Option +Required +Default +Type +Description ++ +connector
required +(none) +String +Specify what connector to use, for Kafka the options are: +'kafka','kafka-0.11','kafka-0.10'.+ +topic
required +(none) +String +Topic name from which the table is read. ++ +properties.bootstrap.servers
required +(none) +String +Comma separated list of Kafka brokers. ++ +properties.group.id
required by source +(none) +String +The id of the consumer group for Kafka source, optional for Kafka sink. ++ +format
required +(none) +String +The format used to deserialize and serialize Kafka messages. + The supported formats are +'csv','json','avro','debezium-json'and'canal-json'. + Please refer to Formats page for more details and more format options. ++ +scan.startup.mode
optional +group-offsets +String +Startup mode for Kafka consumer, valid values are +'earliest-offset','latest-offset','group-offsets','timestamp'and'specific-offsets'. + See the following Start Reading Position for more details.+ +scan.startup.specific-offsets
optional +(none) +String +Specify offsets for each partition in case of +'specific-offsets'startup mode, e.g.'partition:0,offset:42;partition:1,offset:300'. ++ +scan.startup.timestamp-millis
optional +(none) +Long +Start from the specified epoch timestamp (milliseconds) used in case of +'timestamp'startup mode.+ + +sink.partitioner
optional +(none) +String +Output partitioning from Flink's partitions into Kafka's partitions. Valid values are + +-
+
fixed: each Flink partition ends up in at most one Kafka partition.
+ round-robin: a Flink partition is distributed to Kafka partitions round-robin.
+ - Custom
FlinkKafkaPartitionersubclass: e.g.'org.mycompany.MyPartitioner'.
+
-
+
- `group-offsets`: start from committed offsets in ZK / Kafka brokers of a specific consumer group. +
- `earliest-offset`: start from the earliest offset possible. +
- `latest-offset`: start from the latest offset. +
- `timestamp`: start from user-supplied timestamp for each partition. +
- `specific-offsets`: start from user-supplied specific offsets for each partition. +
++ +Connector Options +---------------- + ++{% highlight sql %} +CREATE TABLE kafkaTable ( + user_id BIGINT, + item_id BIGINT, + category_id BIGINT, + behavior STRING, + ts TIMESTAMP(3) +) WITH ( + 'connector' = 'kafka', + 'topic' = 'user_behavior', + 'properties.bootstrap.servers' = 'localhost:9092', + 'properties.group.id' = 'testGroup', + 'format' = 'csv', + 'scan.startup.mode' = 'earliest-offset' +) +{% endhighlight %} +++ +
+ +Features +---------------- + +### Start Reading Position + +The config option `scan.startup.mode` specifies the startup mode for Kafka consumer. The valid enumerations are: ++ + + +Option +Required +Default +Type +Description ++ +connector
required +(none) +String +Specify what connector to use, for Kafka the options are: +'kafka','kafka-0.11','kafka-0.10'.+ +topic
required +(none) +String +Topic name from which the table is read. ++ +properties.bootstrap.servers
required +(none) +String +Comma separated list of Kafka brokers. ++ +properties.group.id
required by source +(none) +String +The id of the consumer group for Kafka source, optional for Kafka sink. ++ +format
required +(none) +String +The format used to deserialize and serialize Kafka messages. + The supported formats are +'csv','json','avro','debezium-json'and'canal-json'. + Please refer to Formats page for more details and more format options. ++ +scan.startup.mode
optional +group-offsets +String +Startup mode for Kafka consumer, valid values are +'earliest-offset','latest-offset','group-offsets','timestamp'and'specific-offsets'. + See the following Start Reading Position for more details.+ +scan.startup.specific-offsets
optional +(none) +String +Specify offsets for each partition in case of +'specific-offsets'startup mode, e.g.'partition:0,offset:42;partition:1,offset:300'. ++ +scan.startup.timestamp-millis
optional +(none) +Long +Start from the specified epoch timestamp (milliseconds) used in case of +'timestamp'startup mode.+ + +sink.partitioner
optional +(none) +String +Output partitioning from Flink's partitions into Kafka's partitions. Valid values are + +-
+
fixed: each Flink partition ends up in at most one Kafka partition.
+ round-robin: a Flink partition is distributed to Kafka partitions round-robin.
+ - Custom
FlinkKafkaPartitionersubclass: e.g.'org.mycompany.MyPartitioner'.
+
-
+
- `group-offsets`: start from committed offsets in ZK / Kafka brokers of a specific consumer group. +
- `earliest-offset`: start from the earliest offset possible. +
- `latest-offset`: start from the latest offset. +
- `timestamp`: start from user-supplied timestamp for each partition. +
- `specific-offsets`: start from user-supplied specific offsets for each partition. +
+ +
+ +The output string format is "$row_kind(f0,f1,f2...)", row_kind is the short string of [RowKind]({{ site.baseurl }}/api/java/org/apache/flink/types/RowKind.html), example is: "+I(1,1)". + +The Print connector is built-in. + +Attention Print sinks print records in runtime tasks, you need to observe the task log. + +How to create a Print table +---------------- + ++ + + +Print +Condition1 +Condition2 ++ +PRINT_IDENTIFIER:taskId> output
PRINT_IDENTIFIER provided +parallelism > 1 ++ +PRINT_IDENTIFIER> output
PRINT_IDENTIFIER provided +parallelism == 1 ++ +taskId> output
no PRINT_IDENTIFIER provided +parallelism > 1 ++ + +output
no PRINT_IDENTIFIER provided +parallelism == 1 +++ +Alternatively, it may be based on an existing schema using the [LIKE Clause]({% link dev/table/sql/create.md %}#create-table). + ++{% highlight sql %} +CREATE TABLE print_table ( + f0 INT, + f1 INT, + f2 STRING, + f3 DOUBLE +) WITH ( + 'connector' = 'print' +) +{% endhighlight %} ++++ +Connector Options +---------------- + ++{% highlight sql %} +CREATE TABLE print_table WITH ('connector' = 'print') +LIKE source_table (EXCLUDING ALL) +{% endhighlight %} +++ +
diff --git a/docs/dev/table/connectors/print.zh.md b/docs/dev/table/connectors/print.zh.md new file mode 100644 index 00000000000000..64fad8782751f8 --- /dev/null +++ b/docs/dev/table/connectors/print.zh.md @@ -0,0 +1,143 @@ +--- +title: "Print SQL 连接器" +nav-title: Print +nav-parent_id: sql-connectors +nav-pos: 11 +--- + + +Sink + +* This will be replaced by the TOC +{:toc} + +Print 连接器允许将每一行写入标准输出流或者标准错误流。 + +设计目的： + +- 简单的流作业测试。 +- 对生产调试带来极大便利。 + +四种 format 选项： + ++ + + +Option +Required +Default +Type +Description ++ +connector
required +(none) +String +Specify what connector to use, here should be 'print'. ++ +print-identifier
optional +(none) +String +Message that identify print and is prefixed to the output of the value. ++ + +standard-error
optional +false +Boolean +True, if the format should print to standard error instead of standard out. ++ +
+ +输出字符串格式为 "$row_kind(f0,f1,f2...)"，row_kind是一个 [RowKind]({{ site.baseurl }}/api/java/org/apache/flink/types/RowKind.html) 类型的短字符串，例如："+I(1,1)"。 + +Print 连接器是内置的。 + +注意 在任务运行时使用 Print Sinks 打印记录，你需要注意观察任务日志。 + +如何创建一张基于 Print 的表 +---------------- + ++ + + +打印内容 +条件 1 +条件 2 ++ +标识符:任务 ID> 输出数据
需要提供前缀打印标识符 +parallelism > 1 ++ +标识符> 输出数据
需要提供前缀打印标识符 +parallelism == 1 ++ +任务 ID> 输出数据
不需要提供前缀打印标识符 +parallelism > 1 ++ + +输出数据
不需要提供前缀打印标识符 +parallelism == 1 +++ +或者，也可以通过 [LIKE子句]({% link dev/table/sql/create.zh.md %}#create-table) 基于已有表的结构去创建新表。 + ++{% highlight sql %} +CREATE TABLE print_table ( + f0 INT, + f1 INT, + f2 STRING, + f3 DOUBLE +) WITH ( + 'connector' = 'print' +) +{% endhighlight %} ++++ +连接器参数 +---------------- + ++{% highlight sql %} +CREATE TABLE print_table WITH ('connector' = 'print') +LIKE source_table (EXCLUDING ALL) +{% endhighlight %} +++ +
diff --git a/docs/dev/table/functions/udfs.md b/docs/dev/table/functions/udfs.md index 26f557a54f570a..70516b7af76b82 100644 --- a/docs/dev/table/functions/udfs.md +++ b/docs/dev/table/functions/udfs.md @@ -22,327 +22,867 @@ specific language governing permissions and limitations under the License. --> -User-defined functions are an important feature, because they significantly extend the expressiveness of queries. +User-defined functions (UDFs) are extension points to call frequently used logic or custom logic that cannot be expressed otherwise in queries. + +User-defined functions can be implemented in a JVM language (such as Java or Scala) or Python. An implementer can use arbitrary third party libraries within a UDF. This page will focus on JVM-based languages. * This will be replaced by the TOC {:toc} -Register User-Defined Functions -------------------------------- -In most cases, a user-defined function must be registered before it can be used in an query. It is not necessary to register functions for the Scala Table API. - -Functions are registered at the `TableEnvironment` by calling a `registerFunction()` method. When a user-defined function is registered, it is inserted into the function catalog of the `TableEnvironment` such that the Table API or SQL parser can recognize and properly translate it. +Overview +-------- -Please find detailed examples of how to register and how to call each type of user-defined function -(`ScalarFunction`, `TableFunction`, and `AggregateFunction`) in the following sub-sessions. +Currently, Flink distinguishes between the following kinds of functions: +- *Scalar functions* map scalar values to a new scalar value. +- *Table functions* map scalar values to new rows. +- *Aggregate functions* map scalar values of multiple rows to a new scalar value. +- *Table aggregate functions* map scalar values of multiple rows to new rows. +- *Async table functions* are special functions for table sources that perform a lookup. -{% top %} +Attention Scalar and table functions have been updated to the new type system based on [data types]({% link dev/table/types.md %}). Aggregating functions still use the old type system based on `TypeInformation`. -Scalar Functions ----------------- +The following example shows how to create a simple scalar function and how to call the function in both Table API and SQL. -If a required scalar function is not contained in the built-in functions, it is possible to define custom, user-defined scalar functions for both the Table API and SQL. A user-defined scalar functions maps zero, one, or multiple scalar values to a new scalar value. +For SQL queries, a function must always be registered under a name. For Table API, a function can be registered or directly used _inline_.+ + + +参数 +是否必选 +默认参数 +数据类型 +描述 ++ +connector
必选 +(none) +String +指定要使用的连接器，此处应为 'print' ++ +print-identifier
可选 +(none) +String +配置一个标识符作为输出数据的前缀。 ++ + +standard-error
可选 +false +Boolean +如果 format 需要打印为标准错误而不是标准输出，则为 True 。 +--In order to define a scalar function, one has to extend the base class `ScalarFunction` in `org.apache.flink.table.functions` and implement (one or more) evaluation methods. The behavior of a scalar function is determined by the evaluation method. An evaluation method must be declared publicly and named `eval`. The parameter types and return type of the evaluation method also determine the parameter and return types of the scalar function. Evaluation methods can also be overloaded by implementing multiple methods named `eval`. Evaluation methods can also support variable arguments, such as `eval(String... strs)`. - -The following example shows how to define your own hash code function, register it in the TableEnvironment, and call it in a query. Note that you can configure your scalar function via a constructor before it is registered: +-By default the result type of an evaluation method is determined by Flink's type extraction facilities. This is sufficient for basic types or simple POJOs but might be wrong for more complex, custom, or composite types. In these cases `TypeInformation` of the result type can be manually defined by overriding `ScalarFunction#getResultType()`. +For interactive sessions, it is also possible to parameterize functions before using or +registering them. In this case, function _instances_ instead of function _classes_ can be +used as temporary functions. -The following example shows an advanced example which takes the internal timestamp representation and also returns the internal timestamp representation as a long value. By overriding `ScalarFunction#getResultType()` we define that the returned long value should be interpreted as a `Types.TIMESTAMP` by the code generation. +It requires that the parameters are serializable for shipping +function instances to the cluster. +{% highlight java %} -public class HashCode extends ScalarFunction { - private int factor = 12; - - public HashCode(int factor) { - this.factor = factor; +import org.apache.flink.table.api.*; +import org.apache.flink.table.functions.ScalarFunction; +import static org.apache.flink.table.api.Expressions.*; + +// define function logic +public static class SubstringFunction extends ScalarFunction { + public String eval(String s, Integer begin, Integer end) { + return s.substring(begin, end); } - - public int eval(String s) { - return s.hashCode() * factor; +} + +TableEnvironment env = TableEnvironment.create(...); + +// call function "inline" without registration in Table API +env.from("MyTable").select(call(SubstringFunction.class, $("myField"), 5, 12)); + +// register function +env.createTemporarySystemFunction("SubstringFunction", SubstringFunction.class); + +// call registered function in Table API +env.from("MyTable").select(call("SubstringFunction", $("myField"), 5, 12)); + +// call registered function in SQL +env.sqlQuery("SELECT SubstringFunction(myField, 5, 12) FROM MyTable"); + +{% endhighlight %} ++ ++{% highlight scala %} +import org.apache.flink.table.api._ +import org.apache.flink.table.functions.ScalarFunction + +// define function logic +class SubstringFunction extends ScalarFunction { + def eval(s: String, begin: Integer, end: Integer): String = { + s.substring(begin, end) } } -BatchTableEnvironment tableEnv = BatchTableEnvironment.create(env); +val env = TableEnvironment.create(...) -// register the function -tableEnv.registerFunction("hashCode", new HashCode(10)); +// call function "inline" without registration in Table API +env.from("MyTable").select(call(classOf[SubstringFunction], $"myField", 5, 12)) -// use the function in Java Table API -myTable.select("string, string.hashCode(), hashCode(string)"); +// register function +env.createTemporarySystemFunction("SubstringFunction", classOf[SubstringFunction]) + +// call registered function in Table API +env.from("MyTable").select(call("SubstringFunction", $"myField", 5, 12)) + +// call registered function in SQL +env.sqlQuery("SELECT SubstringFunction(myField, 5, 12) FROM MyTable") -// use the function in SQL API -tableEnv.sqlQuery("SELECT string, hashCode(string) FROM MyTable"); {% endhighlight %} ++ ++ ++ +{% top %} + +Implementation Guide +-------------------- + +Attention This section only applies to scalar and table functions for now; until aggregate functions have been updated to the new type system. + +Independent of the kind of function, all user-defined functions follow some basic implementation principles. + +### Function Class + +An implementation class must extend from one of the available base classes (e.g. `org.apache.flink.table.functions.ScalarFunction`). + +The class must be declared `public`, not `abstract`, and should be globally accessible. Thus, non-static inner or anonymous classes are not allowed. + +For storing a user-defined function in a persistent catalog, the class must have a default constructor and must be instantiable during runtime. + +### Evaluation Methods + +The base class provides a set of methods that can be overridden such as `open()`, `close()`, or `isDeterministic()`. + +However, in addition to those declared methods, the main runtime logic that is applied to every incoming record must be implemented through specialized _evaluation methods_. + +Depending on the function kind, evaluation methods such as `eval()`, `accumulate()`, or `retract()` are called by code-generated operators during runtime. + +The methods must be declared `public` and take a well-defined set of arguments. + +Regular JVM method calling semantics apply. Therefore, it is possible to: +- implement overloaded methods such as `eval(Integer)` and `eval(LocalDateTime)`, +- use var-args such as `eval(Integer...)`, +- use object inheritance such as `eval(Object)` that takes both `LocalDateTime` and `Integer`, +- and combinations of the above such as `eval(Object...)` that takes all kinds of arguments. + +If you intend to implement functions in Scala, please add the `scala.annotation.varargs` annotation in +case of variable arguments. Furthermore, it is recommended to use boxed primitives (e.g. `java.lang.Integer` +instead of `Int`) to support `NULL`. + +The following snippets shows an example of an overloaded function: + +{% highlight java %} -public static class TimestampModifier extends ScalarFunction { - public long eval(long t) { - return t % 1000; +import org.apache.flink.table.api.*; +import org.apache.flink.table.functions.ScalarFunction; +import static org.apache.flink.table.api.Expressions.*; + +// define parameterizable function logic +public static class SubstringFunction extends ScalarFunction { + + private boolean endInclusive; + + public SubstringFunction(boolean endInclusive) { + this.endInclusive = endInclusive; } - public TypeInformation getResultType(Class[] signature) { - return Types.SQL_TIMESTAMP; + public String eval(String s, Integer begin, Integer end) { + return s.substring(a, endInclusive ? end + 1 : end); } } -{% endhighlight %} ---In order to define a scalar function, one has to extend the base class `ScalarFunction` in `org.apache.flink.table.functions` and implement (one or more) evaluation methods. The behavior of a scalar function is determined by the evaluation method. An evaluation method must be declared publicly and named `eval`. The parameter types and return type of the evaluation method also determine the parameter and return types of the scalar function. Evaluation methods can also be overloaded by implementing multiple methods named `eval`. Evaluation methods can also support variable arguments, such as `@varargs def eval(str: String*)`. +TableEnvironment env = TableEnvironment.create(...); -The following example shows how to define your own hash code function, register it in the TableEnvironment, and call it in a query. Note that you can configure your scalar function via a constructor before it is registered: +// call function "inline" without registration in Table API +env.from("MyTable").select(call(new SubstringFunction(true), $("myField"), 5, 12)); +// register function +env.createTemporarySystemFunction("SubstringFunction", new SubstringFunction(true)); + +{% endhighlight %} ++ +{% highlight scala %} -// must be defined in static/object context -class HashCode(factor: Int) extends ScalarFunction { - def eval(s: String): Int = { - s.hashCode() * factor +import org.apache.flink.table.api._ +import org.apache.flink.table.functions.ScalarFunction + +// define parameterizable function logic +class SubstringFunction(val endInclusive) extends ScalarFunction { + def eval(s: String, begin: Integer, end: Integer): String = { + s.substring(endInclusive ? end + 1 : end) } } -val tableEnv = BatchTableEnvironment.create(env) +val env = TableEnvironment.create(...) -// use the function in Scala Table API -val hashCode = new HashCode(10) -myTable.select('string, hashCode('string)) +// call function "inline" without registration in Table API +env.from("MyTable").select(call(new SubstringFunction(true), $"myField", 5, 12)) + +// register function +env.createTemporarySystemFunction("SubstringFunction", new SubstringFunction(true)) -// register and use the function in SQL -tableEnv.registerFunction("hashCode", new HashCode(10)) -tableEnv.sqlQuery("SELECT string, hashCode(string) FROM MyTable") {% endhighlight %} ++ ++ ++{% highlight java %} +import org.apache.flink.table.functions.ScalarFunction; -By default the result type of an evaluation method is determined by Flink's type extraction facilities. This is sufficient for basic types or simple POJOs but might be wrong for more complex, custom, or composite types. In these cases `TypeInformation` of the result type can be manually defined by overriding `ScalarFunction#getResultType()`. +// function with overloaded evaluation methods +public static class SumFunction extends ScalarFunction { -The following example shows an advanced example which takes the internal timestamp representation and also returns the internal timestamp representation as a long value. By overriding `ScalarFunction#getResultType()` we define that the returned long value should be interpreted as a `Types.TIMESTAMP` by the code generation. + public Integer eval(Integer a, Integer b) { + return a + b; + } + + public Integer eval(String a, String b) { + return Integer.valueOf(a) + Integer.valueOf(); + } + + public Integer eval(Double... d) { + double result = 0; + for (double value : d) + result += value; + return (int) result; + } +} + +{% endhighlight %} ++{% highlight scala %} -object TimestampModifier extends ScalarFunction { - def eval(t: Long): Long = { - t % 1000 +import org.apache.flink.table.functions.ScalarFunction +import java.lang.Integer +import java.lang.Double +import scala.annotation.varargs + +// function with overloaded evaluation methods +class SumFunction extends ScalarFunction { + + def eval(a: Integer, b: Integer): Integer = { + a + b } - override def getResultType(signature: Array[Class[_]]): TypeInformation[_] = { - Types.TIMESTAMP + def eval(a: String, b: String): Integer = { + Integer.valueOf(a) + Integer.valueOf(b) + } + + @varargs // generate var-args like Java + def eval(d: Double*): Integer = { + d.sum.toInt } } + {% endhighlight %}--In order to define a Python scalar function, one can extend the base class `ScalarFunction` in `pyflink.table.udf` and implement an evaluation method. The behavior of a Python scalar function is determined by the evaluation method which is named `eval`. +-The following example shows how to define your own Python hash code function, register it in the TableEnvironment, and call it in a query. Note that you can configure your scalar function via a constructor before it is registered: +### Type Inference -{% highlight python %} -class HashCode(ScalarFunction): - def __init__(self): - self.factor = 12 +The table ecosystem (similar to the SQL standard) is a strongly typed API. Therefore, both function parameters and return types must be mapped to a [data type]({% link dev/table/types.md %}). + +From a logical perspective, the planner needs information about expected types, precision, and scale. From a JVM perspective, the planner needs information about how internal data structures are represented as JVM objects when calling a user-defined function. + +The logic for validating input arguments and deriving data types for both the parameters and the result of a function is summarized under the term _type inference_. - def eval(self, s): - return hash(s) * self.factor +Flink's user-defined functions implement an automatic type inference extraction that derives data types from the function's class and its evaluation methods via reflection. If this implicit reflective extraction approach is not successful, the extraction process can be supported by annotating affected parameters, classes, or methods with `@DataTypeHint` and `@FunctionHint`. More examples on how to annotate functions are shown below. -table_env = BatchTableEnvironment.create(env) +If more advanced type inference logic is required, an implementer can explicitly override the `getTypeInference()` method in every user-defined function. However, the annotation approach is recommended because it keeps custom type inference logic close to the affected locations and falls back to the default behavior for the remaining implementation. -# register the Python function -table_env.register_function("hash_code", udf(HashCode(), DataTypes.BIGINT(), DataTypes.BIGINT())) +#### Automatic Type Inference -# use the function in Python Table API -my_table.select("string, bigint, string.hash_code(), hash_code(string)") +The automatic type inference inspects the function's class and evaluation methods to derive data types for the arguments and result of a function. `@DataTypeHint` and `@FunctionHint` annotations support the automatic extraction. + +For a full list of classes that can be implicitly mapped to a data type, see the [data type extraction section]({% link dev/table/types.md %}#data-type-extraction). + +**`@DataTypeHint`** + +In many scenarios, it is required to support the automatic extraction _inline_ for paramaters and return types of a function + +The following example shows how to use data type hints. More information can be found in the documentation of the annotation class. + ++ +-{% top %} +**`@FunctionHint`** -Table Functions ---------------- +In some scenarios, it is desirable that one evaluation method handles multiple different data types at the same time. Furthermore, in some scenarios, overloaded evaluation methods have a common result type that should be declared only once. + +The `@FunctionHint` annotation can provide a mapping from argument data types to a result data type. It enables annotating entire function classes or evaluation methods for input, accumulator, and result data types. One or more annotations can be declared on top of a class or individually for each evaluation method for overloading function signatures. All hint parameters are optional. If a parameter is not defined, the default reflection-based extraction is used. Hint parameters defined on top of a function class are inherited by all evaluation methods. -Similar to a user-defined scalar function, a user-defined table function takes zero, one, or multiple scalar values as input parameters. However in contrast to a scalar function, it can return an arbitrary number of rows as output instead of a single value. The returned rows may consist of one or more columns. +The following example shows how to use function hints. More information can be found in the documentation of the annotation class.+{% highlight java %} +import org.apache.flink.table.annotation.DataTypeHint; +import org.apache.flink.table.annotation.InputGroup; +import org.apache.flink.table.functions.ScalarFunction; +import org.apache.flink.types.Row; + +// function with overloaded evaluation methods +public static class OverloadedFunction extends ScalarFunction { + + // no hint required + public Long eval(long a, long b) { + return a + b; + } + + // define the precision and scale of a decimal + public @DataTypeHint("DECIMAL(12, 3)") BigDecimal eval(double a, double b) { + return BigDecimal.valueOf(a + b); + } + + // define a nested data type + @DataTypeHint("ROW+ +") + public Row eval(int i) { + return Row.of(String.valueOf(i), Instant.ofEpochSecond(i)); + } + + // allow wildcard input and customly serialized output + @DataTypeHint(value = "RAW", bridgedTo = ByteBuffer.class) + public ByteBuffer eval(@DataTypeHint(inputGroup = InputGroup.ANY) Object o) { + return MyUtils.serializeToByteBuffer(o); + } +} -# use the function in SQL API -table_env.sql_query("SELECT string, bigint, hash_code(bigint) FROM MyTable") {% endhighlight %} ++{% highlight scala %} +import org.apache.flink.table.annotation.DataTypeHint +import org.apache.flink.table.annotation.InputGroup +import org.apache.flink.table.functions.ScalarFunction +import org.apache.flink.types.Row +import scala.annotation.varargs + +// function with overloaded evaluation methods +class OverloadedFunction extends ScalarFunction { + + // no hint required + def eval(a: Long, b: Long): Long = { + a + b + } -There are many ways to define a Python scalar function besides extending the base class `ScalarFunction`. -Please refer to the [Python Scalar Function]({{ site.baseurl }}/dev/table/python/python_udfs.html#scalar-functions) documentation for more details. + // define the precision and scale of a decimal + @DataTypeHint("DECIMAL(12, 3)") + def eval(double a, double b): BigDecimal = { + java.lang.BigDecimal.valueOf(a + b) + } + + // define a nested data type + @DataTypeHint("ROW+") + def eval(Int i): Row = { + Row.of(java.lang.String.valueOf(i), java.time.Instant.ofEpochSecond(i)) + } + + // allow wildcard input and customly serialized output + @DataTypeHint(value = "RAW", bridgedTo = classOf[java.nio.ByteBuffer]) + def eval(@DataTypeHint(inputGroup = InputGroup.ANY) Object o): java.nio.ByteBuffer = { + MyUtils.serializeToByteBuffer(o) + } +} + +{% endhighlight %}-- -{% top %} - -Best Practices for Implementing UDFs ------------------------------------- - -The Table API and SQL code generation internally tries to work with primitive values as much as possible. A user-defined function can introduce much overhead through object creation, casting, and (un)boxing. Therefore, it is highly recommended to declare parameters and result types as primitive types instead of their boxed classes. `Types.DATE` and `Types.TIME` can also be represented as `int`. `Types.TIMESTAMP` can be represented as `long`. - -We recommended that user-defined functions should be written by Java instead of Scala as Scala types pose a challenge for Flink's type extractor. - -{% top %} - -Integrating UDFs with the Runtime ---------------------------------- - -Sometimes it might be necessary for a user-defined function to get global runtime information or do some setup/clean-up work before the actual work. User-defined functions provide `open()` and `close()` methods that can be overridden and provide similar functionality as the methods in `RichFunction` of DataSet or DataStream API. - -The `open()` method is called once before the evaluation method. The `close()` method after the last call to the evaluation method. - -The `open()` method provides a `FunctionContext` that contains information about the context in which user-defined functions are executed, such as the metric group, the distributed cache files, or the global job parameters. - -The following information can be obtained by calling the corresponding methods of `FunctionContext`: - -| Method | Description | -| :------------------------------------ | :----------------------------------------------------- | -| `getMetricGroup()` | Metric group for this parallel subtask. | -| `getCachedFile(name)` | Local temporary file copy of a distributed cache file. | -| `getJobParameter(name, defaultValue)` | Global job parameter value associated with given key. | - -The following example snippet shows how to use `FunctionContext` in a scalar function for accessing a global job parameter: - --In order to define a table function one has to extend the base class `TableFunction` in `org.apache.flink.table.functions` and implement (one or more) evaluation methods. The behavior of a table function is determined by its evaluation methods. An evaluation method must be declared `public` and named `eval`. The `TableFunction` can be overloaded by implementing multiple methods named `eval`. The parameter types of the evaluation methods determine all valid parameters of the table function. Evaluation methods can also support variable arguments, such as `eval(String... strs)`. The type of the returned table is determined by the generic type of `TableFunction`. Evaluation methods emit output rows using the protected `collect(T)` method. -In the Table API, a table function is used with `.joinLateral` or `.leftOuterJoinLateral`. The `joinLateral` operator (cross) joins each row from the outer table (table on the left of the operator) with all rows produced by the table-valued function (which is on the right side of the operator). The `leftOuterJoinLateral` operator joins each row from the outer table (table on the left of the operator) with all rows produced by the table-valued function (which is on the right side of the operator) and preserves outer rows for which the table function returns an empty table. In SQL use `LATERAL TABLE(+ +#### Custom Type Inference + +For most scenarios, `@DataTypeHint` and `@FunctionHint` should be sufficient to model user-defined functions. However, by overriding the automatic type inference defined in `getTypeInference()`, implementers can create arbitrary functions that behave like built-in system functions. + +The following example implemented in Java illustrates the potential of a custom type inference logic. It uses a string literal argument to determine the result type of a function. The function takes two string arguments: the first argument represents the string to be parsed, the second argument represents the target type. -The following example shows how to define table-valued function, register it in the TableEnvironment, and call it in a query. Note that you can configure your table function via a constructor before it is registered: +)` with CROSS JOIN and LEFT JOIN with an ON TRUE join condition (see examples below). + +{% highlight java %} +import org.apache.flink.table.annotation.DataTypeHint; +import org.apache.flink.table.annotation.FunctionHint; +import org.apache.flink.table.functions.TableFunction; +import org.apache.flink.types.Row; + +// function with overloaded evaluation methods +// but globally defined output type +@FunctionHint(output = @DataTypeHint("ROW+ +")) +public static class OverloadedFunction extends TableFunction{ + + public void eval(int a, int b) { + collect(Row.of("Sum", a + b)); + } + + // overloading of arguments is still possible + public void eval() { + collect(Row.of("Empty args", -1)); + } +} + +// decouples the type inference from evaluation methods, +// the type inference is entirely determined by the function hints +@FunctionHint( + input = [@DataTypeHint("INT"), @DataTypeHint("INT")], + output = @DataTypeHint("INT") +) +@FunctionHint( + input = [@DataTypeHint("LONG"), @DataTypeHint("LONG")], + output = @DataTypeHint("LONG") +) +@FunctionHint( + input = [], + output = @DataTypeHint("BOOLEAN") +) +public static class OverloadedFunction extends TableFunction
+{% highlight scala %} + +import org.apache.flink.table.annotation.DataTypeHint +import org.apache.flink.table.annotation.FunctionHint +import org.apache.flink.table.functions.TableFunction +import org.apache.flink.types.Row + +// function with overloaded evaluation methods +// but globally defined output type +@FunctionHint(output = @DataTypeHint("ROW+ +")) +class OverloadedFunction extends TableFunction[Row] { + + def eval(a: Int, b: Int): Unit = { + collect(Row.of("Sum", Int.box(a + b))) + } + + // overloading of arguments is still possible + def eval(): Unit = { + collect(Row.of("Empty args", Int.box(-1))) + } +} + +// decouples the type inference from evaluation methods, +// the type inference is entirely determined by the function hints +@FunctionHint( + input = Array(@DataTypeHint("INT"), @DataTypeHint("INT")), + output = @DataTypeHint("INT") +) +@FunctionHint( + input = Array(@DataTypeHint("LONG"), @DataTypeHint("LONG")), + output = @DataTypeHint("LONG") +) +@FunctionHint( + input = Array(), + output = @DataTypeHint("BOOLEAN") +) +class OverloadedFunction extends TableFunction[AnyRef] { + + // an implementer just needs to make sure that a method exists + // that can be called by the JVM + @varargs + def eval(o: AnyRef*) = { + if (o.length == 0) { + collect(Boolean.box(false)) + } + collect(o(0)) + } +} + +{% endhighlight %} +++ +### Runtime Integration + +Sometimes it might be necessary for a user-defined function to get global runtime information or do some setup/clean-up work before the actual work. User-defined functions provide `open()` and `close()` methods that can be overridden and provide similar functionality as the methods in `RichFunction` of DataStream API. + +The `open()` method is called once before the evaluation method. The `close()` method after the last call to the evaluation method. + +The `open()` method provides a `FunctionContext` that contains information about the context in which user-defined functions are executed, such as the metric group, the distributed cache files, or the global job parameters. + +The following information can be obtained by calling the corresponding methods of `FunctionContext`: + +| Method | Description | +| :------------------------------------ | :----------------------------------------------------- | +| `getMetricGroup()` | Metric group for this parallel subtask. | +| `getCachedFile(name)` | Local temporary file copy of a distributed cache file. | +| `getJobParameter(name, defaultValue)` | Global job parameter value associated with given key. | + +The following example snippet shows how to use `FunctionContext` in a scalar function for accessing a global job parameter: + +{% highlight java %} -// The generic type "Tuple2+ ++ ++{% highlight java %} +import org.apache.flink.table.api.*; +import org.apache.flink.table.functions.FunctionContext; +import org.apache.flink.table.functions.ScalarFunction; + +public static class HashCodeFunction extends ScalarFunction { + + private int factor = 0; + + @Override + public void open(FunctionContext context) throws Exception { + // access the global "hashcode_factor" parameter + // "12" would be the default value if the parameter does not exist + factor = Integer.parseInt(context.getJobParameter("hashcode_factor", "12")); + } + + public int eval(String s) { + return s.hashCode() * factor; } } -BatchTableEnvironment tableEnv = BatchTableEnvironment.create(env); -Table myTable = ... // table schema: [a: String] +TableEnvironment env = TableEnvironment.create(...); -// Register the function. -tableEnv.registerFunction("split", new Split("#")); +// add job parameter +env.getConfig().addJobParameter("hashcode_factor", "31"); -// Use the table function in the Java Table API. "as" specifies the field names of the table. -myTable.joinLateral("split(a) as (word, length)") - .select("a, word, length"); -myTable.leftOuterJoinLateral("split(a) as (word, length)") - .select("a, word, length"); +// register the function +env.createTemporarySystemFunction("hashCode", HashCodeFunction.class); + +// use the function +env.sqlQuery("SELECT myField, hashCode(myField) FROM MyTable"); -// Use the table function in SQL with LATERAL and TABLE keywords. -// CROSS JOIN a table function (equivalent to "join" in Table API). -tableEnv.sqlQuery("SELECT a, word, length FROM MyTable, LATERAL TABLE(split(a)) as T(word, length)"); -// LEFT JOIN a table function (equivalent to "leftOuterJoin" in Table API). -tableEnv.sqlQuery("SELECT a, word, length FROM MyTable LEFT JOIN LATERAL TABLE(split(a)) as T(word, length) ON TRUE"); {% endhighlight %}-In order to define a table function one has to extend the base class `TableFunction` in `org.apache.flink.table.functions` and implement (one or more) evaluation methods. The behavior of a table function is determined by its evaluation methods. An evaluation method must be declared `public` and named `eval`. The `TableFunction` can be overloaded by implementing multiple methods named `eval`. The parameter types of the evaluation methods determine all valid parameters of the table function. Evaluation methods can also support variable arguments, such as `eval(String... strs)`. The type of the returned table is determined by the generic type of `TableFunction`. Evaluation methods emit output rows using the protected `collect(T)` method. +{% highlight scala %} +import org.apache.flink.table.api._ +import org.apache.flink.table.functions.FunctionContext +import org.apache.flink.table.functions.ScalarFunction -In the Table API, a table function is used with `.joinLateral` or `.leftOuterJoinLateral`. The `joinLateral` operator (cross) joins each row from the outer table (table on the left of the operator) with all rows produced by the table-valued function (which is on the right side of the operator). The `leftOuterJoinLateral` operator joins each row from the outer table (table on the left of the operator) with all rows produced by the table-valued function (which is on the right side of the operator) and preserves outer rows for which the table function returns an empty table. In SQL use `LATERAL TABLE(-)` with CROSS JOIN and LEFT JOIN with an ON TRUE join condition (see examples below). +class HashCodeFunction extends ScalarFunction { -The following example shows how to define table-valued function, register it in the TableEnvironment, and call it in a query. Note that you can configure your table function via a constructor before it is registered: + private var factor: Int = 0 -{% highlight scala %} -// The generic type "(String, Int)" determines the schema of the returned table as (String, Integer). -class Split(separator: String) extends TableFunction[(String, Int)] { - def eval(str: String): Unit = { - // use collect(...) to emit a row. - str.split(separator).foreach(x => collect((x, x.length))) + override def open(context: FunctionContext): Unit = { + // access the global "hashcode_factor" parameter + // "12" would be the default value if the parameter does not exist + factor = context.getJobParameter("hashcode_factor", "12").toInt + } + + def eval(s: String): Int = { + s.hashCode * factor } } -val tableEnv = BatchTableEnvironment.create(env) -val myTable = ... // table schema: [a: String] +val env = TableEnvironment.create(...) + +// add job parameter +env.getConfig.addJobParameter("hashcode_factor", "31") -// Use the table function in the Scala Table API (Note: No registration required in Scala Table API). -val split = new Split("#") -// "as" specifies the field names of the generated table. -myTable.joinLateral(split('a) as ('word, 'length)).select('a, 'word, 'length) -myTable.leftOuterJoinLateral(split('a) as ('word, 'length)).select('a, 'word, 'length) +// register the function +env.createTemporarySystemFunction("hashCode", classOf[HashCodeFunction]) -// Register the table function to use it in SQL queries. -tableEnv.registerFunction("split", new Split("#")) +// use the function +env.sqlQuery("SELECT myField, hashCode(myField) FROM MyTable") -// Use the table function in SQL with LATERAL and TABLE keywords. -// CROSS JOIN a table function (equivalent to "join" in Table API) -tableEnv.sqlQuery("SELECT a, word, length FROM MyTable, LATERAL TABLE(split(a)) as T(word, length)") -// LEFT JOIN a table function (equivalent to "leftOuterJoin" in Table API) -tableEnv.sqlQuery("SELECT a, word, length FROM MyTable LEFT JOIN LATERAL TABLE(split(a)) as T(word, length) ON TRUE") {% endhighlight %} -**IMPORTANT:** Do not implement TableFunction as a Scala object. Scala object is a singleton and will cause concurrency issues. -In order to define a Python table function, one can extend the base class `TableFunction` in `pyflink.table.udtf` and Implement an evaluation method. The behavior of a Python table function is determined by the evaluation method which is named eval. +-In the Python Table API, a Python table function is used with `.join_lateral` or `.left_outer_join_lateral`. The `join_lateral` operator (cross) joins each row from the outer table (table on the left of the operator) with all rows produced by the table-valued function (which is on the right side of the operator). The `left_outer_join_lateral` operator joins each row from the outer table (table on the left of the operator) with all rows produced by the table-valued function (which is on the right side of the operator) and preserves outer rows for which the table function returns an empty table. In SQL use `LATERAL TABLE()` with CROSS JOIN and LEFT JOIN with an ON TRUE join condition (see examples below). +{% top %} -The following example shows how to define a Python table function, registered it in the TableEnvironment, and call it in a query. Note that you can configure your table function via a constructor before it is registered: +Scalar Functions +---------------- -{% highlight python %} -class Split(TableFunction): - def eval(self, string): - for s in string.split(" "): - yield s, len(s) +A user-defined scalar function maps zero, one, or multiple scalar values to a new scalar value. Any data type listed in the [data types section]({% link dev/table/types.md %}) can be used as a parameter or return type of an evaluation method. + +In order to define a scalar function, one has to extend the base class `ScalarFunction` in `org.apache.flink.table.functions` and implement one or more evaluation methods named `eval(...)`. + +The following example shows how to define your own hash code function and call it in a query. See the [Implementation Guide](#implementation-guide) for more details. -env = StreamExecutionEnvironment.get_execution_environment() -table_env = StreamTableEnvironment.create(env) -my_table = ... # type: Table, table schema: [a: String] + + +-+{% highlight java %} +import org.apache.flink.table.annotation.InputGroup; +import org.apache.flink.table.api.*; +import org.apache.flink.table.functions.ScalarFunction; +import static org.apache.flink.table.api.Expressions.*; -# register the Python Table Function -table_env.register_function("split", udtf(Split(), DataTypes.STRING(), [DataTypes.STRING(), DataTypes.INT()])) +public static class HashFunction extends ScalarFunction { + + // take any data type and return INT + public int eval(@DataTypeHint(inputGroup = InputGroup.ANY) Object o) { + return o.hashCode(); + } +} -# use the Python Table Function in Python Table API -my_table.join_lateral("split(a) as (word, length)") -my_table.left_outer_join_lateral("split(a) as (word, length)") +TableEnvironment env = TableEnvironment.create(...); -# use the Python Table function in SQL API -table_env.sql_query("SELECT a, word, length FROM MyTable, LATERAL TABLE(split(a)) as T(word, length)") -table_env.sql_query("SELECT a, word, length FROM MyTable LEFT JOIN LATERAL TABLE(split(a)) as T(word, length) ON TRUE") +// call function "inline" without registration in Table API +env.from("MyTable").select(call(HashFunction.class, $("myField"))); + +// register function +env.createTemporarySystemFunction("HashFunction", HashFunction.class); + +// call registered function in Table API +env.from("MyTable").select(call("HashFunction", $("myField"))); + +// call registered function in SQL +env.sqlQuery("SELECT HashFunction(myField) FROM MyTable"); {% endhighlight %} +-There are many ways to define a Python table function besides extending the base class `TableFunction`. -Please refer to the [Python Table Function]({{ site.baseurl }}/dev/table/python/python_udfs.html#table-functions) documentation for more details. - ++{% highlight scala %} +import org.apache.flink.table.annotation.InputGroup +import org.apache.flink.table.api._ +import org.apache.flink.table.functions.ScalarFunction + +class HashFunction extends ScalarFunction { + + // take any data type and return INT + def eval(@DataTypeHint(inputGroup = InputGroup.ANY) o: AnyRef): Int { + return o.hashCode(); + } +} + +val env = TableEnvironment.create(...) + +// call function "inline" without registration in Table API +env.from("MyTable").select(call(classOf[HashFunction], $"myField")) + +// register function +env.createTemporarySystemFunction("HashFunction", classOf[HashFunction]) + +// call registered function in Table API +env.from("MyTable").select(call("HashFunction", $"myField")) + +// call registered function in SQL +env.sqlQuery("SELECT HashFunction(myField) FROM MyTable") + +{% endhighlight %}+-- {% top %} Table Aggregation Functions @@ -1420,104 +1959,4 @@ tab-Please note that POJO types do not have a deterministic field order. Therefore, you cannot rename the fields of POJO returned by a table function using `AS`. +If you intend to implement or call functions in Python, please refer to the [Python Scalar Functions]({% link dev/table/python/python_udfs.md %}#scalar-functions) documentation for more details. -By default the result type of a `TableFunction` is determined by Flink’s automatic type extraction facilities. This works well for basic types and simple POJOs but might be wrong for more complex, custom, or composite types. In such a case, the type of the result can be manually specified by overriding `TableFunction#getResultType()` which returns its `TypeInformation`. +{% top %} -The following example shows an example of a `TableFunction` that returns a `Row` type which requires explicit type information. We define that the returned table type should be `RowTypeInfo(String, Integer)` by overriding `TableFunction#getResultType()`. +Table Functions +--------------- -{% highlight java %} -public class CustomTypeSplit extends TableFunction{ - public void eval(String str) { - for (String s : str.split(" ")) { - Row row = new Row(2); - row.setField(0, s); - row.setField(1, s.length()); - collect(row); - } - } +Similar to a user-defined scalar function, a user-defined table function takes zero, one, or multiple scalar values as input arguments. However, in contrast to a scalar function, it can return an arbitrary number of rows (or structured types) as output instead of a single value. The returned record may consist of one or more fields. If an output record consists of only one field, the structured record can be omitted and a scalar value can be emitted. It will be wrapped into an implicit row by the runtime. - @Override - public TypeInformation
getResultType() { - return Types.ROW(Types.STRING(), Types.INT()); +In order to define a table function, one has to extend the base class `TableFunction` in `org.apache.flink.table.functions` and implement one or more evaluation methods named `eval(...)`. Similar to other functions, input and output data types are automatically extracted using reflection. This includes the generic argument `T` of the class for determining an output data type. In contrast to scalar functions, the evaluation method itself must not have a return type, instead, table functions provide a `collect(T)` method that can be called within every evaluation method for emitting zero, one, or more records. + +In the Table API, a table function is used with `.joinLateral(...)` or `.leftOuterJoinLateral(...)`. The `joinLateral` operator (cross) joins each row from the outer table (table on the left of the operator) with all rows produced by the table-valued function (which is on the right side of the operator). The `leftOuterJoinLateral` operator joins each row from the outer table (table on the left of the operator) with all rows produced by the table-valued function (which is on the right side of the operator) and preserves outer rows for which the table function returns an empty table. + +In SQL, use `LATERAL TABLE(
)` with `JOIN` or `LEFT JOIN` with an `ON TRUE` join condition. + +The following example shows how to define your own split function and call it in a query. See the [Implementation Guide](#implementation-guide) for more details. + + + +-{% top %} +If you intend to implement functions in Scala, do not implement a table function as a Scala `object`. Scala `object`s are singletons and will cause concurrency issues. +If you intend to implement or call functions in Python, please refer to the [Python Table Functions]({% link dev/table/python/python_udfs.md %}#table-functions) documentation for more details. + +{% top %} Aggregation Functions --------------------- @@ -821,7 +1361,6 @@ t_env.sql_query("SELECT user, wAvg(points, level) AS avgPoints FROM userScores G+{% highlight java %} +import org.apache.flink.table.annotation.DataTypeHint; +import org.apache.flink.table.annotation.FunctionHint; +import org.apache.flink.table.api.*; +import org.apache.flink.table.functions.TableFunction; +import org.apache.flink.types.Row; +import static org.apache.flink.table.api.Expressions.*; + +@FunctionHint(output = @DataTypeHint("ROW-")) +public static class SplitFunction extends TableFunction { + + public void eval(String str) { + for (String s : str.split(" ")) { + // use collect(...) to emit a row + collect(Row.of(s, s.length())); } + } } -{% endhighlight %} -
-Please note that POJO types do not have a deterministic field order. Therefore, you cannot rename the fields of POJO returned by a table function using `AS`. +TableEnvironment env = TableEnvironment.create(...); + +// call function "inline" without registration in Table API +env + .from("MyTable") + .joinLateral(call(SplitFunction.class, $("myField"))) + .select($("myField"), $("word"), $("length")); +env + .from("MyTable") + .leftOuterJoinLateral(call(SplitFunction.class, $("myField"))) + .select($("myField"), $("word"), $("length")); -By default the result type of a `TableFunction` is determined by Flink’s automatic type extraction facilities. This works well for basic types and simple POJOs but might be wrong for more complex, custom, or composite types. In such a case, the type of the result can be manually specified by overriding `TableFunction#getResultType()` which returns its `TypeInformation`. +// rename fields of the function in Table API +env + .from("MyTable") + .leftOuterJoinLateral(call(SplitFunction.class, $("myField")).as("newWord", "newLength")) + .select($("myField"), $("newWord"), $("newLength")); -The following example shows an example of a `TableFunction` that returns a `Row` type which requires explicit type information. We define that the returned table type should be `RowTypeInfo(String, Integer)` by overriding `TableFunction#getResultType()`. +// register function +env.createTemporarySystemFunction("SplitFunction", SplitFunction.class); + +// call registered function in Table API +env + .from("MyTable") + .joinLateral(call("SplitFunction", $("myField"))) + .select($("myField"), $("word"), $("length")); +env + .from("MyTable") + .leftOuterJoinLateral(call("SplitFunction", $("myField"))) + .select($("myField"), $("word"), $("length")); + +// call registered function in SQL +env.sqlQuery( + "SELECT myField, word, length " + + "FROM MyTable, LATERAL TABLE(SplitFunction(myField))"); +env.sqlQuery( + "SELECT myField, word, length " + + "FROM MyTable " + + "LEFT JOIN LATERAL TABLE(SplitFunction(myField)) ON TRUE"); + +// rename fields of the function in SQL +env.sqlQuery( + "SELECT myField, newWord, newLength " + + "FROM MyTable " + + "LEFT JOIN LATERAL TABLE(SplitFunction(myField)) AS T(newWord, newLength) ON TRUE"); +{% endhighlight %} ++ +{% highlight scala %} -class CustomTypeSplit extends TableFunction[Row] { - def eval(str: String): Unit = { - str.split(" ").foreach({ s => - val row = new Row(2) - row.setField(0, s) - row.setField(1, s.length) - collect(row) - }) - } +import org.apache.flink.table.annotation.DataTypeHint +import org.apache.flink.table.annotation.FunctionHint +import org.apache.flink.table.api._ +import org.apache.flink.table.functions.TableFunction +import org.apache.flink.types.Row - override def getResultType: TypeInformation[Row] = { - Types.ROW(Types.STRING, Types.INT) +@FunctionHint(output = @DataTypeHint("ROW+")) +class SplitFunction extends TableFunction[Row] { + + def eval(str: String): Unit = { + // use collect(...) to emit a row + str.split(" ").foreach(s => collect(Row.of(s, s.length))) } } + +val env = TableEnvironment.create(...) + +// call function "inline" without registration in Table API +env + .from("MyTable") + .joinLateral(call(classOf[SplitFunction], $"myField") + .select($"myField", $"word", $"length") +env + .from("MyTable") + .leftOuterJoinLateral(call(classOf[SplitFunction], $"myField")) + .select($"myField", $"word", $"length") + +// rename fields of the function in Table API +env + .from("MyTable") + .leftOuterJoinLateral(call(classOf[SplitFunction], $"myField").as("newWord", "newLength")) + .select($"myField", $"newWord", $"newLength") + +// register function +env.createTemporarySystemFunction("SplitFunction", classOf[SplitFunction]) + +// call registered function in Table API +env + .from("MyTable") + .joinLateral(call("SplitFunction", $"myField")) + .select($"myField", $"word", $"length") +env + .from("MyTable") + .leftOuterJoinLateral(call("SplitFunction", $"myField")) + .select($"myField", $"word", $"length") + +// call registered function in SQL +env.sqlQuery( + "SELECT myField, word, length " + + "FROM MyTable, LATERAL TABLE(SplitFunction(myField))"); +env.sqlQuery( + "SELECT myField, word, length " + + "FROM MyTable " + + "LEFT JOIN LATERAL TABLE(SplitFunction(myField)) ON TRUE") + +// rename fields of the function in SQL +env.sqlQuery( + "SELECT myField, newWord, newLength " + + "FROM MyTable " + + "LEFT JOIN LATERAL TABLE(SplitFunction(myField)) AS T(newWord, newLength) ON TRUE") + {% endhighlight %} -- {% top %} - diff --git a/docs/dev/table/functions/udfs.zh.md b/docs/dev/table/functions/udfs.zh.md index c0e7c67a20bbdd..4e3a6be02c9c09 100644 --- a/docs/dev/table/functions/udfs.zh.md +++ b/docs/dev/table/functions/udfs.zh.md @@ -22,364 +22,887 @@ specific language governing permissions and limitations under the License. --> -User-defined functions are an important feature, because they significantly extend the expressiveness of queries. +自定义函数（UDF）是一种扩展开发机制，可以用来在查询语句里调用难以用其他方式表达的频繁使用或自定义的逻辑。 + +自定义函数可以用 JVM 语言（例如 Java 或 Scala）或 Python 实现，实现者可以在 UDF 中使用任意第三方库，本文聚焦于使用 JVM 语言开发自定义函数。 * This will be replaced by the TOC {:toc} -Register User-Defined Functions -------------------------------- -In most cases, a user-defined function must be registered before it can be used in an query. It is not necessary to register functions for the Scala Table API. - -Functions are registered at the `TableEnvironment` by calling a `registerFunction()` method. When a user-defined function is registered, it is inserted into the function catalog of the `TableEnvironment` such that the Table API or SQL parser can recognize and properly translate it. +概述 +-------- -Please find detailed examples of how to register and how to call each type of user-defined function -(`ScalarFunction`, `TableFunction`, and `AggregateFunction`) in the following sub-sessions. +当前 Flink 有如下几种函数： +- *标量函数* 将标量值转换成一个新标量值； +- *表值函数* 将标量值转换成新的行数据； +- *聚合函数* 将多行数据里的标量值转换成一个新标量值； +- *表值聚合函数* 将多行数据里的标量值转换成新的行数据； +- *异步表值函数* 是异步查询外部数据系统的特殊函数。 -{% top %} +注意 标量和表值函数已经使用了新的基于[数据类型]({% link dev/table/types.zh.md %})的类型系统，聚合函数仍然使用基于 `TypeInformation` 的旧类型系统。 -Scalar Functions ----------------- +以下示例展示了如何创建一个基本的标量函数，以及如何在 Table API 和 SQL 里调用这个函数。 -If a required scalar function is not contained in the built-in functions, it is possible to define custom, user-defined scalar functions for both the Table API and SQL. A user-defined scalar functions maps zero, one, or multiple scalar values to a new scalar value. +函数用于 SQL 查询前要先经过注册；而在用于 Table API 时，函数可以先注册后调用，也可以 _内联_ 后直接使用。-{% highlight java %} -public class HashCode extends ScalarFunction { - - private int factor = 0; - - @Override - public void open(FunctionContext context) throws Exception { - // access "hashcode_factor" parameter - // "12" would be the default value if parameter does not exist - factor = Integer.valueOf(context.getJobParameter("hashcode_factor", "12")); - } - - public int eval(String s) { - return s.hashCode() * factor; - } -} - -ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment(); -BatchTableEnvironment tableEnv = BatchTableEnvironment.create(env); - -// set job parameter -Configuration conf = new Configuration(); -conf.setString("hashcode_factor", "31"); -env.getConfig().setGlobalJobParameters(conf); - -// register the function -tableEnv.registerFunction("hashCode", new HashCode()); - -// use the function in Java Table API -myTable.select("string, string.hashCode(), hashCode(string)"); - -// use the function in SQL -tableEnv.sqlQuery("SELECT string, HASHCODE(string) FROM MyTable"); -{% endhighlight %} -- --{% highlight scala %} -object hashCode extends ScalarFunction { - - var hashcode_factor = 12 - - override def open(context: FunctionContext): Unit = { - // access "hashcode_factor" parameter - // "12" would be the default value if parameter does not exist - hashcode_factor = context.getJobParameter("hashcode_factor", "12").toInt - } - - def eval(s: String): Int = { - s.hashCode() * hashcode_factor - } -} - -val tableEnv = BatchTableEnvironment.create(env) - -// use the function in Scala Table API -myTable.select('string, hashCode('string)) - -// register and use the function in SQL -tableEnv.registerFunction("hashCode", hashCode) -tableEnv.sqlQuery("SELECT string, HASHCODE(string) FROM MyTable") -{% endhighlight %} - ----In order to define a scalar function, one has to extend the base class `ScalarFunction` in `org.apache.flink.table.functions` and implement (one or more) evaluation methods. The behavior of a scalar function is determined by the evaluation method. An evaluation method must be declared publicly and named `eval`. The parameter types and return type of the evaluation method also determine the parameter and return types of the scalar function. Evaluation methods can also be overloaded by implementing multiple methods named `eval`. Evaluation methods can also support variable arguments, such as `eval(String... strs)`. - -The following example shows how to define your own hash code function, register it in the TableEnvironment, and call it in a query. Note that you can configure your scalar function via a constructor before it is registered: ++ +对于交互式会话，还可以在使用或注册函数之前对其进行参数化，这样可以把函数 _实例_ 而不是函数 _类_ 用作临时函数。 -The following example shows an advanced example which takes the internal timestamp representation and also returns the internal timestamp representation as a long value. By overriding `ScalarFunction#getResultType()` we define that the returned long value should be interpreted as a `Types.TIMESTAMP` by the code generation. +为确保函数实例可应用于集群环境，参数必须是可序列化的。 +{% highlight java %} -public class HashCode extends ScalarFunction { - private int factor = 12; - - public HashCode(int factor) { - this.factor = factor; +import org.apache.flink.table.api.*; +import org.apache.flink.table.functions.ScalarFunction; +import static org.apache.flink.table.api.Expressions.*; + +// 定义函数逻辑 +public static class SubstringFunction extends ScalarFunction { + public String eval(String s, Integer begin, Integer end) { + return s.substring(begin, end); } - - public int eval(String s) { - return s.hashCode() * factor; +} + +TableEnvironment env = TableEnvironment.create(...); + +// 在 Table API 里不经注册直接“内联”调用函数 +env.from("MyTable").select(call(SubstringFunction.class, $("myField"), 5, 12)); + +// 注册函数 +env.createTemporarySystemFunction("SubstringFunction", SubstringFunction.class); + +// 在 Table API 里调用注册好的函数 +env.from("MyTable").select(call("SubstringFunction", $("myField"), 5, 12)); + +// 在 SQL 里调用注册好的函数 +env.sqlQuery("SELECT SubstringFunction(myField, 5, 12) FROM MyTable"); + +{% endhighlight %} ++ ++{% highlight scala %} +import org.apache.flink.table.api._ +import org.apache.flink.table.functions.ScalarFunction + +// define function logic +class SubstringFunction extends ScalarFunction { + def eval(s: String, begin: Integer, end: Integer): String = { + s.substring(begin, end) } } -BatchTableEnvironment tableEnv = BatchTableEnvironment.create(env); +val env = TableEnvironment.create(...) + +// 在 Table API 里不经注册直接“内联”调用函数 +env.from("MyTable").select(call(classOf[SubstringFunction], $"myField", 5, 12)) -// register the function -tableEnv.registerFunction("hashCode", new HashCode(10)); +// 注册函数 +env.createTemporarySystemFunction("SubstringFunction", classOf[SubstringFunction]) -// use the function in Java Table API -myTable.select("string, string.hashCode(), hashCode(string)"); +// 在 Table API 里调用注册好的函数 +env.from("MyTable").select(call("SubstringFunction", $"myField", 5, 12)) + +// 在 SQL 里调用注册好的函数 +env.sqlQuery("SELECT SubstringFunction(myField, 5, 12) FROM MyTable") -// use the function in SQL API -tableEnv.sqlQuery("SELECT string, hashCode(string) FROM MyTable"); {% endhighlight %} +-By default the result type of an evaluation method is determined by Flink's type extraction facilities. This is sufficient for basic types or simple POJOs but might be wrong for more complex, custom, or composite types. In these cases `TypeInformation` of the result type can be manually defined by overriding `ScalarFunction#getResultType()`. ++ +-The following example shows an advanced example which takes the internal timestamp representation and also returns the internal timestamp representation as a long value. By overriding `ScalarFunction#getResultType()` we define that the returned long value should be interpreted as a `Types.TIMESTAMP` by the code generation. +{% top %} +开发指南 +-------- + +注意在聚合函数使用新的类型系统前，本节仅适用于标量和表值函数。 + +所有的自定义函数都遵循一些基本的实现原则。 + +### 函数类 + +实现类必须继承自合适的基类之一（例如 `org.apache.flink.table.functions.ScalarFunction` ）。 + +该类必须声明为 `public` ，而不是 `abstract` ，并且可以被全局访问。不允许使用非静态内部类或匿名类。 + +为了将自定义函数存储在持久化的 catalog 中，该类必须具有默认构造器，且在运行时可实例化。 + +### 求值方法 + +基类提供了一组可以被重写的方法，例如 `open()`、 `close()` 或 `isDeterministic()` 。 + +但是，除了上述方法之外，作用于每条传入记录的主要逻辑还必须通过专门的 _求值方法_ 来实现。 + +根据函数的种类，后台生成的运算符会在运行时调用诸如 `eval()`、`accumulate()` 或 `retract()` 之类的求值方法。 + +这些方法必须声明为 `public` ，并带有一组定义明确的参数。 + +常规的 JVM 方法调用语义是适用的。因此可以： +- 实现重载的方法，例如 `eval(Integer)` 和 `eval(LocalDateTime)`； +- 使用变长参数，例如 `eval(Integer...)`; +- 使用对象继承，例如 `eval(Object)` 可接受 `LocalDateTime` 和 `Integer` 作为参数； +- 也可组合使用，例如 `eval(Object...)` 可接受所有类型的参数。 + +以下代码片段展示了一个重载函数的示例： + +{% highlight java %} -public static class TimestampModifier extends ScalarFunction { - public long eval(long t) { - return t % 1000; +import org.apache.flink.table.api.*; +import org.apache.flink.table.functions.ScalarFunction; +import static org.apache.flink.table.api.Expressions.*; + +// 定义可参数化的函数逻辑 +public static class SubstringFunction extends ScalarFunction { + + private boolean endInclusive; + + public SubstringFunction(boolean endInclusive) { + this.endInclusive = endInclusive; } - public TypeInformation getResultType(Class[] signature) { - return Types.SQL_TIMESTAMP; + public String eval(String s, Integer begin, Integer end) { + return s.substring(a, endInclusive ? end + 1 : end); } } -{% endhighlight %} ---In order to define a scalar function, one has to extend the base class `ScalarFunction` in `org.apache.flink.table.functions` and implement (one or more) evaluation methods. The behavior of a scalar function is determined by the evaluation method. An evaluation method must be declared publicly and named `eval`. The parameter types and return type of the evaluation method also determine the parameter and return types of the scalar function. Evaluation methods can also be overloaded by implementing multiple methods named `eval`. Evaluation methods can also support variable arguments, such as `@varargs def eval(str: String*)`. +TableEnvironment env = TableEnvironment.create(...); -The following example shows how to define your own hash code function, register it in the TableEnvironment, and call it in a query. Note that you can configure your scalar function via a constructor before it is registered: +// 在 Table API 里不经注册直接“内联”调用函数 +env.from("MyTable").select(call(new SubstringFunction(true), $("myField"), 5, 12)); +// 注册函数 +env.createTemporarySystemFunction("SubstringFunction", new SubstringFunction(true)); + +{% endhighlight %} ++ +{% highlight scala %} -// must be defined in static/object context -class HashCode(factor: Int) extends ScalarFunction { - def eval(s: String): Int = { - s.hashCode() * factor +import org.apache.flink.table.api._ +import org.apache.flink.table.functions.ScalarFunction + +// 定义可参数化的函数逻辑 +class SubstringFunction(val endInclusive) extends ScalarFunction { + def eval(s: String, begin: Integer, end: Integer): String = { + s.substring(endInclusive ? end + 1 : end) } } -val tableEnv = BatchTableEnvironment.create(env) +val env = TableEnvironment.create(...) + +// 在 Table API 里不经注册直接“内联”调用函数 +env.from("MyTable").select(call(new SubstringFunction(true), $"myField", 5, 12)) -// use the function in Scala Table API -val hashCode = new HashCode(10) -myTable.select('string, hashCode('string)) +// 注册函数 +env.createTemporarySystemFunction("SubstringFunction", new SubstringFunction(true)) -// register and use the function in SQL -tableEnv.registerFunction("hashCode", new HashCode(10)) -tableEnv.sqlQuery("SELECT string, hashCode(string) FROM MyTable") {% endhighlight %} +-By default the result type of an evaluation method is determined by Flink's type extraction facilities. This is sufficient for basic types or simple POJOs but might be wrong for more complex, custom, or composite types. In these cases `TypeInformation` of the result type can be manually defined by overriding `ScalarFunction#getResultType()`. ++ +{% top %} -## Supported Syntax +## Execute a Query +A SELECT statement or a VALUES statement can be executed to collect the content to local through the `TableEnvironment.executeSql()` method. The method returns the result of the SELECT statement (or the VALUES statement) as a `TableResult`. Similar to a SELECT statement, a `Table` object can be executed using the `Table.execute()` method to collect the content of the query to the local client. +`TableResult.collect()` method returns a closeable row iterator. The select job will not be finished unless all result data has been collected. We should actively close the job to avoid resource leak through the `CloseableIterator#close()` method. +We can also print the select result to client console through the `TableResult.print()` method. The result data in `TableResult` can be accessed only once. Thus, `collect()` and `print()` must not be called after each other. + +For streaming job, `TableResult.collect()` method or `TableResult.print` method guarantee end-to-end exactly-once record delivery. This requires the checkpointing mechanism to be enabled. By default, checkpointing is disabled. To enable checkpointing, we can set checkpointing properties (see the checkpointing config for details) through `TableConfig`. +So a result record can be accessed by client only after its corresponding checkpoint completes. + +**Notes:** For streaming mode, only append-only query is supported now. + ++{% highlight java %} +import org.apache.flink.table.functions.ScalarFunction; + +// 有多个重载求值方法的函数 +public static class SumFunction extends ScalarFunction { + + public Integer eval(Integer a, Integer b) { + return a + b; + } + + public Integer eval(String a, String b) { + return Integer.valueOf(a) + Integer.valueOf(); + } + + public Integer eval(Double... d) { + double result = 0; + for (double value : d) + result += value; + return (int) result; + } +} + +{% endhighlight %} ++ +{% highlight scala %} -object TimestampModifier extends ScalarFunction { - def eval(t: Long): Long = { - t % 1000 +import org.apache.flink.table.functions.ScalarFunction +import scala.annotation.varargs + +// 有多个重载求值方法的函数 +class SumFunction extends ScalarFunction { + + def eval(a: Integer, b: Integer): Integer = { + a + b + } + + def eval(a: String, b: String): Integer = { + Integer.valueOf(a) + Integer.valueOf(b) } - override def getResultType(signature: Array[Class[_]]): TypeInformation[_] = { - Types.TIMESTAMP + @varargs // generate var-args like Java + def eval(d: Double*): Integer = { + d.sum.toInt } } + {% endhighlight %}--In order to define a Python scalar function, one can extend the base class `ScalarFunction` in `pyflink.table.udf` and implement an evaluation method. The behavior of a Python scalar function is determined by the evaluation method which is named `eval`. +-The following example shows how to define your own Python hash code function, register it in the TableEnvironment, and call it in a query. Note that you can configure your scalar function via a constructor before it is registered: +### 类型推导 -{% highlight python %} -class HashCode(ScalarFunction): - def __init__(self): - self.factor = 12 +Table（类似于 SQL 标准）是一种强类型的 API。因此，函数的参数和返回类型都必须映射到[数据类型]({%link dev/table/types.zh.md %})。 + +从逻辑角度看，Planner 需要知道数据类型、精度和小数位数；从 JVM 角度来看，Planner 在调用自定义函数时需要知道如何将内部数据结构表示为 JVM 对象。 + +术语 _类型推导_ 概括了意在验证输入值、派生出参数/返回值数据类型的逻辑。 + +Flink 自定义函数实现了自动的类型推导提取，通过反射从函数的类及其求值方法中派生数据类型。如果这种隐式的反射提取方法不成功，则可以通过使用 `@DataTypeHint` 和 `@FunctionHint` 注解相关参数、类或方法来支持提取过程，下面展示了有关如何注解函数的例子。 - def eval(self, s): - return hash(s) * self.factor +如果需要更高级的类型推导逻辑，实现者可以在每个自定义函数中显式重写 `getTypeInference()` 方法。但是，建议使用注解方式，因为它可使自定义类型推导逻辑保持在受影响位置附近，而在其他位置则保持默认状态。 -table_env = BatchTableEnvironment.create(env) -# register the Python function -table_env.register_function("hash_code", udf(HashCode(), DataTypes.BIGINT(), DataTypes.BIGINT())) +#### 自动类型推导 -# use the function in Python Table API -my_table.select("string, bigint, string.hash_code(), hash_code(string)") +自动类型推导会检查函数的类和求值方法，派生出函数参数和结果的数据类型， `@DataTypeHint` 和 `@FunctionHint` 注解支持自动类型推导。 + +有关可以隐式映射到数据类型的类的完整列表，请参阅[数据类型]({%link dev/table/types.zh.md %}#数据类型注解)。 + +**`@DataTypeHint`** + +在许多情况下，需要支持以 _内联_ 方式自动提取出函数参数、返回值的类型。 + +以下例子展示了如何使用 `@DataTypeHint`，详情可参考该注解类的文档。 + ++ +-{% top %} +**`@FunctionHint`** -Table Functions ---------------- +有时我们希望一种求值方法可以同时处理多种数据类型，有时又要求对重载的多个求值方法仅声明一次通用的结果类型。 + +`@FunctionHint` 注解可以提供从入参数据类型到结果数据类型的映射，它可以在整个函数类或求值方法上注解输入、累加器和结果的数据类型。可以在类顶部声明一个或多个注解，也可以为类的所有求值方法分别声明一个或多个注解。所有的 hint 参数都是可选的，如果未定义参数，则使用默认的基于反射的类型提取。在函数类顶部定义的 hint 参数被所有求值方法继承。 -Similar to a user-defined scalar function, a user-defined table function takes zero, one, or multiple scalar values as input parameters. However in contrast to a scalar function, it can return an arbitrary number of rows as output instead of a single value. The returned rows may consist of one or more columns. +以下例子展示了如何使用 `@FunctionHint`，详情可参考该注解类的文档。+{% highlight java %} +import org.apache.flink.table.annotation.DataTypeHint; +import org.apache.flink.table.annotation.InputGroup; +import org.apache.flink.table.functions.ScalarFunction; +import org.apache.flink.types.Row; + +// 有多个重载求值方法的函数 +public static class OverloadedFunction extends ScalarFunction { + + // no hint required + public Long eval(long a, long b) { + return a + b; + } + + // 定义 decimal 的精度和小数位 + public @DataTypeHint("DECIMAL(12, 3)") BigDecimal eval(double a, double b) { + return BigDecimal.valueOf(a + b); + } + + // 定义嵌套数据类型 + @DataTypeHint("ROW-There are many ways to define a Python scalar function besides extending the base class `ScalarFunction`. -Please refer to the [Python Scalar Function]({{ site.baseurl }}/zh/dev/table/python/python_udfs.html#scalar-functions) documentation for more details. +") + public Row eval(int i) { + return Row.of(String.valueOf(i), Instant.ofEpochSecond(i)); + } + + // 允许任意类型的符入，并输出序列化定制后的值 + @DataTypeHint(value = "RAW", bridgedTo = ByteBuffer.class) + public ByteBuffer eval(@DataTypeHint(inputGroup = InputGroup.ANY) Object o) { + return MyUtils.serializeToByteBuffer(o); + } +} -# use the function in SQL API -table_env.sql_query("SELECT string, bigint, hash_code(bigint) FROM MyTable") {% endhighlight %} ++{% highlight scala %} +import org.apache.flink.table.annotation.DataTypeHint +import org.apache.flink.table.annotation.InputGroup +import org.apache.flink.table.functions.ScalarFunction +import org.apache.flink.types.Row +import scala.annotation.varargs + +// function with overloaded evaluation methods +class OverloadedFunction extends ScalarFunction { + + // no hint required + def eval(a: Long, b: Long): Long = { + a + b + } + + // 定义 decimal 的精度和小数位 + @DataTypeHint("DECIMAL(12, 3)") + def eval(double a, double b): BigDecimal = { + java.lang.BigDecimal.valueOf(a + b) + } + + // 定义嵌套数据类型 + @DataTypeHint("ROW+") + def eval(Int i): Row = { + Row.of(java.lang.String.valueOf(i), java.time.Instant.ofEpochSecond(i)) + } + + // 允许任意类型的符入，并输出定制序列化后的值 + @DataTypeHint(value = "RAW", bridgedTo = classOf[java.nio.ByteBuffer]) + def eval(@DataTypeHint(inputGroup = InputGroup.ANY) Object o): java.nio.ByteBuffer = { + MyUtils.serializeToByteBuffer(o) + } +} + +{% endhighlight %}--In order to define a table function one has to extend the base class `TableFunction` in `org.apache.flink.table.functions` and implement (one or more) evaluation methods. The behavior of a table function is determined by its evaluation methods. An evaluation method must be declared `public` and named `eval`. The `TableFunction` can be overloaded by implementing multiple methods named `eval`. The parameter types of the evaluation methods determine all valid parameters of the table function. Evaluation methods can also support variable arguments, such as `eval(String... strs)`. The type of the returned table is determined by the generic type of `TableFunction`. Evaluation methods emit output rows using the protected `collect(T)` method. ++ +#### 定制类型推导 + +在大多数情况下，`@DataTypeHint` 和 `@FunctionHint` 足以构建自定义函数，然而通过重写 `getTypeInference()` 定制自动类型推导逻辑，实现者可以创建任意像系统内置函数那样有用的函数。 + +以下用 Java 实现的例子展示了定制类型推导的潜力，它根据字符串参数来确定函数的结果类型。该函数带有两个字符串参数：第一个参数表示要分析的字符串，第二个参数表示目标类型。 + ++{% highlight java %} +import org.apache.flink.table.annotation.DataTypeHint; +import org.apache.flink.table.annotation.FunctionHint; +import org.apache.flink.table.functions.TableFunction; +import org.apache.flink.types.Row; + +// 为函数类的所有求值方法指定同一个输出类型 +@FunctionHint(output = @DataTypeHint("ROW+ +")) +public static class OverloadedFunction extends TableFunction{ -In the Table API, a table function is used with `.joinLateral` or `.leftOuterJoinLateral`. The `joinLateral` operator (cross) joins each row from the outer table (table on the left of the operator) with all rows produced by the table-valued function (which is on the right side of the operator). The `leftOuterJoinLateral` operator joins each row from the outer table (table on the left of the operator) with all rows produced by the table-valued function (which is on the right side of the operator) and preserves outer rows for which the table function returns an empty table. In SQL use `LATERAL TABLE(
)` with CROSS JOIN and LEFT JOIN with an ON TRUE join condition (see examples below). + public void eval(int a, int b) { + collect(Row.of("Sum", a + b)); + } -The following example shows how to define table-valued function, register it in the TableEnvironment, and call it in a query. Note that you can configure your table function via a constructor before it is registered: + // overloading of arguments is still possible + public void eval() { + collect(Row.of("Empty args", -1)); + } +} +// 解耦类型推导与求值方法，类型推导完全取决于 FunctionHint +@FunctionHint( + input = [@DataTypeHint("INT"), @DataTypeHint("INT")], + output = @DataTypeHint("INT") +) +@FunctionHint( + input = [@DataTypeHint("LONG"), @DataTypeHint("LONG")], + output = @DataTypeHint("LONG") +) +@FunctionHint( + input = [], + output = @DataTypeHint("BOOLEAN") +) +public static class OverloadedFunction extends TableFunction +{% highlight scala %} + +import org.apache.flink.table.annotation.DataTypeHint +import org.apache.flink.table.annotation.FunctionHint +import org.apache.flink.table.functions.TableFunction +import org.apache.flink.types.Row + +// 为函数类的所有求值方法指定同一个输出类型 +@FunctionHint(output = @DataTypeHint("ROW+ +")) +class OverloadedFunction extends TableFunction[Row] { + + def eval(a: Int, b: Int): Unit = { + collect(Row.of("Sum", Int.box(a + b))) + } + + // overloading of arguments is still possible + def eval(): Unit = { + collect(Row.of("Empty args", Int.box(-1))) + } +} + +// 解耦类型推导与求值方法，类型推导完全取决于 @FunctionHint +@FunctionHint( + input = Array(@DataTypeHint("INT"), @DataTypeHint("INT")), + output = @DataTypeHint("INT") +) +@FunctionHint( + input = Array(@DataTypeHint("LONG"), @DataTypeHint("LONG")), + output = @DataTypeHint("LONG") +) +@FunctionHint( + input = Array(), + output = @DataTypeHint("BOOLEAN") +) +class OverloadedFunction extends TableFunction[AnyRef] { + + // an implementer just needs to make sure that a method exists + // that can be called by the JVM + @varargs + def eval(o: AnyRef*) = { + if (o.length == 0) { + collect(Boolean.box(false)) + } + collect(o(0)) + } +} + +{% endhighlight %} ++ ++ +### 运行时集成 +------------------- + +有时候自定义函数需要获取一些全局信息，或者在真正被调用之前做一些配置（setup）/清理（clean-up）的工作。自定义函数也提供了 `open()` 和 `close()` 方法，你可以重写这两个方法做到类似于 DataStream API 中 `RichFunction` 的功能。 + +open() 方法在求值方法被调用之前先调用。close() 方法在求值方法调用完之后被调用。 + +open() 方法提供了一个 FunctionContext，它包含了一些自定义函数被执行时的上下文信息，比如 metric group、分布式文件缓存，或者是全局的作业参数等。 + +下面的信息可以通过调用 `FunctionContext` 的对应的方法来获得： + +| 方法 | 描述 | +| :------------------------------------ | :----------------------------------------------------- | +| `getMetricGroup()` | 执行该函数的 subtask 的 Metric Group。 | +| `getCachedFile(name)` | 分布式文件缓存的本地临时文件副本。| +| `getJobParameter(name, defaultValue)` | 跟对应的 key 关联的全局参数值。 | + +下面的例子展示了如何在一个标量函数中通过 FunctionContext 来获取一个全局的任务参数： + +{% highlight java %} -// The generic type "Tuple2+ ++ +@@ -123,7 +121,6 @@ table = table_env.from_elements(..., ['user', 'product', 'amount']) result = table_env \ .sql_query("SELECT SUM(amount) FROM %s WHERE product LIKE '%%Rubber%%'" % table) -# SQL update with a registered table # create and register a TableSink t_env.connect(FileSystem().path("/path/to/file"))) .with_format(Csv() @@ -134,16 +131,107 @@ t_env.connect(FileSystem().path("/path/to/file"))) .field("amount", DataTypes.BIGINT())) .create_temporary_table("RubberOrders") -# run a SQL update query on the Table and emit the result to the TableSink +# run an INSERT SQL on the Table and emit the result to the TableSink table_env \ - .sql_update("INSERT INTO RubberOrders SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'") + .execute_sql("INSERT INTO RubberOrders SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'") {% endhighlight %}+{% highlight java %} +import org.apache.flink.table.api.*; +import org.apache.flink.table.functions.FunctionContext; +import org.apache.flink.table.functions.ScalarFunction; + +public static class HashCodeFunction extends ScalarFunction { + + private int factor = 0; + + @Override + public void open(FunctionContext context) throws Exception { + // 获取参数 "hashcode_factor" + // 如果不存在，则使用默认值 "12" + factor = Integer.parseInt(context.getJobParameter("hashcode_factor", "12")); + } + + public int eval(String s) { + return s.hashCode() * factor; } } -BatchTableEnvironment tableEnv = BatchTableEnvironment.create(env); -Table myTable = ... // table schema: [a: String] +TableEnvironment env = TableEnvironment.create(...); + +// 设置任务参数 +env.getConfig().addJobParameter("hashcode_factor", "31"); -// Register the function. -tableEnv.registerFunction("split", new Split("#")); +// 注册函数 +env.createTemporarySystemFunction("hashCode", HashCodeFunction.class); -// Use the table function in the Java Table API. "as" specifies the field names of the table. -myTable.joinLateral("split(a) as (word, length)") - .select("a, word, length"); -myTable.leftOuterJoinLateral("split(a) as (word, length)") - .select("a, word, length"); +// 调用函数 +env.sqlQuery("SELECT myField, hashCode(myField) FROM MyTable"); -// Use the table function in SQL with LATERAL and TABLE keywords. -// CROSS JOIN a table function (equivalent to "join" in Table API). -tableEnv.sqlQuery("SELECT a, word, length FROM MyTable, LATERAL TABLE(split(a)) as T(word, length)"); -// LEFT JOIN a table function (equivalent to "leftOuterJoin" in Table API). -tableEnv.sqlQuery("SELECT a, word, length FROM MyTable LEFT JOIN LATERAL TABLE(split(a)) as T(word, length) ON TRUE"); {% endhighlight %}+{% highlight scala %} +import org.apache.flink.table.api._ +import org.apache.flink.table.functions.FunctionContext +import org.apache.flink.table.functions.ScalarFunction -In order to define a table function one has to extend the base class `TableFunction` in `org.apache.flink.table.functions` and implement (one or more) evaluation methods. The behavior of a table function is determined by its evaluation methods. An evaluation method must be declared `public` and named `eval`. The `TableFunction` can be overloaded by implementing multiple methods named `eval`. The parameter types of the evaluation methods determine all valid parameters of the table function. Evaluation methods can also support variable arguments, such as `eval(String... strs)`. The type of the returned table is determined by the generic type of `TableFunction`. Evaluation methods emit output rows using the protected `collect(T)` method. +class HashCodeFunction extends ScalarFunction { -In the Table API, a table function is used with `.joinLateral` or `.leftOuterJoinLateral`. The `joinLateral` operator (cross) joins each row from the outer table (table on the left of the operator) with all rows produced by the table-valued function (which is on the right side of the operator). The `leftOuterJoinLateral` operator joins each row from the outer table (table on the left of the operator) with all rows produced by the table-valued function (which is on the right side of the operator) and preserves outer rows for which the table function returns an empty table. In SQL use `LATERAL TABLE(-)` with CROSS JOIN and LEFT JOIN with an ON TRUE join condition (see examples below). + private var factor: Int = 0 -The following example shows how to define table-valued function, register it in the TableEnvironment, and call it in a query. Note that you can configure your table function via a constructor before it is registered: + override def open(context: FunctionContext): Unit = { + // 获取参数 "hashcode_factor" + // 如果不存在，则使用默认值 "12" + factor = context.getJobParameter("hashcode_factor", "12").toInt + } -{% highlight scala %} -// The generic type "(String, Int)" determines the schema of the returned table as (String, Integer). -class Split(separator: String) extends TableFunction[(String, Int)] { - def eval(str: String): Unit = { - // use collect(...) to emit a row. - str.split(separator).foreach(x => collect((x, x.length))) + def eval(s: String): Int = { + s.hashCode * factor } } -val tableEnv = BatchTableEnvironment.create(env) -val myTable = ... // table schema: [a: String] +val env = TableEnvironment.create(...) -// Use the table function in the Scala Table API (Note: No registration required in Scala Table API). -val split = new Split("#") -// "as" specifies the field names of the generated table. -myTable.joinLateral(split('a) as ('word, 'length)).select('a, 'word, 'length) -myTable.leftOuterJoinLateral(split('a) as ('word, 'length)).select('a, 'word, 'length) +// 设置任务参数 +env.getConfig.addJobParameter("hashcode_factor", "31") -// Register the table function to use it in SQL queries. -tableEnv.registerFunction("split", new Split("#")) +// 注册函数 +env.createTemporarySystemFunction("hashCode", classOf[HashCodeFunction]) + +// 调用函数 +env.sqlQuery("SELECT myField, hashCode(myField) FROM MyTable") -// Use the table function in SQL with LATERAL and TABLE keywords. -// CROSS JOIN a table function (equivalent to "join" in Table API) -tableEnv.sqlQuery("SELECT a, word, length FROM MyTable, LATERAL TABLE(split(a)) as T(word, length)") -// LEFT JOIN a table function (equivalent to "leftOuterJoin" in Table API) -tableEnv.sqlQuery("SELECT a, word, length FROM MyTable LEFT JOIN LATERAL TABLE(split(a)) as T(word, length) ON TRUE") {% endhighlight %} -**IMPORTANT:** Do not implement TableFunction as a Scala object. Scala object is a singleton and will cause concurrency issues. -In order to define a Python table function, one can extend the base class `TableFunction` in `pyflink.table.udtf` and Implement an evaluation method. The behavior of a Python table function is determined by the evaluation method which is named eval. +-In the Python Table API, a Python table function is used with `.join_lateral` or `.left_outer_join_lateral`. The `join_lateral` operator (cross) joins each row from the outer table (table on the left of the operator) with all rows produced by the table-valued function (which is on the right side of the operator). The `left_outer_join_lateral` operator joins each row from the outer table (table on the left of the operator) with all rows produced by the table-valued function (which is on the right side of the operator) and preserves outer rows for which the table function returns an empty table. In SQL use `LATERAL TABLE()` with CROSS JOIN and LEFT JOIN with an ON TRUE join condition (see examples below). +{% top %} -The following example shows how to define a Python table function, registered it in the TableEnvironment, and call it in a query. Note that you can configure your table function via a constructor before it is registered: +标量函数 +---------------- -{% highlight python %} -class Split(TableFunction): - def eval(self, string): - for s in string.split(" "): - yield s, len(s) +自定义标量函数可以把 0 到多个标量值映射成 1 个标量值，[数据类型]({%link dev/table/types.zh.md %})里列出的任何数据类型都可作为求值方法的参数和返回值类型。 + +想要实现自定义标量函数，你需要扩展 `org.apache.flink.table.functions` 里面的 `ScalarFunction` 并且实现一个或者多个求值方法。标量函数的行为取决于你写的求值方法。求值方法必须是 `public` 的，而且名字必须是 `eval`。 -env = StreamExecutionEnvironment.get_execution_environment() -table_env = StreamTableEnvironment.create(env) -my_table = ... # type: Table, table schema: [a: String] +下面的例子展示了如何实现一个求哈希值的函数并在查询里调用它，详情可参考[开发指南](#开发指南)： -# register the Python Table Function -table_env.register_function("split", udtf(Split(), DataTypes.STRING(), [DataTypes.STRING(), DataTypes.INT()])) + + +-+{% highlight java %} +import org.apache.flink.table.annotation.InputGroup; +import org.apache.flink.table.api.*; +import org.apache.flink.table.functions.ScalarFunction; +import static org.apache.flink.table.api.Expressions.*; -# use the Python Table Function in Python Table API -my_table.join_lateral("split(a) as (word, length)") -my_table.left_outer_join_lateral("split(a) as (word, length)") +public static class HashFunction extends ScalarFunction { -# use the Python Table function in SQL API -table_env.sql_query("SELECT a, word, length FROM MyTable, LATERAL TABLE(split(a)) as T(word, length)") -table_env.sql_query("SELECT a, word, length FROM MyTable LEFT JOIN LATERAL TABLE(split(a)) as T(word, length) ON TRUE") + // 接受任意类型输入，返回 INT 型输出 + public int eval(@DataTypeHint(inputGroup = InputGroup.ANY) Object o) { + return o.hashCode(); + } +} + +TableEnvironment env = TableEnvironment.create(...); + +// 在 Table API 里不经注册直接“内联”调用函数 +env.from("MyTable").select(call(HashFunction.class, $("myField"))); + +// 注册函数 +env.createTemporarySystemFunction("HashFunction", HashFunction.class); + +// 在 Table API 里调用注册好的函数 +env.from("MyTable").select(call("HashFunction", $("myField"))); + +// 在 SQL 里调用注册好的函数 +env.sqlQuery("SELECT HashFunction(myField) FROM MyTable"); {% endhighlight %} ++ ++{% highlight scala %} +import org.apache.flink.table.annotation.InputGroup +import org.apache.flink.table.api._ +import org.apache.flink.table.functions.ScalarFunction + +class HashFunction extends ScalarFunction { + + // 接受任意类型输入，返回 INT 型输出 + def eval(@DataTypeHint(inputGroup = InputGroup.ANY) o: AnyRef): Int { + return o.hashCode(); + } +} + +val env = TableEnvironment.create(...) + +// 在 Table API 里不经注册直接“内联”调用函数 +env.from("MyTable").select(call(classOf[HashFunction], $"myField")) + +// 注册函数 +env.createTemporarySystemFunction("HashFunction", classOf[HashFunction]) -There are many ways to define a Python table function besides extending the base class `TableFunction`. -Please refer to the [Python Table Function]({{ site.baseurl }}/zh/dev/table/python/python_udfs.html#table-functions) documentation for more details. +// 在 Table API 里调用注册好的函数 +env.from("MyTable").select(call("HashFunction", $"myField")) +// 在 SQL 里调用注册好的函数 +env.sqlQuery("SELECT HashFunction(myField) FROM MyTable") + +{% endhighlight %}+-@@ -95,7 +94,6 @@ tableEnv.createTemporaryView("Orders", ds, $"user", $"product", $"amount") val result2 = tableEnv.sqlQuery( "SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'") -// SQL update with a registered table // create and register a TableSink val schema = new Schema() .field("product", DataTypes.STRING()) @@ -106,8 +104,8 @@ tableEnv.connect(new FileSystem("/path/to/file")) .withSchema(schema) .createTemporaryTable("RubberOrders") -// run a SQL update query on the Table and emit the result to the TableSink -tableEnv.sqlUpdate( +// run an INSERT SQL on the Table and emit the result to the TableSink +tableEnv.executeSql( "INSERT INTO RubberOrders SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'") {% endhighlight %}-Please note that POJO types do not have a deterministic field order. Therefore, you cannot rename the fields of POJO returned by a table function using `AS`. +如果你打算使用 Python 实现或调用标量函数，详情可参考 [Python 标量函数]({% link dev/table/python/python_udfs.zh.md %}#scalar-functions)。 -By default the result type of a `TableFunction` is determined by Flink’s automatic type extraction facilities. This works well for basic types and simple POJOs but might be wrong for more complex, custom, or composite types. In such a case, the type of the result can be manually specified by overriding `TableFunction#getResultType()` which returns its `TypeInformation`. +{% top %} -The following example shows an example of a `TableFunction` that returns a `Row` type which requires explicit type information. We define that the returned table type should be `RowTypeInfo(String, Integer)` by overriding `TableFunction#getResultType()`. +表值函数 +--------------- -{% highlight java %} -public class CustomTypeSplit extends TableFunction{ - public void eval(String str) { - for (String s : str.split(" ")) { - Row row = new Row(2); - row.setField(0, s); - row.setField(1, s.length()); - collect(row); - } - } +跟自定义标量函数一样，自定义表值函数的输入参数也可以是 0 到多个标量。但是跟标量函数只能返回一个值不同的是，它可以返回任意多行。返回的每一行可以包含 1 到多列，如果输出行只包含 1 列，会省略结构化信息并生成标量值，这个标量值在运行阶段会隐式地包装进行里。 - @Override - public TypeInformation
getResultType() { - return Types.ROW(Types.STRING(), Types.INT()); +要定义一个表值函数，你需要扩展 `org.apache.flink.table.functions` 下的 `TableFunction`，可以通过实现多个名为 `eval` 的方法对求值方法进行重载。像其他函数一样，输入和输出类型也可以通过反射自动提取出来。表值函数返回的表的类型取决于 `TableFunction` 类的泛型参数 `T`，不同于标量函数，表值函数的求值方法本身不包含返回类型，而是通过 `collect(T)` 方法来发送要输出的行。 + +在 Table API 中，表值函数是通过 `.joinLateral(...)` 或者 `.leftOuterJoinLateral(...)` 来使用的。`joinLateral` 算子会把外表（算子左侧的表）的每一行跟跟表值函数返回的所有行（位于算子右侧）进行 （cross）join。`leftOuterJoinLateral` 算子也是把外表（算子左侧的表）的每一行跟表值函数返回的所有行（位于算子右侧）进行（cross）join，并且如果表值函数返回 0 行也会保留外表的这一行。 + +在 SQL 里面用 `JOIN` 或者 以 `ON TRUE` 为条件的 `LEFT JOIN` 来配合 `LATERAL TABLE(
)` 的使用。 + +下面的例子展示了如何实现一个分隔函数并在查询里调用它，详情可参考[开发指南](#开发指南)： + + + +@@ -65,32 +68,36 @@ val settings = EnvironmentSettings.newInstance()... val tableEnv = TableEnvironment.create(settings) // 注册名为 “Orders” 的表 -tableEnv.sqlUpdate("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)"); +tableEnv.executeSql("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)") // 字符串数组： ["Orders"] -val tables = tableEnv.listTable() +val tables = tableEnv.listTables() +// or tableEnv.executeSql("SHOW TABLES").print() // 从 catalog 删除 “Orders” 表 -tableEnv.sqlUpdate("DROP TABLE Orders") +tableEnv.executeSql("DROP TABLE Orders") // 空字符串数组 -val tables = tableEnv.listTable() +val tables = tableEnv.listTables() +// or tableEnv.executeSql("SHOW TABLES").print() {% endhighlight %}+{% highlight java %} +import org.apache.flink.table.annotation.DataTypeHint; +import org.apache.flink.table.annotation.FunctionHint; +import org.apache.flink.table.api.*; +import org.apache.flink.table.functions.TableFunction; +import org.apache.flink.types.Row; +import static org.apache.flink.table.api.Expressions.*; + +@FunctionHint(output = @DataTypeHint("ROW-")) +public static class SplitFunction extends TableFunction { + + public void eval(String str) { + for (String s : str.split(" ")) { + // use collect(...) to emit a row + collect(Row.of(s, s.length())); } + } } + +TableEnvironment env = TableEnvironment.create(...); + +// 在 Table API 里不经注册直接“内联”调用函数 +env + .from("MyTable") + .joinLateral(call(SplitFunction.class, $("myField"))) + .select($("myField"), $("word"), $("length")); +env + .from("MyTable") + .leftOuterJoinLateral(call(SplitFunction.class, $("myField"))) + .select($("myField"), $("word"), $("length")); + +// 在 Table API 里重命名函数字段 +env + .from("MyTable") + .leftOuterJoinLateral(call(SplitFunction.class, $("myField")).as("newWord", "newLength")) + .select($("myField"), $("newWord"), $("newLength")); + +// 注册函数 +env.createTemporarySystemFunction("SplitFunction", SplitFunction.class); + +// 在 Table API 里调用注册好的函数 +env + .from("MyTable") + .joinLateral(call("SplitFunction", $("myField"))) + .select($("myField"), $("word"), $("length")); +env + .from("MyTable") + .leftOuterJoinLateral(call("SplitFunction", $("myField"))) + .select($("myField"), $("word"), $("length")); + +// 在 SQL 里调用注册好的函数 +env.sqlQuery( + "SELECT myField, word, length " + + "FROM MyTable, LATERAL TABLE(SplitFunction(myField))"); +env.sqlQuery( + "SELECT myField, word, length " + + "FROM MyTable " + + "LEFT JOIN LATERAL TABLE(SplitFunction(myField)) ON TRUE"); + +// 在 SQL 里重命名函数字段 +env.sqlQuery( + "SELECT myField, newWord, newLength " + + "FROM MyTable " + + "LEFT JOIN LATERAL TABLE(SplitFunction(myField)) AS T(newWord, newLength) ON TRUE"); + {% endhighlight %}
-Please note that POJO types do not have a deterministic field order. Therefore, you cannot rename the fields of POJO returned by a table function using `AS`. - -By default the result type of a `TableFunction` is determined by Flink’s automatic type extraction facilities. This works well for basic types and simple POJOs but might be wrong for more complex, custom, or composite types. In such a case, the type of the result can be manually specified by overriding `TableFunction#getResultType()` which returns its `TypeInformation`. +-{% top %} +如果你打算使用 Scala，不要把表值函数声明为 Scala `object`，Scala `object` 是单例对象，将导致并发问题。 +如果你打算使用 Python 实现或调用表值函数，详情可参考 [Python 表值函数]({% link dev/table/python/python_udfs.zh.md %}#table-functions)。 -Aggregation Functions +{% top %} + +聚合函数 --------------------- -User-Defined Aggregate Functions (UDAGGs) aggregate a table (one or more rows with one or more attributes) to a scalar value. +自定义聚合函数（UDAGG）是把一个表（一行或者多行，每行可以有一列或者多列）聚合成一个标量值。+{% highlight scala %} +import org.apache.flink.table.annotation.DataTypeHint +import org.apache.flink.table.annotation.FunctionHint +import org.apache.flink.table.api._ +import org.apache.flink.table.functions.TableFunction +import org.apache.flink.types.Row -The following example shows an example of a `TableFunction` that returns a `Row` type which requires explicit type information. We define that the returned table type should be `RowTypeInfo(String, Integer)` by overriding `TableFunction#getResultType()`. +@FunctionHint(output = @DataTypeHint("ROW+")) +class SplitFunction extends TableFunction[Row] { -{% highlight scala %} -class CustomTypeSplit extends TableFunction[Row] { def eval(str: String): Unit = { - str.split(" ").foreach({ s => - val row = new Row(2) - row.setField(0, s) - row.setField(1, s.length) - collect(row) - }) - } - - override def getResultType: TypeInformation[Row] = { - Types.ROW(Types.STRING, Types.INT) + // use collect(...) to emit a row + str.split(" ").foreach(s => collect(Row.of(s, s.length))) } } + +val env = TableEnvironment.create(...) + +// 在 Table API 里不经注册直接“内联”调用函数 +env + .from("MyTable") + .joinLateral(call(classOf[SplitFunction], $"myField") + .select($"myField", $"word", $"length") +env + .from("MyTable") + .leftOuterJoinLateral(call(classOf[SplitFunction], $"myField")) + .select($"myField", $"word", $"length") + +// 在 Table API 里重命名函数字段 +env + .from("MyTable") + .leftOuterJoinLateral(call(classOf[SplitFunction], $"myField").as("newWord", "newLength")) + .select($"myField", $"newWord", $"newLength") + +// 注册函数 +env.createTemporarySystemFunction("SplitFunction", classOf[SplitFunction]) + +// 在 Table API 里调用注册好的函数 +env + .from("MyTable") + .joinLateral(call("SplitFunction", $"myField")) + .select($"myField", $"word", $"length") +env + .from("MyTable") + .leftOuterJoinLateral(call("SplitFunction", $"myField")) + .select($"myField", $"word", $"length") + +// 在 SQL 里调用注册好的函数 +env.sqlQuery( + "SELECT myField, word, length " + + "FROM MyTable, LATERAL TABLE(SplitFunction(myField))"); +env.sqlQuery( + "SELECT myField, word, length " + + "FROM MyTable " + + "LEFT JOIN LATERAL TABLE(SplitFunction(myField)) ON TRUE") + +// 在 SQL 里重命名函数字段 +env.sqlQuery( + "SELECT myField, newWord, newLength " + + "FROM MyTable " + + "LEFT JOIN LATERAL TABLE(SplitFunction(myField)) AS T(newWord, newLength) ON TRUE") + {% endhighlight %} 
@@ -418,7 +941,7 @@ public abstract class UserDefinedAggregateFunction-The following example shows how to +下面的例子展示了如何： -- define an `AggregateFunction` that calculates the weighted average on a given column, -- register the function in the `TableEnvironment`, and -- use the function in a query. +- 定义一个聚合函数来计算某一列的加权平均， +- 在 `TableEnvironment` 中注册函数， +- 在查询中使用函数。 -To calculate an weighted average value, the accumulator needs to store the weighted sum and count of all the data that has been accumulated. In our example we define a class `WeightedAvgAccum` to be the accumulator. Accumulators are automatically backup-ed by Flink's checkpointing mechanism and restored in case of a failure to ensure exactly-once semantics. +为了计算加权平均值，accumulator 需要存储加权总和以及数据的条数。在我们的例子里，我们定义了一个类 `WeightedAvgAccum` 来作为 accumulator。Flink 的 checkpoint 机制会自动保存 accumulator，在失败时进行恢复，以此来保证精确一次的语义。 -The `accumulate()` method of our `WeightedAvg` `AggregateFunction` has three inputs. The first one is the `WeightedAvgAccum` accumulator, the other two are user-defined inputs: input value `ivalue` and weight of the input `iweight`. Although the `retract()`, `merge()`, and `resetAccumulator()` methods are not mandatory for most aggregation types, we provide them below as examples. Please note that we used Java primitive types and defined `getResultType()` and `getAccumulatorType()` methods in the Scala example because Flink type extraction does not work very well for Scala types. +我们的 `WeightedAvg`（聚合函数）的 `accumulate` 方法有三个输入参数。第一个是 `WeightedAvgAccum` accumulator，另外两个是用户自定义的输入：输入的值 `ivalue` 和 输入的权重 `iweight`。尽管 `retract()`、`merge()`、`resetAccumulator()` 这几个方法在大多数聚合类型中都不是必须实现的，我们也在样例中提供了他们的实现。请注意我们在 Scala 样例中也是用的是 Java 的基础类型，并且定义了 `getResultType()` 和 `getAccumulatorType()`，因为 Flink 的类型推导对于 Scala 的类型推导做的不是很好。the type of the aggregation result * @param the type of the aggregation accumulator. The accumulator is used to keep the @@ -527,7 +1050,7 @@ abstract class UserDefinedAggregateFunction[T, ACC] extends UserDefinedFunction } /** - * Base class for aggregation functions. + * Base class for aggregation functions. * * @tparam T the type of the aggregation result * @tparam ACC the type of the aggregation accumulator. The accumulator is used to keep the @@ -570,7 +1093,7 @@ abstract class AggregateFunction[T, ACC] extends UserDefinedAggregateFunction[T, * merged. */ def merge(accumulator: ACC, its: java.lang.Iterable[ACC]): Unit // OPTIONAL - + /** * Called every time when an aggregation result should be materialized. * The returned value could be either an early and incomplete result @@ -603,15 +1126,15 @@ abstract class AggregateFunction[T, ACC] extends UserDefinedAggregateFunction[T, @@ -65,32 +68,36 @@ val settings = EnvironmentSettings.newInstance()... val tableEnv = TableEnvironment.create(settings) // register a table named "Orders" -tableEnv.sqlUpdate("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)"); +tableEnv.executeSql("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)") // a string array: ["Orders"] -val tables = tableEnv.listTable() +val tables = tableEnv.listTables() +// or tableEnv.executeSql("SHOW TABLES").print() // drop "Orders" table from catalog -tableEnv.sqlUpdate("DROP TABLE Orders") +tableEnv.executeSql("DROP TABLE Orders") // an empty string array -val tables = tableEnv.listTable() +val tables = tableEnv.listTables() +// or tableEnv.executeSql("SHOW TABLES").print() {% endhighlight %}@@ -652,7 +1175,7 @@ public static class WeightedAvg extends AggregateFunction- {% top %} -Table Aggregation Functions +表值聚合函数 --------------------- -User-Defined Table Aggregate Functions (UDTAGGs) aggregate a table (one or more rows with one or more attributes) to a result table with multi rows and columns. +自定义表值聚合函数（UDTAGG）可以把一个表（一行或者多行，每行有一列或者多列）聚合成另一张表，结果中可以有多行多列。iter = it.iterator(); while (iter.hasNext()) { @@ -661,18 +1184,18 @@ public static class WeightedAvg extends AggregateFunction iter = it.iterator(); while (iter.hasNext()) { @@ -804,7 +1328,7 @@ public static class WeightedAvg extends AggregateFunction 
@@ -905,7 +1426,7 @@ public abstract class UserDefinedAggregateFunction-The following example shows how to +下面的例子展示了如何 -- define a `TableAggregateFunction` that calculates the top 2 values on a given column, -- register the function in the `TableEnvironment`, and -- use the function in a Table API query(TableAggregateFunction is only supported by Table API). +- 定义一个 `TableAggregateFunction` 来计算给定列的最大的 2 个值， +- 在 `TableEnvironment` 中注册函数， +- 在 Table API 查询中使用函数（当前只在 Table API 中支持 TableAggregateFunction）。 -To calculate the top 2 values, the accumulator needs to store the biggest 2 values of all the data that has been accumulated. In our example we define a class `Top2Accum` to be the accumulator. Accumulators are automatically backup-ed by Flink's checkpointing mechanism and restored in case of a failure to ensure exactly-once semantics. +为了计算最大的 2 个值，accumulator 需要保存当前看到的最大的 2 个值。在我们的例子中，我们定义了类 `Top2Accum` 来作为 accumulator。Flink 的 checkpoint 机制会自动保存 accumulator，并且在失败时进行恢复，来保证精确一次的语义。 -The `accumulate()` method of our `Top2` `TableAggregateFunction` has two inputs. The first one is the `Top2Accum` accumulator, the other one is the user-defined input: input value `v`. Although the `merge()` method is not mandatory for most table aggregation types, we provide it below as examples. Please note that we used Java primitive types and defined `getResultType()` and `getAccumulatorType()` methods in the Scala example because Flink type extraction does not work very well for Scala types. +我们的 `Top2` 表值聚合函数（`TableAggregateFunction`）的 `accumulate()` 方法有两个输入，第一个是 `Top2Accum` accumulator，另一个是用户定义的输入：输入的值 `v`。尽管 `merge()` 方法在大多数聚合类型中不是必须的，我们也在样例中提供了它的实现。请注意，我们在 Scala 样例中也使用的是 Java 的基础类型，并且定义了 `getResultType()` 和 `getAccumulatorType()` 方法，因为 Flink 的类型推导对于 Scala 的类型推导支持的不是很好。the type of the aggregation result * @param the type of the aggregation accumulator. The accumulator is used to keep the @@ -958,7 +1479,7 @@ public abstract class TableAggregateFunction out); // OPTIONAL - + /** * Called every time when an aggregation result should be materialized. The returned value * could be either an early and incomplete result (periodically emitted as data arrive) or @@ -978,7 +1499,7 @@ public abstract class TableAggregateFunction out); // OPTIONAL - + /** * Collects a record and forwards it. The collector can output retract messages with the retract * method. Note: only use it in {@code emitRetractValueIncrementally}. @@ -1032,7 +1553,7 @@ abstract class UserDefinedAggregateFunction[T, ACC] extends UserDefinedFunction } /** - * Base class for table aggregation functions. + * Base class for table aggregation functions. * * @tparam T the type of the aggregation result * @tparam ACC the type of the aggregation accumulator. The accumulator is used to keep the @@ -1075,7 +1596,7 @@ abstract class TableAggregateFunction[T, ACC] extends UserDefinedAggregateFuncti * merged. */ def merge(accumulator: ACC, its: java.lang.Iterable[ACC]): Unit // OPTIONAL - + /** * Called every time when an aggregation result should be materialized. The returned value * could be either an early and incomplete result (periodically emitted as data arrive) or @@ -1106,13 +1627,13 @@ abstract class TableAggregateFunction[T, ACC] extends UserDefinedAggregateFuncti * records. */ def emitUpdateWithRetract(accumulator: ACC, out: RetractableCollector[T]): Unit // OPTIONAL - + /** * Collects a record and forwards it. The collector can output retract messages with the retract * method. Note: only use it in `emitRetractValueIncrementally`. */ trait RetractableCollector[T] extends Collector[T] { - + /** * Retract a record. * @@ -1126,15 +1647,15 @@ abstract class TableAggregateFunction[T, ACC] extends UserDefinedAggregateFuncti @@ -66,40 +67,40 @@ tableEnv.sqlUpdate( val settings = EnvironmentSettings.newInstance()... val tableEnv = TableEnvironment.create(settings) -// 对已经已经注册的表进行 SQL 查询 +// 对已注册的表进行 SQL 查询 // 注册名为 “Orders” 的表 -tableEnv.sqlUpdate("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)"); +tableEnv.executeSql("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)"); // 在表上执行 SQL 查询，并把得到的结果作为一个新的表 val result = tableEnv.sqlQuery( "SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'"); -// SQL 对已注册的表进行 update 操作 +// 对已注册的表进行 INSERT 操作 // 注册 TableSink -tableEnv.sqlUpdate("CREATE TABLE RubberOrders(product STRING, amount INT) WITH ('connector.path'='/path/to/file' ...)"); -// 在表上执行 SQL 更新查询并向 TableSink 发出结果 -tableEnv.sqlUpdate( +tableEnv.executeSql("CREATE TABLE RubberOrders(product STRING, amount INT) WITH ('connector.path'='/path/to/file' ...)"); +// 在表上执行 INSERT 语句并向 TableSink 发出结果 +tableEnv.executeSql( "INSERT INTO RubberOrders SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'") {% endhighlight %}@@ -1188,14 +1709,14 @@ public static class Top2 extends TableAggregateFunction-The following example shows how to use `emitUpdateWithRetract` method to emit only updates. To emit only updates, in our example, the accumulator keeps both old and new top 2 values. Note: if the N of topN is big, it may inefficient to keep both old and new values. One way to solve this case is to store the input record into the accumulator in `accumulate` method and then perform calculation in `emitUpdateWithRetract`. +下面的例子展示了如何使用 `emitUpdateWithRetract` 方法来只发送更新的数据。为了只发送更新的结果，accumulator 保存了上一次的最大的2个值，也保存了当前最大的2个值。注意：如果 TopN 中的 n 非常大，这种既保存上次的结果，也保存当前的结果的方式不太高效。一种解决这种问题的方式是把输入数据直接存储到 `accumulator` 中，然后在调用 `emitUpdateWithRetract` 方法时再进行计算。- -{% top %} - -Best Practices for Implementing UDFs ------------------------------------- - -The Table API and SQL code generation internally tries to work with primitive values as much as possible. A user-defined function can introduce much overhead through object creation, casting, and (un)boxing. Therefore, it is highly recommended to declare parameters and result types as primitive types instead of their boxed classes. `Types.DATE` and `Types.TIME` can also be represented as `int`. `Types.TIMESTAMP` can be represented as `long`. - -We recommended that user-defined functions should be written by Java instead of Scala as Scala types pose a challenge for Flink's type extractor. - {% top %} - -Integrating UDFs with the Runtime ---------------------------------- - -Sometimes it might be necessary for a user-defined function to get global runtime information or do some setup/clean-up work before the actual work. User-defined functions provide `open()` and `close()` methods that can be overridden and provide similar functionality as the methods in `RichFunction` of DataSet or DataStream API. - -The `open()` method is called once before the evaluation method. The `close()` method after the last call to the evaluation method. - -The `open()` method provides a `FunctionContext` that contains information about the context in which user-defined functions are executed, such as the metric group, the distributed cache files, or the global job parameters. - -The following information can be obtained by calling the corresponding methods of `FunctionContext`: - -| Method | Description | -| :------------------------------------ | :----------------------------------------------------- | -| `getMetricGroup()` | Metric group for this parallel subtask. | -| `getCachedFile(name)` | Local temporary file copy of a distributed cache file. | -| `getJobParameter(name, defaultValue)` | Global job parameter value associated with given key. | - -The following example snippet shows how to use `FunctionContext` in a scalar function for accessing a global job parameter: - -@@ -1332,14 +1853,14 @@ public static class Top2 extends TableAggregateFunction-- -{% top %} - diff --git a/docs/dev/table/hive/hive_catalog.md b/docs/dev/table/hive/hive_catalog.md index d907703fe24b7e..7b59d9aff73d40 100644 --- a/docs/dev/table/hive/hive_catalog.md +++ b/docs/dev/table/hive/hive_catalog.md @@ -387,3 +387,7 @@ Something to note about the type mapping: * Hive's `TIMESTAMP` always has precision 9 and doesn't support other precisions. Hive UDFs, on the other hand, can process `TIMESTAMP` values with a precision <= 9. * Hive doesn't support Flink's `TIMESTAMP_WITH_TIME_ZONE`, `TIMESTAMP_WITH_LOCAL_TIME_ZONE`, and `MULTISET` * Flink's `INTERVAL` type cannot be mapped to Hive `INTERVAL` type yet + +## Scala Shell + +NOTE: since blink planner is not well supported in Scala Shell at the moment, it's **NOT** recommended to use Hive connector in Scala Shell. diff --git a/docs/dev/table/hive/hive_catalog.zh.md b/docs/dev/table/hive/hive_catalog.zh.md index d907703fe24b7e..05479b9a035eca 100644 --- a/docs/dev/table/hive/hive_catalog.zh.md +++ b/docs/dev/table/hive/hive_catalog.zh.md @@ -387,3 +387,7 @@ Something to note about the type mapping: * Hive's `TIMESTAMP` always has precision 9 and doesn't support other precisions. Hive UDFs, on the other hand, can process `TIMESTAMP` values with a precision <= 9. * Hive doesn't support Flink's `TIMESTAMP_WITH_TIME_ZONE`, `TIMESTAMP_WITH_LOCAL_TIME_ZONE`, and `MULTISET` * Flink's `INTERVAL` type cannot be mapped to Hive `INTERVAL` type yet + +## Scala Shell + +注意：目前 blink planner 还不能很好的支持 Scala Shell，因此 **不** 建议在 Scala Shell 中使用 Hive 连接器。 diff --git a/docs/dev/table/hive/hive_dialect.md b/docs/dev/table/hive/hive_dialect.md new file mode 100644 index 00000000000000..c5c039ca226cbe --- /dev/null +++ b/docs/dev/table/hive/hive_dialect.md @@ -0,0 +1,350 @@ +--- +title: "Hive Dialect" +nav-parent_id: hive_tableapi +nav-pos: 1 +--- + + +Starting from 1.11.0, Flink allows users to write SQL statements in Hive syntax when Hive dialect +is used. By providing compatibility with Hive syntax, we aim to improve the interoperability with +Hive and reduce the scenarios when users need to switch between Flink and Hive in order to execute +different statements. + +* This will be replaced by the TOC +{:toc} + +## Use Hive Dialect + +Flink currently supports two SQL dialects: `default` and `hive`. You need to switch to Hive dialect +before you can write in Hive syntax. The following describes how to set dialect with +SQL Client and Table API. Also notice that you can dynamically switch dialect for each +statement you execute. There's no need to restart a session to use a different dialect. + +### SQL Client + +SQL dialect can be specified via the `table.sql-dialect` property. Therefore you can set the initial dialect to use in +the `configuration` section of the yaml file for your SQL Client. + +{% highlight yaml %} + +execution: + planner: blink + type: batch + result-mode: table + +configuration: + table.sql-dialect: hive + +{% endhighlight %} + +You can also set the dialect after the SQL Client has launched. + +{% highlight bash %} + +Flink SQL> set table.sql-dialect=hive; -- to use hive dialect +[INFO] Session property has been set. + +Flink SQL> set table.sql-dialect=default; -- to use default dialect +[INFO] Session property has been set. + +{% endhighlight %} + +### Table API + +You can set dialect for your TableEnvironment with Table API. + +{% highlight java %} + +EnvironmentSettings settings = EnvironmentSettings.newInstance().useBlinkPlanner()...build(); +TableEnvironment tableEnv = TableEnvironment.create(settings); +// to use hive dialect +tableEnv.getConfig().setSqlDialect(SqlDialect.HIVE); +// to use default dialect +tableEnv.getConfig().setSqlDialect(SqlDialect.DEFAULT); + +{% endhighlight %} + +## DDL + +This section lists the supported DDLs with the Hive dialect. We'll mainly focus on the syntax +here. You can refer to [Hive doc](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL) +for the semantics of each DDL statement. + +### DATABASE + +#### Show + +{% highlight sql %} +SHOW DATABASES; +{% endhighlight %} + +#### Create + +{% highlight sql %} +CREATE (DATABASE|SCHEMA) [IF NOT EXISTS] database_name + [COMMENT database_comment] + [LOCATION fs_path] + [WITH DBPROPERTIES (property_name=property_value, ...)]; +{% endhighlight %} + +#### Alter + +##### Update Properties + +{% highlight sql %} +ALTER (DATABASE|SCHEMA) database_name SET DBPROPERTIES (property_name=property_value, ...); +{% endhighlight %} + +##### Update Owner + +{% highlight sql %} +ALTER (DATABASE|SCHEMA) database_name SET OWNER [USER|ROLE] user_or_role; +{% endhighlight %} + +##### Update Location + +{% highlight sql %} +ALTER (DATABASE|SCHEMA) database_name SET LOCATION fs_path; +{% endhighlight %} + +#### Drop + +{% highlight sql %} +DROP (DATABASE|SCHEMA) [IF EXISTS] database_name [RESTRICT|CASCADE]; +{% endhighlight %} + +#### Use + +{% highlight sql %} +USE database_name; +{% endhighlight %} + +### TABLE + +#### Show + +{% highlight sql %} +SHOW TABLES; +{% endhighlight %} + +#### Create + +{% highlight sql %} +CREATE [EXTERNAL] TABLE [IF NOT EXISTS] table_name + [(col_name data_type [column_constraint] [COMMENT col_comment], ... [table_constraint])] + [COMMENT table_comment] + [PARTITIONED BY (col_name data_type [COMMENT col_comment], ...)] + [ + [ROW FORMAT row_format] + [STORED AS file_format] + ] + [LOCATION fs_path] + [TBLPROPERTIES (property_name=property_value, ...)] + +row_format: + : DELIMITED [FIELDS TERMINATED BY char [ESCAPED BY char]] [COLLECTION ITEMS TERMINATED BY char] + [MAP KEYS TERMINATED BY char] [LINES TERMINATED BY char] + [NULL DEFINED AS char] + | SERDE serde_name [WITH SERDEPROPERTIES (property_name=property_value, ...)] + +file_format: + : SEQUENCEFILE + | TEXTFILE + | RCFILE + | ORC + | PARQUET + | AVRO + | INPUTFORMAT input_format_classname OUTPUTFORMAT output_format_classname + +column_constraint: + : NOT NULL [[ENABLE|DISABLE] [VALIDATE|NOVALIDATE] [RELY|NORELY]] + +table_constraint: + : [CONSTRAINT constraint_name] PRIMARY KEY (col_name, ...) [[ENABLE|DISABLE] [VALIDATE|NOVALIDATE] [RELY|NORELY]] +{% endhighlight %} + +#### Alter + +##### Rename + +{% highlight sql %} +ALTER TABLE table_name RENAME TO new_table_name; +{% endhighlight %} + +##### Update Properties + +{% highlight sql %} +ALTER TABLE table_name SET TBLPROPERTIES (property_name = property_value, property_name = property_value, ... ); +{% endhighlight %} + +##### Update Location + +{% highlight sql %} +ALTER TABLE table_name [PARTITION partition_spec] SET LOCATION fs_path; +{% endhighlight %} + +The `partition_spec`, if present, needs to be a full spec, i.e. has values for all partition columns. And when it's +present, the operation will be applied to the corresponding partition instead of the table. + +##### Update File Format + +{% highlight sql %} +ALTER TABLE table_name [PARTITION partition_spec] SET FILEFORMAT file_format; +{% endhighlight %} + +The `partition_spec`, if present, needs to be a full spec, i.e. has values for all partition columns. And when it's +present, the operation will be applied to the corresponding partition instead of the table. + +##### Update SerDe Properties + +{% highlight sql %} +ALTER TABLE table_name [PARTITION partition_spec] SET SERDE serde_class_name [WITH SERDEPROPERTIES serde_properties]; + +ALTER TABLE table_name [PARTITION partition_spec] SET SERDEPROPERTIES serde_properties; + +serde_properties: + : (property_name = property_value, property_name = property_value, ... ) +{% endhighlight %} + +The `partition_spec`, if present, needs to be a full spec, i.e. has values for all partition columns. And when it's +present, the operation will be applied to the corresponding partition instead of the table. + +##### Add Partitions + +{% highlight sql %} +ALTER TABLE table_name ADD [IF NOT EXISTS] (PARTITION partition_spec [LOCATION fs_path])+; +{% endhighlight %} + +##### Drop Partitions + +{% highlight sql %} +ALTER TABLE table_name DROP [IF EXISTS] PARTITION partition_spec[, PARTITION partition_spec, ...]; +{% endhighlight %} + +##### Add/Replace Columns + +{% highlight sql %} +ALTER TABLE table_name + ADD|REPLACE COLUMNS (col_name data_type [COMMENT col_comment], ...) + [CASCADE|RESTRICT] +{% endhighlight %} + +##### Change Column + +{% highlight sql %} +ALTER TABLE table_name CHANGE [COLUMN] col_old_name col_new_name column_type + [COMMENT col_comment] [FIRST|AFTER column_name] [CASCADE|RESTRICT]; +{% endhighlight %} + +#### Drop + +{% highlight sql %} +DROP TABLE [IF EXISTS] table_name; +{% endhighlight %} + +### VIEW + +#### Create + +{% highlight sql %} +CREATE VIEW [IF NOT EXISTS] view_name [(column_name, ...) ] + [COMMENT view_comment] + [TBLPROPERTIES (property_name = property_value, ...)] + AS SELECT ...; +{% endhighlight %} + +#### Alter + +**NOTE**: Altering view only works in Table API, but not supported via SQL client. + +##### Rename + +{% highlight sql %} +ALTER VIEW view_name RENAME TO new_view_name; +{% endhighlight %} + +##### Update Properties + +{% highlight sql %} +ALTER VIEW view_name SET TBLPROPERTIES (property_name = property_value, ... ); +{% endhighlight %} + +##### Update As Select + +{% highlight sql %} +ALTER VIEW view_name AS select_statement; +{% endhighlight %} + +#### Drop + +{% highlight sql %} +DROP VIEW [IF EXISTS] view_name; +{% endhighlight %} + +### FUNCTION + +#### Show + +{% highlight sql %} +SHOW FUNCTIONS; +{% endhighlight %} + +#### Create + +{% highlight sql %} +CREATE FUNCTION function_name AS class_name; +{% endhighlight %} + +#### Drop + +{% highlight sql %} +DROP FUNCTION [IF EXISTS] function_name; +{% endhighlight %} + +## DML + +### INSERT + +{% highlight sql %} +INSERT (INTO|OVERWRITE) [TABLE] table_name [PARTITION partition_spec] SELECT ...; +{% endhighlight %} + +The `partition_spec`, if present, can be either a full spec or partial spec. If the `partition_spec` is a partial +spec, the dynamic partition column names can be omitted. + +## DQL + +At the moment, Hive dialect supports the same syntax as Flink SQL for DQLs. Refer to +[Flink SQL queries]({{ site.baseurl }}/dev/table/sql/queries.html) for more details. And it's recommended to switch to +`default` dialect to execute DQLs. + +## Notice + +The following are some precautions for using the Hive dialect. + +- Hive dialect should only be used to manipulate Hive tables, not generic tables. And Hive dialect should be used together +with a [HiveCatalog]({{ site.baseurl }}/dev/table/hive/hive_catalog.html). +- While all Hive versions support the same syntax, whether a specific feature is available still depends on the +[Hive version]({{ site.baseurl }}/dev/table/hive/#supported-hive-versions) you use. For example, updating database +location is only supported in Hive-2.4.0 or later. +- Hive and Calcite have different sets of reserved keywords. For example, `default` is a reserved keyword in Calcite and +a non-reserved keyword in Hive. Even with Hive dialect, you have to quote such keywords with backtick ( ` ) in order to +use them as identifiers. +- Due to expanded query incompatibility, views created in Flink cannot be queried in Hive. diff --git a/docs/dev/table/hive/hive_dialect.zh.md b/docs/dev/table/hive/hive_dialect.zh.md new file mode 100644 index 00000000000000..1236ad8324fef3 --- /dev/null +++ b/docs/dev/table/hive/hive_dialect.zh.md @@ -0,0 +1,350 @@ +--- +title: "Hive Dialect" +nav-parent_id: hive_tableapi +nav-pos: 1 +--- + + +Starting from 1.11.0, Flink allows users to write SQL statements in Hive syntax when Hive dialect +is used. By providing compatibility with Hive syntax, we aim to improve the interoperability with +Hive and reduce the scenarios when users need to switch between Flink and Hive in order to execute +different statements. + +* This will be replaced by the TOC +{:toc} + +## Use Hive Dialect + +Flink currently supports two SQL dialects: `default` and `hive`. You need to switch to Hive dialect +before you can write in Hive syntax. The following describes how to set dialect with +SQL Client and Table API. Also notice that you can dynamically switch dialect for each +statement you execute. There's no need to restart a session to use a different dialect. + +### SQL Client + +SQL dialect can be specified via the `table.sql-dialect` property. Therefore you can set the initial dialect to use in +the `configuration` section of the yaml file for your SQL Client. + +{% highlight yaml %} + +execution: + planner: blink + type: batch + result-mode: table + +configuration: + table.sql-dialect: hive + +{% endhighlight %} + +You can also set the dialect after the SQL Client has launched. + +{% highlight bash %} + +Flink SQL> set table.sql-dialect=hive; -- to use hive dialect +[INFO] Session property has been set. + +Flink SQL> set table.sql-dialect=default; -- to use default dialect +[INFO] Session property has been set. + +{% endhighlight %} + +### Table API + +You can set dialect for your TableEnvironment with Table API. + +{% highlight java %} + +EnvironmentSettings settings = EnvironmentSettings.newInstance().useBlinkPlanner()...build(); +TableEnvironment tableEnv = TableEnvironment.create(settings); +// to use hive dialect +tableEnv.getConfig().setSqlDialect(SqlDialect.HIVE); +// to use default dialect +tableEnv.getConfig().setSqlDialect(SqlDialect.DEFAULT); + +{% endhighlight %} + +## DDL + +This section lists the supported DDLs with the Hive dialect. We'll mainly focus on the syntax +here. You can refer to [Hive doc](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL) +for the semantics of each DDL statement. + +### DATABASE + +#### Show + +{% highlight sql %} +SHOW DATABASES; +{% endhighlight %} + +#### Create + +{% highlight sql %} +CREATE (DATABASE|SCHEMA) [IF NOT EXISTS] database_name + [COMMENT database_comment] + [LOCATION fs_path] + [WITH DBPROPERTIES (property_name=property_value, ...)]; +{% endhighlight %} + +#### Alter + +##### Update Properties + +{% highlight sql %} +ALTER (DATABASE|SCHEMA) database_name SET DBPROPERTIES (property_name=property_value, ...); +{% endhighlight %} + +##### Update Owner + +{% highlight sql %} +ALTER (DATABASE|SCHEMA) database_name SET OWNER [USER|ROLE] user_or_role; +{% endhighlight %} + +##### Update Location + +{% highlight sql %} +ALTER (DATABASE|SCHEMA) database_name SET LOCATION fs_path; +{% endhighlight %} + +#### Drop + +{% highlight sql %} +DROP (DATABASE|SCHEMA) [IF EXISTS] database_name [RESTRICT|CASCADE]; +{% endhighlight %} + +#### Use + +{% highlight sql %} +USE database_name; +{% endhighlight %} + +### TABLE + +#### Show + +{% highlight sql %} +SHOW TABLES; +{% endhighlight %} + +#### Create + +{% highlight sql %} +CREATE [EXTERNAL] TABLE [IF NOT EXISTS] table_name + [(col_name data_type [column_constraint] [COMMENT col_comment], ... [table_constraint])] + [COMMENT table_comment] + [PARTITIONED BY (col_name data_type [COMMENT col_comment], ...)] + [ + [ROW FORMAT row_format] + [STORED AS file_format] + ] + [LOCATION fs_path] + [TBLPROPERTIES (property_name=property_value, ...)] + +row_format: + : DELIMITED [FIELDS TERMINATED BY char [ESCAPED BY char]] [COLLECTION ITEMS TERMINATED BY char] + [MAP KEYS TERMINATED BY char] [LINES TERMINATED BY char] + [NULL DEFINED AS char] + | SERDE serde_name [WITH SERDEPROPERTIES (property_name=property_value, ...)] + +file_format: + : SEQUENCEFILE + | TEXTFILE + | RCFILE + | ORC + | PARQUET + | AVRO + | INPUTFORMAT input_format_classname OUTPUTFORMAT output_format_classname + +column_constraint: + : NOT NULL [[ENABLE|DISABLE] [VALIDATE|NOVALIDATE] [RELY|NORELY]] + +table_constraint: + : [CONSTRAINT constraint_name] PRIMARY KEY (col_name, ...) [[ENABLE|DISABLE] [VALIDATE|NOVALIDATE] [RELY|NORELY]] +{% endhighlight %} + +#### Alter + +##### Rename + +{% highlight sql %} +ALTER TABLE table_name RENAME TO new_table_name; +{% endhighlight %} + +##### Update Properties + +{% highlight sql %} +ALTER TABLE table_name SET TBLPROPERTIES (property_name = property_value, property_name = property_value, ... ); +{% endhighlight %} + +##### Update Location + +{% highlight sql %} +ALTER TABLE table_name [PARTITION partition_spec] SET LOCATION fs_path; +{% endhighlight %} + +The `partition_spec`, if present, needs to be a full spec, i.e. has values for all partition columns. And when it's +present, the operation will be applied to the corresponding partition instead of the table. + +##### Update File Format + +{% highlight sql %} +ALTER TABLE table_name [PARTITION partition_spec] SET FILEFORMAT file_format; +{% endhighlight %} + +The `partition_spec`, if present, needs to be a full spec, i.e. has values for all partition columns. And when it's +present, the operation will be applied to the corresponding partition instead of the table. + +##### Update SerDe Properties + +{% highlight sql %} +ALTER TABLE table_name [PARTITION partition_spec] SET SERDE serde_class_name [WITH SERDEPROPERTIES serde_properties]; + +ALTER TABLE table_name [PARTITION partition_spec] SET SERDEPROPERTIES serde_properties; + +serde_properties: + : (property_name = property_value, property_name = property_value, ... ) +{% endhighlight %} + +The `partition_spec`, if present, needs to be a full spec, i.e. has values for all partition columns. And when it's +present, the operation will be applied to the corresponding partition instead of the table. + +##### Add Partitions + +{% highlight sql %} +ALTER TABLE table_name ADD [IF NOT EXISTS] (PARTITION partition_spec [LOCATION fs_path])+; +{% endhighlight %} + +##### Drop Partitions + +{% highlight sql %} +ALTER TABLE table_name DROP [IF EXISTS] PARTITION partition_spec[, PARTITION partition_spec, ...]; +{% endhighlight %} + +##### Add/Replace Columns + +{% highlight sql %} +ALTER TABLE table_name + ADD|REPLACE COLUMNS (col_name data_type [COMMENT col_comment], ...) + [CASCADE|RESTRICT] +{% endhighlight %} + +##### Change Column + +{% highlight sql %} +ALTER TABLE table_name CHANGE [COLUMN] col_old_name col_new_name column_type + [COMMENT col_comment] [FIRST|AFTER column_name] [CASCADE|RESTRICT]; +{% endhighlight %} + +#### Drop + +{% highlight sql %} +DROP TABLE [IF EXISTS] table_name; +{% endhighlight %} + +### VIEW + +#### Create + +{% highlight sql %} +CREATE VIEW [IF NOT EXISTS] view_name [(column_name, ...) ] + [COMMENT view_comment] + [TBLPROPERTIES (property_name = property_value, ...)] + AS SELECT ...; +{% endhighlight %} + +#### Alter + +**NOTE**: Altering view only works in Table API, but not supported via SQL client. + +##### Rename + +{% highlight sql %} +ALTER VIEW view_name RENAME TO new_view_name; +{% endhighlight %} + +##### Update Properties + +{% highlight sql %} +ALTER VIEW view_name SET TBLPROPERTIES (property_name = property_value, ... ); +{% endhighlight %} + +##### Update As Select + +{% highlight sql %} +ALTER VIEW view_name AS select_statement; +{% endhighlight %} + +#### Drop + +{% highlight sql %} +DROP VIEW [IF EXISTS] view_name; +{% endhighlight %} + +### FUNCTION + +#### Show + +{% highlight sql %} +SHOW FUNCTIONS; +{% endhighlight %} + +#### Create + +{% highlight sql %} +CREATE FUNCTION function_name AS class_name; +{% endhighlight %} + +#### Drop + +{% highlight sql %} +DROP FUNCTION [IF EXISTS] function_name; +{% endhighlight %} + +## DML + +### INSERT + +{% highlight sql %} +INSERT (INTO|OVERWRITE) [TABLE] table_name [PARTITION partition_spec] SELECT ...; +{% endhighlight %} + +The `partition_spec`, if present, can be either a full spec or partial spec. If the `partition_spec` is a partial +spec, the dynamic partition column names can be omitted. + +## DQL + +At the moment, Hive dialect supports the same syntax as Flink SQL for DQLs. Refer to +[Flink SQL queries]({{ site.baseurl }}/zh/dev/table/sql/queries.html) for more details. And it's recommended to switch to +`default` dialect to execute DQLs. + +## Notice + +The following are some precautions for using the Hive dialect. + +- Hive dialect should only be used to manipulate Hive tables, not generic tables. And Hive dialect should be used together +with a [HiveCatalog]({{ site.baseurl }}/zh/dev/table/hive/hive_catalog.html). +- While all Hive versions support the same syntax, whether a specific feature is available still depends on the +[Hive version]({{ site.baseurl }}/zh/dev/table/hive/#支持的hive版本) you use. For example, updating database +location is only supported in Hive-2.4.0 or later. +- Hive and Calcite have different sets of reserved keywords. For example, `default` is a reserved keyword in Calcite and +a non-reserved keyword in Hive. Even with Hive dialect, you have to quote such keywords with backtick ( ` ) in order to +use them as identifiers. +- Due to expanded query incompatibility, views created in Flink cannot be queried in Hive. diff --git a/docs/dev/table/hive/hive_functions.zh.md b/docs/dev/table/hive/hive_functions.zh.md index 537d684aa37d71..c2e46cf4f6dcb8 100644 --- a/docs/dev/table/hive/hive_functions.zh.md +++ b/docs/dev/table/hive/hive_functions.zh.md @@ -1,5 +1,5 @@ --- -title: "Hive functions" +title: "Hive 函数" nav-parent_id: hive_tableapi nav-pos: 3 --- @@ -22,11 +22,11 @@ specific language governing permissions and limitations under the License. --> -## Use Hive Built-in Functions via HiveModule +## 通过 HiveModule 使用 Hive 内置函数 -The `HiveModule` provides Hive built-in functions as Flink system (built-in) functions to Flink SQL and Table API users. +在 Flink SQL 和 Table API 中，可以通过系统内置的 `HiveModule` 来使用 Hive 内置函数， -For detailed information, please refer to [HiveModule]({{ site.baseurl }}/dev/table/modules.html#hivemodule). +详细信息，请参考 [HiveModule]({{ site.baseurl }}/zh/dev/table/modules.html#hivemodule)。-{% highlight java %} -public class HashCode extends ScalarFunction { - - private int factor = 0; - - @Override - public void open(FunctionContext context) throws Exception { - // access "hashcode_factor" parameter - // "12" would be the default value if parameter does not exist - factor = Integer.valueOf(context.getJobParameter("hashcode_factor", "12")); - } - - public int eval(String s) { - return s.hashCode() * factor; - } -} - -ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment(); -BatchTableEnvironment tableEnv = BatchTableEnvironment.create(env); - -// set job parameter -Configuration conf = new Configuration(); -conf.setString("hashcode_factor", "31"); -env.getConfig().setGlobalJobParameters(conf); - -// register the function -tableEnv.registerFunction("hashCode", new HashCode()); - -// use the function in Java Table API -myTable.select("string, string.hashCode(), hashCode(string)"); - -// use the function in SQL -tableEnv.sqlQuery("SELECT string, HASHCODE(string) FROM MyTable"); -{% endhighlight %} -- --{% highlight scala %} -object hashCode extends ScalarFunction { - - var hashcode_factor = 12 - - override def open(context: FunctionContext): Unit = { - // access "hashcode_factor" parameter - // "12" would be the default value if parameter does not exist - hashcode_factor = context.getJobParameter("hashcode_factor", "12").toInt - } - - def eval(s: String): Int = { - s.hashCode() * hashcode_factor - } -} - -val tableEnv = BatchTableEnvironment.create(env) - -// use the function in Scala Table API -myTable.select('string, hashCode('string)) - -// register and use the function in SQL -tableEnv.registerFunction("hashCode", hashCode) -tableEnv.sqlQuery("SELECT string, HASHCODE(string) FROM MyTable") -{% endhighlight %} - ---* NOTE that some Hive built-in functions in older versions have [thread safety issues](https://issues.apache.org/jira/browse/HIVE-16183). -We recommend users patch their own Hive to fix them. +* 请注意旧版本的部分 Hive 内置函数存在[线程安全问题](https://issues.apache.org/jira/browse/HIVE-16183)。 +我们建议用户及时通过补丁修正 Hive 中的这些问题。 -## Hive User Defined Functions +## Hive 用户自定义函数(User Defined Functions) -Users can use their existing Hive User Defined Functions in Flink. +在 Flink 中用户可以使用 Hive 里已经存在的 UDF 函数。 -Supported UDF types include: +支持的 UDF 类型包括： - UDF - GenericUDF @@ -73,24 +73,23 @@ Supported UDF types include: - UDAF - GenericUDAFResolver2 -Upon query planning and execution, Hive's UDF and GenericUDF are automatically translated into Flink's ScalarFunction, -Hive's GenericUDTF is automatically translated into Flink's TableFunction, -and Hive's UDAF and GenericUDAFResolver2 are translated into Flink's AggregateFunction. +在进行查询规划和执行时，Hive UDF 和 GenericUDF 函数会自动转换成 Flink 中的 ScalarFunction，GenericUDTF 会被自动转换成 Flink 中的 + TableFunction，UDAF 和 GenericUDAFResolver2 则转换成 Flink 聚合函数(AggregateFunction). -To use a Hive User Defined Function, user have to +想要使用 Hive UDF 函数，需要如下几步： -- set a HiveCatalog backed by Hive Metastore that contains that function as current catalog of the session -- include a jar that contains that function in Flink's classpath -- use Blink planner. +- 通过 Hive Metastore 将带有 UDF 的 HiveCatalog 设置为当前会话的 catalog 后端。 +- 将带有 UDF 的 jar 包放入 Flink classpath 中，并在代码中引入。 +- 使用 Blink planner。 -## Using Hive User Defined Functions +## 使用 Hive UDF -Assuming we have the following Hive functions registered in Hive Metastore: +假设我们在 Hive Metastore 中已经注册了下面的 UDF 函数： {% highlight java %} /** - * Test simple udf. Registered under name 'myudf' + * 注册为 'myudf' 的简单 UDF 测试类. */ public class TestHiveSimpleUDF extends UDF { @@ -104,7 +103,7 @@ public class TestHiveSimpleUDF extends UDF { } /** - * Test generic udf. Registered under name 'mygenericudf' + * 注册为 'mygenericudf' 的普通 UDF 测试类 */ public class TestHiveGenericUDF extends GenericUDF { @@ -137,7 +136,7 @@ public class TestHiveGenericUDF extends GenericUDF { } /** - * Test split udtf. Registered under name 'mygenericudtf' + * 注册为 'mygenericudtf' 的字符串分割 UDF 测试类 */ public class TestHiveUDTF extends GenericUDTF { @@ -172,7 +171,7 @@ public class TestHiveUDTF extends GenericUDTF { {% endhighlight %} -From Hive CLI, we can see they are registered: +在 Hive CLI 中，可以查询到已经注册的 UDF 函数: {% highlight bash %} hive> show functions; @@ -184,8 +183,7 @@ myudtf {% endhighlight %} - -Then, users can use them in SQL as: +此时，用户如果想使用这些 UDF，在 SQL 中就可以这样写： {% highlight bash %} diff --git a/docs/dev/table/hive/hive_read_write.md b/docs/dev/table/hive/hive_read_write.md new file mode 100644 index 00000000000000..5185cc7df7be96 --- /dev/null +++ b/docs/dev/table/hive/hive_read_write.md @@ -0,0 +1,228 @@ +--- +title: "Hive Read & Write" +nav-parent_id: hive_tableapi +nav-pos: 2 +--- + + +Using the `HiveCatalog` and Flink's connector to Hive, Flink can read and write from Hive data as an alternative to Hive's batch engine. +Be sure to follow the instructions to include the correct [dependencies]({{ site.baseurl }}/dev/table/hive/#depedencies) in your application. +And please also note that Hive connector only works with blink planner. + +* This will be replaced by the TOC +{:toc} + +## Reading From Hive + +Assume Hive contains a single table in its `default` database, named people that contains several rows. + +{% highlight bash %} +hive> show databases; +OK +default +Time taken: 0.841 seconds, Fetched: 1 row(s) + +hive> show tables; +OK +Time taken: 0.087 seconds + +hive> CREATE TABLE mytable(name string, value double); +OK +Time taken: 0.127 seconds + +hive> SELECT * FROM mytable; +OK +Tom 4.72 +John 8.0 +Tom 24.2 +Bob 3.14 +Bob 4.72 +Tom 34.9 +Mary 4.79 +Tiff 2.72 +Bill 4.33 +Mary 77.7 +Time taken: 0.097 seconds, Fetched: 10 row(s) +{% endhighlight %} + +With the data ready your can connect to Hive [connect to an existing Hive installation]({{ site.baseurl }}/dev/table/hive/#connecting-to-hive) and begin querying. + +{% highlight bash %} + +Flink SQL> show catalogs; +myhive +default_catalog + +# ------ Set the current catalog to be 'myhive' catalog if you haven't set it in the yaml file ------ + +Flink SQL> use catalog myhive; + +# ------ See all registered database in catalog 'mytable' ------ + +Flink SQL> show databases; +default + +# ------ See the previously registered table 'mytable' ------ + +Flink SQL> show tables; +mytable + +# ------ The table schema that Flink sees is the same that we created in Hive, two columns - name as string and value as double ------ +Flink SQL> describe mytable; +root + |-- name: name + |-- type: STRING + |-- name: value + |-- type: DOUBLE + +# ------ Select from hive table or hive view ------ +Flink SQL> SELECT * FROM mytable; + + name value +__________ __________ + + Tom 4.72 + John 8.0 + Tom 24.2 + Bob 3.14 + Bob 4.72 + Tom 34.9 + Mary 4.79 + Tiff 2.72 + Bill 4.33 + Mary 77.7 + +{% endhighlight %} + +### Querying Hive views + +If you need to query Hive views, please note: + +1. You have to use the Hive catalog as your current catalog before you can query views in that catalog. It can be done by either `tableEnv.useCatalog(...)` in Table API or `USE CATALOG ...` in SQL Client. +2. Hive and Flink SQL have different syntax, e.g. different reserved keywords and literals. Make sure the view's query is compatible with Flink grammar. + +## Writing To Hive + +Similarly, data can be written into hive using an `INSERT` clause. + +Consider there is an example table named "mytable" with two columns: name and age, in string and int type. + +{% highlight bash %} +# ------ INSERT INTO will append to the table or partition, keeping the existing data intact ------ +Flink SQL> INSERT INTO mytable SELECT 'Tom', 25; + +# ------ INSERT OVERWRITE will overwrite any existing data in the table or partition ------ +Flink SQL> INSERT OVERWRITE mytable SELECT 'Tom', 25; +{% endhighlight %} + +We support partitioned table too, Consider there is a partitioned table named myparttable with four columns: name, age, my_type and my_date, in types ...... my_type and my_date are the partition keys. + +{% highlight bash %} +# ------ Insert with static partition ------ +Flink SQL> INSERT OVERWRITE myparttable PARTITION (my_type='type_1', my_date='2019-08-08') SELECT 'Tom', 25; + +# ------ Insert with dynamic partition ------ +Flink SQL> INSERT OVERWRITE myparttable SELECT 'Tom', 25, 'type_1', '2019-08-08'; + +# ------ Insert with static(my_type) and dynamic(my_date) partition ------ +Flink SQL> INSERT OVERWRITE myparttable PARTITION (my_type='type_1') SELECT 'Tom', 25, '2019-08-08'; +{% endhighlight %} + + +## Formats + +We have tested on the following of table storage formats: text, csv, SequenceFile, ORC, and Parquet. + + +## Optimizations + +### Partition Pruning + +Flink uses partition pruning as a performance optimization to limits the number of files and partitions +that Flink reads when querying Hive tables. When your data is partitioned, Flink only reads a subset of the partitions in +a Hive table when a query matches certain filter criteria. + +### Projection Pushdown + +Flink leverages projection pushdown to minimize data transfer between Flink and Hive tables by omitting +unnecessary fields from table scans. + +It is especially beneficial when a table contains many columns. + +### Limit Pushdown + +For queries with LIMIT clause, Flink will limit the number of output records wherever possible to minimize the +amount of data transferred across network. + +### Vectorized Optimization upon Read + +Optimization is used automatically when the following conditions are met: + +- Format: ORC or Parquet. +- Columns without complex data type, like hive types: List, Map, Struct, Union. + +This feature is turned on by default. If there is a problem, you can use this config option to close Vectorized Optimization: + +{% highlight bash %} +table.exec.hive.fallback-mapred-reader=true +{% endhighlight %} + +### Source Parallelism Inference + +By default, Flink infers the hive source parallelism based on the number of splits, and the number of +splits is based on the number of files and the number of blocks in the files. + +Flink allows you to flexibly configure the policy of parallelism inference. You can configure the +following parameters in `TableConfig` (note that these parameters affect all sources of the job): + +@@ -58,14 +58,14 @@ modules:+ +
+ +## Roadmap + +We are planning and actively working on supporting features like + +- ACID tables +- bucketed tables +- more formats + +Please reach out to the community for more feature request https://flink.apache.org/community.html#mailing-lists diff --git a/docs/dev/table/hive/hive_read_write.zh.md b/docs/dev/table/hive/hive_read_write.zh.md new file mode 100644 index 00000000000000..5185cc7df7be96 --- /dev/null +++ b/docs/dev/table/hive/hive_read_write.zh.md @@ -0,0 +1,228 @@ +--- +title: "Hive Read & Write" +nav-parent_id: hive_tableapi +nav-pos: 2 +--- + + +Using the `HiveCatalog` and Flink's connector to Hive, Flink can read and write from Hive data as an alternative to Hive's batch engine. +Be sure to follow the instructions to include the correct [dependencies]({{ site.baseurl }}/dev/table/hive/#depedencies) in your application. +And please also note that Hive connector only works with blink planner. + +* This will be replaced by the TOC +{:toc} + +## Reading From Hive + +Assume Hive contains a single table in its `default` database, named people that contains several rows. + +{% highlight bash %} +hive> show databases; +OK +default +Time taken: 0.841 seconds, Fetched: 1 row(s) + +hive> show tables; +OK +Time taken: 0.087 seconds + +hive> CREATE TABLE mytable(name string, value double); +OK +Time taken: 0.127 seconds + +hive> SELECT * FROM mytable; +OK +Tom 4.72 +John 8.0 +Tom 24.2 +Bob 3.14 +Bob 4.72 +Tom 34.9 +Mary 4.79 +Tiff 2.72 +Bill 4.33 +Mary 77.7 +Time taken: 0.097 seconds, Fetched: 10 row(s) +{% endhighlight %} + +With the data ready your can connect to Hive [connect to an existing Hive installation]({{ site.baseurl }}/dev/table/hive/#connecting-to-hive) and begin querying. + +{% highlight bash %} + +Flink SQL> show catalogs; +myhive +default_catalog + +# ------ Set the current catalog to be 'myhive' catalog if you haven't set it in the yaml file ------ + +Flink SQL> use catalog myhive; + +# ------ See all registered database in catalog 'mytable' ------ + +Flink SQL> show databases; +default + +# ------ See the previously registered table 'mytable' ------ + +Flink SQL> show tables; +mytable + +# ------ The table schema that Flink sees is the same that we created in Hive, two columns - name as string and value as double ------ +Flink SQL> describe mytable; +root + |-- name: name + |-- type: STRING + |-- name: value + |-- type: DOUBLE + +# ------ Select from hive table or hive view ------ +Flink SQL> SELECT * FROM mytable; + + name value +__________ __________ + + Tom 4.72 + John 8.0 + Tom 24.2 + Bob 3.14 + Bob 4.72 + Tom 34.9 + Mary 4.79 + Tiff 2.72 + Bill 4.33 + Mary 77.7 + +{% endhighlight %} + +### Querying Hive views + +If you need to query Hive views, please note: + +1. You have to use the Hive catalog as your current catalog before you can query views in that catalog. It can be done by either `tableEnv.useCatalog(...)` in Table API or `USE CATALOG ...` in SQL Client. +2. Hive and Flink SQL have different syntax, e.g. different reserved keywords and literals. Make sure the view's query is compatible with Flink grammar. + +## Writing To Hive + +Similarly, data can be written into hive using an `INSERT` clause. + +Consider there is an example table named "mytable" with two columns: name and age, in string and int type. + +{% highlight bash %} +# ------ INSERT INTO will append to the table or partition, keeping the existing data intact ------ +Flink SQL> INSERT INTO mytable SELECT 'Tom', 25; + +# ------ INSERT OVERWRITE will overwrite any existing data in the table or partition ------ +Flink SQL> INSERT OVERWRITE mytable SELECT 'Tom', 25; +{% endhighlight %} + +We support partitioned table too, Consider there is a partitioned table named myparttable with four columns: name, age, my_type and my_date, in types ...... my_type and my_date are the partition keys. + +{% highlight bash %} +# ------ Insert with static partition ------ +Flink SQL> INSERT OVERWRITE myparttable PARTITION (my_type='type_1', my_date='2019-08-08') SELECT 'Tom', 25; + +# ------ Insert with dynamic partition ------ +Flink SQL> INSERT OVERWRITE myparttable SELECT 'Tom', 25, 'type_1', '2019-08-08'; + +# ------ Insert with static(my_type) and dynamic(my_date) partition ------ +Flink SQL> INSERT OVERWRITE myparttable PARTITION (my_type='type_1') SELECT 'Tom', 25, '2019-08-08'; +{% endhighlight %} + + +## Formats + +We have tested on the following of table storage formats: text, csv, SequenceFile, ORC, and Parquet. + + +## Optimizations + +### Partition Pruning + +Flink uses partition pruning as a performance optimization to limits the number of files and partitions +that Flink reads when querying Hive tables. When your data is partitioned, Flink only reads a subset of the partitions in +a Hive table when a query matches certain filter criteria. + +### Projection Pushdown + +Flink leverages projection pushdown to minimize data transfer between Flink and Hive tables by omitting +unnecessary fields from table scans. + +It is especially beneficial when a table contains many columns. + +### Limit Pushdown + +For queries with LIMIT clause, Flink will limit the number of output records wherever possible to minimize the +amount of data transferred across network. + +### Vectorized Optimization upon Read + +Optimization is used automatically when the following conditions are met: + +- Format: ORC or Parquet. +- Columns without complex data type, like hive types: List, Map, Struct, Union. + +This feature is turned on by default. If there is a problem, you can use this config option to close Vectorized Optimization: + +{% highlight bash %} +table.exec.hive.fallback-mapred-reader=true +{% endhighlight %} + +### Source Parallelism Inference + +By default, Flink infers the hive source parallelism based on the number of splits, and the number of +splits is based on the number of files and the number of blocks in the files. + +Flink allows you to flexibly configure the policy of parallelism inference. You can configure the +following parameters in `TableConfig` (note that these parameters affect all sources of the job): + ++ + + +Key +Default +Type +Description ++ +table.exec.hive.infer-source-parallelism
true +Boolean +If is true, source parallelism is inferred according to splits number. If is false, parallelism of source are set by config. ++ + +table.exec.hive.infer-source-parallelism.max
1000 +Integer +Sets max infer parallelism for source operator. ++ +
+ +## Roadmap + +We are planning and actively working on supporting features like + +- ACID tables +- bucketed tables +- more formats + +Please reach out to the community for more feature request https://flink.apache.org/community.html#mailing-lists diff --git a/docs/dev/table/hive/hive_streaming.md b/docs/dev/table/hive/hive_streaming.md new file mode 100644 index 00000000000000..f5d94509052ade --- /dev/null +++ b/docs/dev/table/hive/hive_streaming.md @@ -0,0 +1,200 @@ +--- +title: "Hive Streaming" +nav-parent_id: hive_tableapi +nav-pos: 2 +--- + + +A typical hive job is scheduled periodically to execute, so there will be a large delay. + +Flink supports to write, read and join the hive table in the form of streaming. + +* This will be replaced by the TOC +{:toc} + +There are three types of streaming: + +- Writing streaming data into Hive table. +- Reading Hive table incrementally in the form of streaming. +- Streaming table join Hive table using [Temporal Table]({{ site.baseurl }}/dev/table/streaming/temporal_tables.html#temporal-table). + +## Streaming Writing + +The Hive table supports streaming writes, based on [Filesystem Streaming Sink]({{ site.baseurl }}/dev/table/connectors/filesystem.html#streaming-sink). + +The Hive Streaming Sink re-use Filesystem Streaming Sink to integrate Hadoop OutputFormat/RecordWriter to streaming writing. +Hadoop RecordWriters are Bulk-encoded Formats, Bulk Formats rolls files on every checkpoint. + +By default, now only have renaming committer, this means S3 filesystem can not supports exactly-once, +if you want to use Hive streaming sink in S3 filesystem, You can configure the following parameter to +false to use Flink native writers (only work for parquet and orc) in `TableConfig` (note that these +parameters affect all sinks of the job): + ++ + + +Key +Default +Type +Description ++ +table.exec.hive.infer-source-parallelism
true +Boolean +If is true, source parallelism is inferred according to splits number. If is false, parallelism of source are set by config. ++ + +table.exec.hive.infer-source-parallelism.max
1000 +Integer +Sets max infer parallelism for source operator. ++ +
+ +The below shows how the streaming sink can be used to write a streaming query to write data from Kafka into a Hive table with partition-commit, +and runs a batch query to read that data back out. + +{% highlight sql %} + +SET table.sql-dialect=hive; +CREATE TABLE hive_table ( + user_id STRING, + order_amount DOUBLE +) PARTITIONED BY (dt STRING, hr STRING) STORED AS parquet TBLPROPERTIES ( + 'partition.time-extractor.timestamp-pattern'='$dt $hr:00:00', + 'sink.partition-commit.trigger'='partition-time', + 'sink.partition-commit.delay'='1 h', + 'sink.partition-commit.policy.kind'='metastore,success-file' +); + +SET table.sql-dialect=default; +CREATE TABLE kafka_table ( + user_id STRING, + order_amount DOUBLE, + log_ts TIMESTAMP(3), + WATERMARK FOR log_ts AS log_ts - INTERVAL '5' SECOND +) WITH (...); + +-- streaming sql, insert into hive table +INSERT INTO TABLE hive_table SELECT user_id, order_amount, DATE_FORMAT(log_ts, 'yyyy-MM-dd'), DATE_FORMAT(log_ts, 'HH') FROM kafka_table; + +-- batch sql, select with partition pruning +SELECT * FROM hive_table WHERE dt='2020-05-20' and hr='12'; + +{% endhighlight %} + +## Streaming Reading + +To improve the real-time performance of hive reading, Flink support real-time Hive table stream read: + +- Partition table, monitor the generation of partition, and read the new partition incrementally. +- Non-partition table, monitor the generation of new files in the folder, and read new files incrementally. + +You can even use the 10 minute level partition strategy, and use Flink's Hive streaming reading and +Hive streaming writing to greatly improve the real-time performance of Hive data warehouse to quasi +real-time minute level. + ++ + + +Key +Default +Type +Description ++ + +table.exec.hive.fallback-mapred-writer
true +Boolean +If it is false, using flink native writer to write parquet and orc files; if it is true, using hadoop mapred record writer to write parquet and orc files. ++ +
+ +Note: + +- Monitor strategy is to scan all directories/files in location path now. If there are too many partitions, there will be performance problems. +- Streaming reading for non-partitioned requires that each file should be put atomically into the target directory. +- Streaming reading for partitioned requires that each partition should be add atomically in the view of hive metastore. This means that new data added to an existing partition won't be consumed. +- Streaming reading not support watermark grammar in Flink DDL. So it can not be used for window operators. + +The below shows how to read Hive table incrementally. + +{% highlight sql %} + +SELECT * FROM hive_table /*+ OPTIONS('streaming-source.enable'='true', 'streaming-source.consume-start-offset'='2020-05-20') */; + +{% endhighlight %} + +## Hive Table As Temporal Tables + +You can use a Hive table as temporal table and join streaming data with it. Please follow +the [example]({{ site.baseurl }}/dev/table/streaming/temporal_tables.html#temporal-table) to find out how to join a +temporal table. + +When performing the join, the Hive table will be cached in TM memory and each record from the stream +is looked up in the Hive table to decide whether a match is found. You don't need any extra settings to use a Hive table +as temporal table. But optionally, you can configure the TTL of the Hive table cache with the following +property. After the cache expires, the Hive table will be scanned again to load the latest data. + ++ + + +Key +Default +Type +Description ++ +streaming-source.enable
false +Boolean +Enable streaming source or not. NOTES: Please make sure that each partition/file should be written atomically, otherwise the reader may get incomplete data. ++ +streaming-source.monitor-interval
1 m +Duration +Time interval for consecutively monitoring partition/file. ++ +streaming-source.consume-order
create-time +String +The consume order of streaming source, support create-time and partition-time. create-time compare partition/file creation time, this is not the partition create time in Hive metaStore, but the folder/file modification time in filesystem; partition-time compare time represented by partition name, if the partition folder somehow gets updated, e.g. add new file into folder, it can affect how the data is consumed. For non-partition table, this value should always be 'create-time'. ++ + +streaming-source.consume-start-offset
1970-00-00 +String +Start offset for streaming consuming. How to parse and compare offsets depends on your order. For create-time and partition-time, should be a timestamp string (yyyy-[m]m-[d]d [hh:mm:ss]). For partition-time, will use partition time extractor to extract time from partition. ++ +
+ +**Note**: +1. Each joining subtask needs to keep its own cache of the Hive table. Please make sure the Hive table can fit into +the memory of a TM task slot. +2. You should set a relatively large value for `lookup.join.cache.ttl`. You'll probably have performance issue if +your Hive table needs to be updated and reloaded too frequently. +3. Currently we simply load the whole Hive table whenever the cache needs refreshing. There's no way to differentiate +new data from the old. diff --git a/docs/dev/table/hive/hive_streaming.zh.md b/docs/dev/table/hive/hive_streaming.zh.md new file mode 100644 index 00000000000000..8046ee5b08b749 --- /dev/null +++ b/docs/dev/table/hive/hive_streaming.zh.md @@ -0,0 +1,200 @@ +--- +title: "Hive Streaming" +nav-parent_id: hive_tableapi +nav-pos: 2 +--- + + +A typical hive job is scheduled periodically to execute, so there will be a large delay. + +Flink supports to write, read and join the hive table in the form of streaming. + +* This will be replaced by the TOC +{:toc} + +There are three types of streaming: + +- Writing streaming data into Hive table. +- Reading Hive table incrementally in the form of streaming. +- Streaming table join Hive table using [Temporal Table]({{ site.baseurl }}/dev/table/streaming/temporal_tables.html#temporal-table). + +## Streaming Writing + +The Hive table supports streaming writes, based on [Filesystem Streaming Sink]({{ site.baseurl }}/dev/table/connectors/filesystem.html#streaming-sink). + +The Hive Streaming Sink re-use Filesystem Streaming Sink to integrate Hadoop OutputFormat/RecordWriter to streaming writing. +Hadoop RecordWriters are Bulk-encoded Formats, Bulk Formats rolls files on every checkpoint. + +By default, now only have renaming committer, this means S3 filesystem can not supports exactly-once, +if you want to use Hive streaming sink in S3 filesystem, You can configure the following parameter to +false to use Flink native writers (only work for parquet and orc) in `TableConfig` (note that these +parameters affect all sinks of the job): + ++ + + +Key +Default +Type +Description ++ + +lookup.join.cache.ttl
60 min +Duration +The cache TTL (e.g. 10min) for the build table in lookup join. By default the TTL is 60 minutes. ++ +
+ +The below shows how the streaming sink can be used to write a streaming query to write data from Kafka into a Hive table with partition-commit, +and runs a batch query to read that data back out. + +{% highlight sql %} + +SET table.sql-dialect=hive; +CREATE TABLE hive_table ( + user_id STRING, + order_amount DOUBLE +) PARTITIONED BY (dt STRING, hr STRING) STORED AS parquet TBLPROPERTIES ( + 'partition.time-extractor.timestamp-pattern'='$dt $hr:00:00', + 'sink.partition-commit.trigger'='partition-time', + 'sink.partition-commit.delay'='1 h', + 'sink.partition-commit.policy.kind'='metastore,success-file' +); + +SET table.sql-dialect=default; +CREATE TABLE kafka_table ( + user_id STRING, + order_amount DOUBLE, + log_ts TIMESTAMP(3), + WATERMARK FOR log_ts AS log_ts - INTERVAL '5' SECOND +) WITH (...); + +-- streaming sql, insert into hive table +INSERT INTO TABLE hive_table SELECT user_id, order_amount, DATE_FORMAT(log_ts, 'yyyy-MM-dd'), DATE_FORMAT(log_ts, 'HH') FROM kafka_table; + +-- batch sql, select with partition pruning +SELECT * FROM hive_table WHERE dt='2020-05-20' and hr='12'; + +{% endhighlight %} + +## Streaming Reading + +To improve the real-time performance of hive reading, Flink support real-time Hive table stream read: + +- Partition table, monitor the generation of partition, and read the new partition incrementally. +- Non-partition table, monitor the generation of new files in the folder, and read new files incrementally. + +You can even use the 10 minute level partition strategy, and use Flink's Hive streaming reading and +Hive streaming writing to greatly improve the real-time performance of Hive data warehouse to quasi +real-time minute level. + ++ + + +Key +Default +Type +Description ++ + +table.exec.hive.fallback-mapred-writer
true +Boolean +If it is false, using flink native writer to write parquet and orc files; if it is true, using hadoop mapred record writer to write parquet and orc files. ++ +
+ +Note: + +- Monitor strategy is to scan all directories/files in location path now. If there are too many partitions, there will be performance problems. +- Streaming reading for non-partitioned requires that each file should be put atomically into the target directory. +- Streaming reading for partitioned requires that each partition should be add atomically in the view of hive metastore. This means that new data added to an existing partition won't be consumed. +- Streaming reading not support watermark grammar in Flink DDL. So it can not be used for window operators. + +The below shows how to read Hive table incrementally. + +{% highlight sql %} + +SELECT * FROM hive_table /*+ OPTIONS('streaming-source.enable'='true', 'streaming-source.consume-start-offset'='2020-05-20') */; + +{% endhighlight %} + +## Hive Table As Temporal Tables + +You can use a Hive table as temporal table and join streaming data with it. Please follow +the [example]({{ site.baseurl }}/zh/dev/table/streaming/temporal_tables.html#temporal-table) to find out how to join a +temporal table. + +When performing the join, the Hive table will be cached in TM memory and each record from the stream +is looked up in the Hive table to decide whether a match is found. You don't need any extra settings to use a Hive table +as temporal table. But optionally, you can configure the TTL of the Hive table cache with the following +property. After the cache expires, the Hive table will be scanned again to load the latest data. + ++ + + +Key +Default +Type +Description ++ +streaming-source.enable
false +Boolean +Enable streaming source or not. NOTES: Please make sure that each partition/file should be written atomically, otherwise the reader may get incomplete data. ++ +streaming-source.monitor-interval
1 m +Duration +Time interval for consecutively monitoring partition/file. ++ +streaming-source.consume-order
create-time +String +The consume order of streaming source, support create-time and partition-time. create-time compare partition/file creation time, this is not the partition create time in Hive metaStore, but the folder/file modification time in filesystem; partition-time compare time represented by partition name, if the partition folder somehow gets updated, e.g. add new file into folder, it can affect how the data is consumed. For non-partition table, this value should always be 'create-time'. ++ + +streaming-source.consume-start-offset
1970-00-00 +String +Start offset for streaming consuming. How to parse and compare offsets depends on your order. For create-time and partition-time, should be a timestamp string (yyyy-[m]m-[d]d [hh:mm:ss]). For partition-time, will use partition time extractor to extract time from partition. ++ +
+ +**Note**: +1. Each joining subtask needs to keep its own cache of the Hive table. Please make sure the Hive table can fit into +the memory of a TM task slot. +2. You should set a relatively large value for `lookup.join.cache.ttl`. You'll probably have performance issue if +your Hive table needs to be updated and reloaded too frequently. +3. Currently we simply load the whole Hive table whenever the cache needs refreshing. There's no way to differentiate +new data from the old. diff --git a/docs/dev/table/hive/index.md b/docs/dev/table/hive/index.md index 3698df2571f845..35b53cd2604603 100644 --- a/docs/dev/table/hive/index.md +++ b/docs/dev/table/hive/index.md @@ -101,20 +101,12 @@ There are two ways to add Hive dependencies. First is to use Flink's bundled Hiv The following tables list all available bundled hive jars. You can pick one to the `/lib/` directory in Flink distribution. -{% if site.is_stable %} - | Metastore version | Maven dependency | SQL Client JAR | | :---------------- | :--------------------------- | :----------------------| -| 1.0.0 - 1.2.2 | `flink-connector-hive-1.2.2` | [Download](http://central.maven.org/maven2/org/apache/flink/flink-sql-connector-hive-1.2.2{{site.scala_version_suffix}}/{{site.version}}/flink-sql-connector-hive-1.2.2{{site.scala_version_suffix}}-{{site.version}}.jar) | -| 2.0.0 - 2.2.0 | `flink-connector-hive-2.2.0` | [Download](http://central.maven.org/maven2/org/apache/flink/flink-sql-connector-hive-2.2.0{{site.scala_version_suffix}}/{{site.version}}/flink-sql-connector-hive-2.2.0{{site.scala_version_suffix}}-{{site.version}}.jar) | -| 2.3.0 - 2.3.6 | `flink-connector-hive-2.3.6` | [Download](http://central.maven.org/maven2/org/apache/flink/flink-sql-connector-hive-2.3.6{{site.scala_version_suffix}}/{{site.version}}/flink-sql-connector-hive-2.3.6{{site.scala_version_suffix}}-{{site.version}}.jar) | -| 3.0.0 - 3.1.2 | `flink-connector-hive-3.1.2` | [Download](http://central.maven.org/maven2/org/apache/flink/flink-sql-connector-hive-3.1.2{{site.scala_version_suffix}}/{{site.version}}/flink-sql-connector-hive-3.1.2{{site.scala_version_suffix}}-{{site.version}}.jar) | - -{% else %} - -These tables are only available for stable releases. - -{% endif %} +| 1.0.0 - 1.2.2 | `flink-sql-connector-hive-1.2.2` | {% if site.is_stable %}[Download](https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-hive-1.2.2{{site.scala_version_suffix}}/{{site.version}}/flink-sql-connector-hive-1.2.2{{site.scala_version_suffix}}-{{site.version}}.jar) {% else %} Only available for stable releases {% endif %} | +| 2.0.0 - 2.2.0 | `flink-sql-connector-hive-2.2.0` | {% if site.is_stable %}[Download](https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-hive-2.2.0{{site.scala_version_suffix}}/{{site.version}}/flink-sql-connector-hive-2.2.0{{site.scala_version_suffix}}-{{site.version}}.jar) {% else %} Only available for stable releases {% endif %} | +| 2.3.0 - 2.3.6 | `flink-sql-connector-hive-2.3.6` | {% if site.is_stable %}[Download](https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-hive-2.3.6{{site.scala_version_suffix}}/{{site.version}}/flink-sql-connector-hive-2.3.6{{site.scala_version_suffix}}-{{site.version}}.jar) {% else %} Only available for stable releases {% endif %} | +| 3.0.0 - 3.1.2 | `flink-sql-connector-hive-3.1.2` | {% if site.is_stable %}[Download](https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-hive-3.1.2{{site.scala_version_suffix}}/{{site.version}}/flink-sql-connector-hive-3.1.2{{site.scala_version_suffix}}-{{site.version}}.jar) {% else %} Only available for stable releases {% endif %} | #### User defined dependencies @@ -311,7 +303,7 @@ Take Hive version 2.3.4 for example:+ + + +Key +Default +Type +Description ++ + +lookup.join.cache.ttl
60 min +Duration +The cache TTL (e.g. 10min) for the build table in lookup join. By default the TTL is 60 minutes. +{% highlight java %} -EnvironmentSettings settings = EnvironmentSettings.newInstance().useBlinkPlanner().inBatchMode().build(); +EnvironmentSettings settings = EnvironmentSettings.newInstance().inBatchMode().build(); TableEnvironment tableEnv = TableEnvironment.create(settings); String name = "myhive"; @@ -329,7 +321,7 @@ tableEnv.useCatalog("myhive");{% highlight scala %} -val settings = EnvironmentSettings.newInstance().useBlinkPlanner().inBatchMode().build() +val settings = EnvironmentSettings.newInstance().inBatchMode().build() val tableEnv = TableEnvironment.create(settings) val name = "myhive" @@ -364,8 +356,9 @@ catalogs: ## DDL -DDL to create Hive tables, views, partitions, functions within Flink will be supported soon. +It's recommended to use [Hive dialect]({{ site.baseurl }}/dev/table/hive/hive_dialect.html) to execute DDLs to create +Hive tables, views, partitions, functions within Flink. ## DML -Flink supports DML writing to Hive tables. Please refer to details in [Reading & Writing Hive Tables]({{ site.baseurl }}/dev/table/hive/read_write_hive.html) +Flink supports DML writing to Hive tables. Please refer to details in [Reading & Writing Hive Tables]({{ site.baseurl }}/dev/table/hive/hive_read_write.html) diff --git a/docs/dev/table/hive/index.zh.md b/docs/dev/table/hive/index.zh.md index 8136a3b7ce8df3..735a3558f1ca76 100644 --- a/docs/dev/table/hive/index.zh.md +++ b/docs/dev/table/hive/index.zh.md @@ -100,21 +100,12 @@ Apache Hive 是基于 Hadoop 之上构建的, 首先您需要 Hadoop 的依赖 下表列出了所有可用的 Hive jar。您可以选择一个并放在 Flink 发行版的`/lib/` 目录中。 - -{% if site.is_stable %} - -| Metastore 版本 | Maven 依赖 | SQL Client JAR | +| Metastore version | Maven dependency | SQL Client JAR | | :---------------- | :--------------------------- | :----------------------| -| 1.0.0 - 1.2.2 | `flink-connector-hive-1.2.2` | [Download](http://central.maven.org/maven2/org/apache/flink/flink-sql-connector-hive-1.2.2{{site.scala_version_suffix}}/{{site.version}}/flink-sql-connector-hive-1.2.2{{site.scala_version_suffix}}-{{site.version}}.jar) | -| 2.0.0 - 2.2.0 | `flink-connector-hive-2.2.0` | [Download](http://central.maven.org/maven2/org/apache/flink/flink-sql-connector-hive-2.2.0{{site.scala_version_suffix}}/{{site.version}}/flink-sql-connector-hive-2.2.0{{site.scala_version_suffix}}-{{site.version}}.jar) | -| 2.3.0 - 2.3.6 | `flink-connector-hive-2.3.6` | [Download](http://central.maven.org/maven2/org/apache/flink/flink-sql-connector-hive-2.3.6{{site.scala_version_suffix}}/{{site.version}}/flink-sql-connector-hive-2.3.6{{site.scala_version_suffix}}-{{site.version}}.jar) | -| 3.0.0 - 3.1.2 | `flink-connector-hive-3.1.2` | [Download](http://central.maven.org/maven2/org/apache/flink/flink-sql-connector-hive-3.1.2{{site.scala_version_suffix}}/{{site.version}}/flink-sql-connector-hive-3.1.2{{site.scala_version_suffix}}-{{site.version}}.jar) | - -{% else %} - -下表仅适用与稳定版本。 - -{% endif %} +| 1.0.0 - 1.2.2 | `flink-sql-connector-hive-1.2.2` | {% if site.is_stable %}[Download](https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-hive-1.2.2{{site.scala_version_suffix}}/{{site.version}}/flink-sql-connector-hive-1.2.2{{site.scala_version_suffix}}-{{site.version}}.jar) {% else %} Only available for stable releases {% endif %} | +| 2.0.0 - 2.2.0 | `flink-sql-connector-hive-2.2.0` | {% if site.is_stable %}[Download](https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-hive-2.2.0{{site.scala_version_suffix}}/{{site.version}}/flink-sql-connector-hive-2.2.0{{site.scala_version_suffix}}-{{site.version}}.jar) {% else %} Only available for stable releases {% endif %} | +| 2.3.0 - 2.3.6 | `flink-sql-connector-hive-2.3.6` | {% if site.is_stable %}[Download](https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-hive-2.3.6{{site.scala_version_suffix}}/{{site.version}}/flink-sql-connector-hive-2.3.6{{site.scala_version_suffix}}-{{site.version}}.jar) {% else %} Only available for stable releases {% endif %} | +| 3.0.0 - 3.1.2 | `flink-sql-connector-hive-3.1.2` | {% if site.is_stable %}[Download](https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-hive-3.1.2{{site.scala_version_suffix}}/{{site.version}}/flink-sql-connector-hive-3.1.2{{site.scala_version_suffix}}-{{site.version}}.jar) {% else %} Only available for stable releases {% endif %} | #### 用户定义的依赖项 @@ -307,7 +298,7 @@ Apache Hive 是基于 Hadoop 之上构建的, 首先您需要 Hadoop 的依赖@@ -68,38 +69,38 @@ val tableEnv = TableEnvironment.create(settings) // SQL query with a registered table // register a table named "Orders" -tableEnv.sqlUpdate("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)"); +tableEnv.executeSql("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)"); // run a SQL query on the Table and retrieve the result as a new Table val result = tableEnv.sqlQuery( "SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'"); -// SQL update with a registered table +// Execute insert SQL with a registered table // register a TableSink -tableEnv.sqlUpdate("CREATE TABLE RubberOrders(product STRING, amount INT) WITH ('connector.path'='/path/to/file' ...)"); -// run a SQL update query on the Table and emit the result to the TableSink -tableEnv.sqlUpdate( +tableEnv.executeSql("CREATE TABLE RubberOrders(product STRING, amount INT) WITH ('connector.path'='/path/to/file' ...)"); +// run an insert SQL on the Table and emit the result to the TableSink +tableEnv.executeSql( "INSERT INTO RubberOrders SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'") {% endhighlight %}{% highlight java %} -EnvironmentSettings settings = EnvironmentSettings.newInstance().useBlinkPlanner().inBatchMode().build(); +EnvironmentSettings settings = EnvironmentSettings.newInstance().inBatchMode().build(); TableEnvironment tableEnv = TableEnvironment.create(settings); String name = "myhive"; @@ -325,7 +316,7 @@ tableEnv.useCatalog("myhive");{% highlight scala %} -val settings = EnvironmentSettings.newInstance().useBlinkPlanner().inBatchMode().build() +val settings = EnvironmentSettings.newInstance().inBatchMode().build() val tableEnv = TableEnvironment.create(settings) val name = "myhive" @@ -364,4 +355,4 @@ catalogs: ## DML -Flink 支持 DML 写入 Hive 表，请参考[读写 Hive 表]({{ site.baseurl }}/zh/dev/table/hive/read_write_hive.html) +Flink 支持 DML 写入 Hive 表，请参考[读写 Hive 表]({{ site.baseurl }}/zh/dev/table/hive/hive_read_write.html) diff --git a/docs/dev/table/hive/read_write_hive.md b/docs/dev/table/hive/read_write_hive.md deleted file mode 100644 index 8aa83b91a07f55..00000000000000 --- a/docs/dev/table/hive/read_write_hive.md +++ /dev/null @@ -1,196 +0,0 @@ ---- -title: "Reading & Writing Hive Tables" -nav-parent_id: hive_tableapi -nav-pos: 2 ---- - - -Using the `HiveCatalog` and Flink's connector to Hive, Flink can read and write from Hive data as an alternative to Hive's batch engine. -Be sure to follow the instructions to include the correct [dependencies]({{ site.baseurl }}/dev/table/hive/#depedencies) in your application. -And please also note that Hive connector only works with blink planner. - -* This will be replaced by the TOC -{:toc} - -## Reading From Hive - -Assume Hive contains a single table in its `default` database, named people that contains several rows. - -{% highlight bash %} -hive> show databases; -OK -default -Time taken: 0.841 seconds, Fetched: 1 row(s) - -hive> show tables; -OK -Time taken: 0.087 seconds - -hive> CREATE TABLE mytable(name string, value double); -OK -Time taken: 0.127 seconds - -hive> SELECT * FROM mytable; -OK -Tom 4.72 -John 8.0 -Tom 24.2 -Bob 3.14 -Bob 4.72 -Tom 34.9 -Mary 4.79 -Tiff 2.72 -Bill 4.33 -Mary 77.7 -Time taken: 0.097 seconds, Fetched: 10 row(s) -{% endhighlight %} - -With the data ready your can connect to Hive [connect to an existing Hive installation]({{ site.baseurl }}/dev/table/hive/#connecting-to-hive) and begin querying. - -{% highlight bash %} - -Flink SQL> show catalogs; -myhive -default_catalog - -# ------ Set the current catalog to be 'myhive' catalog if you haven't set it in the yaml file ------ - -Flink SQL> use catalog myhive; - -# ------ See all registered database in catalog 'mytable' ------ - -Flink SQL> show databases; -default - -# ------ See the previously registered table 'mytable' ------ - -Flink SQL> show tables; -mytable - -# ------ The table schema that Flink sees is the same that we created in Hive, two columns - name as string and value as double ------ -Flink SQL> describe mytable; -root - |-- name: name - |-- type: STRING - |-- name: value - |-- type: DOUBLE - -# ------ Select from hive table or hive view ------ -Flink SQL> SELECT * FROM mytable; - - name value -__________ __________ - - Tom 4.72 - John 8.0 - Tom 24.2 - Bob 3.14 - Bob 4.72 - Tom 34.9 - Mary 4.79 - Tiff 2.72 - Bill 4.33 - Mary 77.7 - -{% endhighlight %} - -### Querying Hive views - -If you need to query Hive views, please note: - -1. You have to use the Hive catalog as your current catalog before you can query views in that catalog. It can be done by either `tableEnv.useCatalog(...)` in Table API or `USE CATALOG ...` in SQL Client. -2. Hive and Flink SQL have different syntax, e.g. different reserved keywords and literals. Make sure the view's query is compatible with Flink grammar. - -## Writing To Hive - -Similarly, data can be written into hive using an `INSERT` clause. - -Consider there is an example table named "mytable" with two columns: name and age, in string and int type. - -{% highlight bash %} -# ------ INSERT INTO will append to the table or partition, keeping the existing data intact ------ -Flink SQL> INSERT INTO mytable SELECT 'Tom', 25; - -# ------ INSERT OVERWRITE will overwrite any existing data in the table or partition ------ -Flink SQL> INSERT OVERWRITE mytable SELECT 'Tom', 25; -{% endhighlight %} - -We support partitioned table too, Consider there is a partitioned table named myparttable with four columns: name, age, my_type and my_date, in types ...... my_type and my_date are the partition keys. - -{% highlight bash %} -# ------ Insert with static partition ------ -Flink SQL> INSERT OVERWRITE myparttable PARTITION (my_type='type_1', my_date='2019-08-08') SELECT 'Tom', 25; - -# ------ Insert with dynamic partition ------ -Flink SQL> INSERT OVERWRITE myparttable SELECT 'Tom', 25, 'type_1', '2019-08-08'; - -# ------ Insert with static(my_type) and dynamic(my_date) partition ------ -Flink SQL> INSERT OVERWRITE myparttable PARTITION (my_type='type_1') SELECT 'Tom', 25, '2019-08-08'; -{% endhighlight %} - - -## Formats - -We have tested on the following of table storage formats: text, csv, SequenceFile, ORC, and Parquet. - - -## Optimizations - -### Partition Pruning - -Flink uses partition pruning as a performance optimization to limits the number of files and partitions -that Flink reads when querying Hive tables. When your data is partitioned, Flink only reads a subset of the partitions in -a Hive table when a query matches certain filter criteria. - -### Projection Pushdown - -Flink leverages projection pushdown to minimize data transfer between Flink and Hive tables by omitting -unnecessary fields from table scans. - -It is especially beneficial when a table contains many columns. - -### Limit Pushdown - -For queries with LIMIT clause, Flink will limit the number of output records wherever possible to minimize the -amount of data transferred across network. - -### Vectorized Optimization upon Read - -Optimization is used automatically when the following conditions are met: - -- Format: ORC or Parquet. -- Columns without complex data type, like hive types: List, Map, Struct, Union. - -This feature is turned on by default. If there is a problem, you can use this config option to close Vectorized Optimization: - -{% highlight bash %} -table.exec.hive.fallback-mapred-reader=true -{% endhighlight %} - - -## Roadmap - -We are planning and actively working on supporting features like - -- ACID tables -- bucketed tables -- more formats - -Please reach out to the community for more feature request https://flink.apache.org/community.html#mailing-lists diff --git a/docs/dev/table/hive/read_write_hive.zh.md b/docs/dev/table/hive/read_write_hive.zh.md deleted file mode 100644 index 8aa83b91a07f55..00000000000000 --- a/docs/dev/table/hive/read_write_hive.zh.md +++ /dev/null @@ -1,196 +0,0 @@ ---- -title: "Reading & Writing Hive Tables" -nav-parent_id: hive_tableapi -nav-pos: 2 ---- - - -Using the `HiveCatalog` and Flink's connector to Hive, Flink can read and write from Hive data as an alternative to Hive's batch engine. -Be sure to follow the instructions to include the correct [dependencies]({{ site.baseurl }}/dev/table/hive/#depedencies) in your application. -And please also note that Hive connector only works with blink planner. - -* This will be replaced by the TOC -{:toc} - -## Reading From Hive - -Assume Hive contains a single table in its `default` database, named people that contains several rows. - -{% highlight bash %} -hive> show databases; -OK -default -Time taken: 0.841 seconds, Fetched: 1 row(s) - -hive> show tables; -OK -Time taken: 0.087 seconds - -hive> CREATE TABLE mytable(name string, value double); -OK -Time taken: 0.127 seconds - -hive> SELECT * FROM mytable; -OK -Tom 4.72 -John 8.0 -Tom 24.2 -Bob 3.14 -Bob 4.72 -Tom 34.9 -Mary 4.79 -Tiff 2.72 -Bill 4.33 -Mary 77.7 -Time taken: 0.097 seconds, Fetched: 10 row(s) -{% endhighlight %} - -With the data ready your can connect to Hive [connect to an existing Hive installation]({{ site.baseurl }}/dev/table/hive/#connecting-to-hive) and begin querying. - -{% highlight bash %} - -Flink SQL> show catalogs; -myhive -default_catalog - -# ------ Set the current catalog to be 'myhive' catalog if you haven't set it in the yaml file ------ - -Flink SQL> use catalog myhive; - -# ------ See all registered database in catalog 'mytable' ------ - -Flink SQL> show databases; -default - -# ------ See the previously registered table 'mytable' ------ - -Flink SQL> show tables; -mytable - -# ------ The table schema that Flink sees is the same that we created in Hive, two columns - name as string and value as double ------ -Flink SQL> describe mytable; -root - |-- name: name - |-- type: STRING - |-- name: value - |-- type: DOUBLE - -# ------ Select from hive table or hive view ------ -Flink SQL> SELECT * FROM mytable; - - name value -__________ __________ - - Tom 4.72 - John 8.0 - Tom 24.2 - Bob 3.14 - Bob 4.72 - Tom 34.9 - Mary 4.79 - Tiff 2.72 - Bill 4.33 - Mary 77.7 - -{% endhighlight %} - -### Querying Hive views - -If you need to query Hive views, please note: - -1. You have to use the Hive catalog as your current catalog before you can query views in that catalog. It can be done by either `tableEnv.useCatalog(...)` in Table API or `USE CATALOG ...` in SQL Client. -2. Hive and Flink SQL have different syntax, e.g. different reserved keywords and literals. Make sure the view's query is compatible with Flink grammar. - -## Writing To Hive - -Similarly, data can be written into hive using an `INSERT` clause. - -Consider there is an example table named "mytable" with two columns: name and age, in string and int type. - -{% highlight bash %} -# ------ INSERT INTO will append to the table or partition, keeping the existing data intact ------ -Flink SQL> INSERT INTO mytable SELECT 'Tom', 25; - -# ------ INSERT OVERWRITE will overwrite any existing data in the table or partition ------ -Flink SQL> INSERT OVERWRITE mytable SELECT 'Tom', 25; -{% endhighlight %} - -We support partitioned table too, Consider there is a partitioned table named myparttable with four columns: name, age, my_type and my_date, in types ...... my_type and my_date are the partition keys. - -{% highlight bash %} -# ------ Insert with static partition ------ -Flink SQL> INSERT OVERWRITE myparttable PARTITION (my_type='type_1', my_date='2019-08-08') SELECT 'Tom', 25; - -# ------ Insert with dynamic partition ------ -Flink SQL> INSERT OVERWRITE myparttable SELECT 'Tom', 25, 'type_1', '2019-08-08'; - -# ------ Insert with static(my_type) and dynamic(my_date) partition ------ -Flink SQL> INSERT OVERWRITE myparttable PARTITION (my_type='type_1') SELECT 'Tom', 25, '2019-08-08'; -{% endhighlight %} - - -## Formats - -We have tested on the following of table storage formats: text, csv, SequenceFile, ORC, and Parquet. - - -## Optimizations - -### Partition Pruning - -Flink uses partition pruning as a performance optimization to limits the number of files and partitions -that Flink reads when querying Hive tables. When your data is partitioned, Flink only reads a subset of the partitions in -a Hive table when a query matches certain filter criteria. - -### Projection Pushdown - -Flink leverages projection pushdown to minimize data transfer between Flink and Hive tables by omitting -unnecessary fields from table scans. - -It is especially beneficial when a table contains many columns. - -### Limit Pushdown - -For queries with LIMIT clause, Flink will limit the number of output records wherever possible to minimize the -amount of data transferred across network. - -### Vectorized Optimization upon Read - -Optimization is used automatically when the following conditions are met: - -- Format: ORC or Parquet. -- Columns without complex data type, like hive types: List, Map, Struct, Union. - -This feature is turned on by default. If there is a problem, you can use this config option to close Vectorized Optimization: - -{% highlight bash %} -table.exec.hive.fallback-mapred-reader=true -{% endhighlight %} - - -## Roadmap - -We are planning and actively working on supporting features like - -- ACID tables -- bucketed tables -- more formats - -Please reach out to the community for more feature request https://flink.apache.org/community.html#mailing-lists diff --git a/docs/dev/table/hive/scala_shell_hive.md b/docs/dev/table/hive/scala_shell_hive.md deleted file mode 100644 index d3fbd9195abaae..00000000000000 --- a/docs/dev/table/hive/scala_shell_hive.md +++ /dev/null @@ -1,45 +0,0 @@ ---- -title: "Use Hive connector in Scala Shell" -nav-parent_id: hive_tableapi -nav-pos: 3 ---- - - -NOTE: since blink planner is not well supported in Scala Shell at the moment, it's **NOT** recommended to use Hive connector in Scala Shell. - -[Flink Scala Shell]({{ site.baseurl }}/ops/scala_shell.html) is a convenient quick way to try flink. -You can use hive in Scala Shell as well instead of specifying hive dependencies in pom file, packaging your program and submitting it via flink run command. -In order to use hive connector in Scala Shell, you need to put the following [hive connector dependencies]({{ site.baseurl }}/dev/table/hive/#depedencies) under lib folder of flink dist . - -* flink-connector-hive_{scala_version}-{flink.version}.jar -* flink-hadoop-compatibility_{scala_version}-{flink.version}.jar -* flink-shaded-hadoop-2-uber-{hadoop.version}-{flink-shaded.version}.jar -* hive-exec-2.x.jar (for Hive 1.x, you need to copy hive-exec-1.x.jar, hive-metastore-1.x.jar, libfb303-0.9.2.jar and libthrift-0.9.2.jar) - -Then you can use hive connector in Scala Shell like following: - -{% highlight scala %} -Scala-Flink> import org.apache.flink.table.catalog.hive.HiveCatalog -Scala-Flink> val hiveCatalog = new HiveCatalog("hive", "default", "@@ -65,32 +67,36 @@ val settings = EnvironmentSettings.newInstance()... val tableEnv = TableEnvironment.create(settings) // 注册名为 “Orders” 的表 -tableEnv.sqlUpdate("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)"); +tableEnv.executeSql("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)"); // 字符串数组： ["Orders"] -val tables = tableEnv.listTable() +val tables = tableEnv.listTables() +// or tableEnv.executeSql("SHOW TABLES").print() // 把 “Orders” 的表名改为 “NewOrders” -tableEnv.sqlUpdate("ALTER TABLE Orders RENAME TO NewOrders;") +tableEnv.executeSql("ALTER TABLE Orders RENAME TO NewOrders;") // 字符串数组：["NewOrders"] -val tables = tableEnv.listTable() +val tables = tableEnv.listTables() +// or tableEnv.executeSql("SHOW TABLES").print() {% endhighlight %}", "2.3.4"); -Scala-Flink> btenv.registerCatalog("hive", hiveCatalog) -Scala-Flink> btenv.useCatalog("hive") -Scala-Flink> btenv.listTables -Scala-Flink> btenv.sqlQuery(" ").toDataSet[Row].print() -{% endhighlight %} diff --git a/docs/dev/table/hive/scala_shell_hive.zh.md b/docs/dev/table/hive/scala_shell_hive.zh.md deleted file mode 100644 index b9cefc65760b98..00000000000000 --- a/docs/dev/table/hive/scala_shell_hive.zh.md +++ /dev/null @@ -1,45 +0,0 @@ ---- -title: "在 Scala Shell 中使用 Hive 连接器" -nav-parent_id: hive_tableapi -nav-pos: 3 ---- - - -注意：目前 blink planner 还不能很好的支持 Scala Shell，因此 **不** 建议在 Scala Shell 中使用 Hive 连接器。 - -[Flink Scala Shell]({{ site.baseurl }}/zh/ops/scala_shell.html) 是快速上手 Flink 的好方法。 -你可以在 Scala Shell 中直接使用 Hive 连接器，而不需要在 pom 中引入 Hive 相关依赖，并打包提交作业。 -想要在 Scala Shell 中使用 Hive 连接器，你需要把 [Hive 连接器依赖项]({{ site.baseurl }}/zh/dev/table/hive/#depedencies) 放在 Flink dist 包中的 lib 文件夹下。 - -* flink-connector-hive_{scala_version}-{flink.version}.jar -* flink-hadoop-compatibility_{scala_version}-{flink.version}.jar -* flink-shaded-hadoop-2-uber-{hadoop.version}-{flink-shaded.version}.jar -* hive-exec-2.x.jar (对于 Hive 1.x 版本，你需要复制 hive-exec-1.x.jar, hive-metastore-1.x.jar, libfb303-0.9.2.jar and libthrift-0.9.2.jar) - -然后你就可以在 Scala Shell 中使用 Hive 连接器，如下所示： - -{% highlight scala %} -Scala-Flink> import org.apache.flink.table.catalog.hive.HiveCatalog -Scala-Flink> val hiveCatalog = new HiveCatalog("hive", "default", " ", "2.3.4"); -Scala-Flink> btenv.registerCatalog("hive", hiveCatalog) -Scala-Flink> btenv.useCatalog("hive") -Scala-Flink> btenv.listTables -Scala-Flink> btenv.sqlQuery(" ").toDataSet[Row].print() -{% endhighlight %} diff --git a/docs/dev/table/index.md b/docs/dev/table/index.md index 165f7de0125f78..eb6b700529db59 100644 --- a/docs/dev/table/index.md +++ b/docs/dev/table/index.md @@ -38,7 +38,7 @@ Starting from Flink 1.9, Flink provides two different planner implementations fo translating relational operators into an executable, optimized Flink job. Both of the planners come with different optimization rules and runtime classes. They may also differ in the set of supported features. -Attention For production use cases, we recommend the old planner that was present before Flink 1.9 for now. +Attention For production use cases, we recommend the blink planner that has become the default planner since 1.11. All Table API and SQL components are bundled in the `flink-table` or `flink-table-blink` Maven artifacts. @@ -49,8 +49,8 @@ The following dependencies are relevant for most projects: * `flink-table-api-scala`: The Table & SQL API for pure table programs using the Scala programming language (in early development stage, not recommended!). * `flink-table-api-java-bridge`: The Table & SQL API with DataStream/DataSet API support using the Java programming language. * `flink-table-api-scala-bridge`: The Table & SQL API with DataStream/DataSet API support using the Scala programming language. -* `flink-table-planner`: The table program planner and runtime. This was the only planner of Flink before the 1.9 release. It is still the recommended one. -* `flink-table-planner-blink`: The new Blink planner. +* `flink-table-planner`: The table program planner and runtime. This was the only planner of Flink before the 1.9 release. It's no longer recommended since Flink 1.11. +* `flink-table-planner-blink`: The new Blink planner, which has become the default one since Flink 1.11. * `flink-table-runtime-blink`: The new Blink runtime. * `flink-table-uber`: Packages the API modules above plus the old planner into a distribution for most Table & SQL API use cases. The uber JAR file `flink-table-*.jar` is located in the `/lib` directory of a Flink release by default. * `flink-table-uber-blink`: Packages the API modules above plus the Blink specific modules into a distribution for most Table & SQL API use cases. The uber JAR file `flink-table-blink-*.jar` is located in the `/lib` directory of a Flink release by default. diff --git a/docs/dev/table/index.zh.md b/docs/dev/table/index.zh.md index 8613b117830dc0..e7bb0f5dd4e7e3 100644 --- a/docs/dev/table/index.zh.md +++ b/docs/dev/table/index.zh.md @@ -27,7 +27,7 @@ under the License. Apache Flink 有两种关系型 API 来做流批统一处理：Table API 和 SQL。Table API 是用于 Scala 和 Java 语言的查询API，它可以用一种非常直观的方式来组合使用选取、过滤、join 等关系型算子。Flink SQL 是基于 [Apache Calcite](https://calcite.apache.org) 来实现的标准 SQL。这两种 API 中的查询对于批（DataSet）和流（DataStream）的输入有相同的语义，也会产生同样的计算结果。 -Table API 和 SQL 两种 API 是紧密集成的，以及 DataStream 和 DataSet API。你可以在这些 API 之间，以及一些基于这些 API 的库之间轻松的切换。比如，你可以先用 [CEP]({{ site.baseurl }}/zh/dev/libs/cep.html) 从 DataStream 中做模式匹配，然后用 Table API 来分析匹配的结果；或者你可以用 SQL 来扫描、过滤、聚合一个批式的表，然后再跑一个 [Gelly 图算法]({{ site.baseurl }}/zh/dev/libs/gelly) 来处理已经预处理好的数据。 +Table API 和 SQL 两种 API 是紧密集成的，以及 DataStream 和 DataSet API。你可以在这些 API 之间，以及一些基于这些 API 的库之间轻松的切换。比如，你可以先用 [CEP]({{ site.baseurl }}/zh/dev/libs/cep.html) 从 DataStream 中做模式匹配，然后用 Table API 来分析匹配的结果；或者你可以用 SQL 来扫描、过滤、聚合一个批式的表，然后再跑一个 [Gelly 图算法]({{ site.baseurl }}/zh/dev/libs/gelly/index.html) 来处理已经预处理好的数据。 **注意：Table API 和 SQL 现在还处于活跃开发阶段，还没有完全实现所有的特性。不是所有的 \[Table API，SQL\] 和 \[流，批\] 的组合都是支持的。** @@ -38,7 +38,7 @@ Table API 和 SQL 两种 API 是紧密集成的，以及 DataStream 和 DataSet Planner 的作用主要是把关系型的操作翻译成可执行的、经过优化的 Flink 任务。两种 Planner 所使用的优化规则以及运行时类都不一样。 它们在支持的功能上也有些差异。 -注意 对于生产环境，我们建议使用在1.9之前就已经存在的 Old Planner。 +注意 对于生产环境，我们建议使用在1.11版本之后已经变成默认的Blink Planner。 所有的 Table API 和 SQL 的代码都在 `flink-table` 或者 `flink-table-blink` Maven artifacts 下。 @@ -49,8 +49,8 @@ Planner 的作用主要是把关系型的操作翻译成可执行的、经过优 * `flink-table-api-scala`: Table 和 SQL API，使用 Scala 语言编写的，给纯 table 程序使用（还在早期开发阶段，不建议使用） * `flink-table-api-java-bridge`: Table 和 SQL API 结合 DataStream/DataSet API 一起使用，给 Java 语言使用。 * `flink-table-api-scala-bridge`: Table 和 SQL API 结合 DataStream/DataSet API 一起使用，给 Scala 语言使用。 -* `flink-table-planner`: table Planner 和运行时。这是在1.9之前 Flink 的唯一的 Planner，现在仍然建议使用这个。 -* `flink-table-planner-blink`: 新的 Blink Planner。 +* `flink-table-planner`: table Planner 和运行时。这是在1.9之前 Flink 的唯一的 Planner，但是从1.11版本开始我们不推荐继续使用。 +* `flink-table-planner-blink`: 新的 Blink Planner，从1.11版本开始成为默认的 Planner。 * `flink-table-runtime-blink`: 新的 Blink 运行时。 * `flink-table-uber`: 把上述模块以及 Old Planner 打包到一起，可以在大部分 Table & SQL API 场景下使用。打包到一起的 jar 文件 `flink-table-*.jar` 默认会直接放到 Flink 发行版的 `/lib` 目录下。 * `flink-table-uber-blink`: 把上述模块以及 Blink Planner 打包到一起，可以在大部分 Table & SQL API 场景下使用。打包到一起的 jar 文件 `flink-table-blink-*.jar` 默认会放到 Flink 发行版的 `/lib` 目录下。 diff --git a/docs/dev/table/legacySourceSinks.md b/docs/dev/table/legacySourceSinks.md new file mode 100644 index 00000000000000..3d0d0a8b6c5026 --- /dev/null +++ b/docs/dev/table/legacySourceSinks.md @@ -0,0 +1,815 @@ +--- +title: "Legacy Table Source & Sink Interfaces" +--- + + +A `TableSource` provides access to data which is stored in external systems (database, key-value store, message queue) or files. After a [TableSource is registered in a TableEnvironment]({% link dev/table/common.md %}#register-a-tablesource) it can be accessed by [Table API]({% link dev/table/tableApi.md %}) or [SQL]({% link dev/table/sql/queries.md %}) queries. + +A `TableSink` [emits a Table]({% link dev/table/common.md %}#emit-a-table) to an external storage system, such as a database, key-value store, message queue, or file system (in different encodings, e.g., CSV, Parquet, or ORC). + +A `TableFactory` allows for separating the declaration of a connection to an external system from the actual implementation. A table factory creates configured instances of table sources and sinks from normalized, string-based properties. The properties can be generated programmatically using a `Descriptor` or via YAML configuration files for the [SQL Client]({% link dev/table/sqlClient.md %}). + +Have a look at the [common concepts and API]({% link dev/table/common.md %}) page for details how to [register a TableSource]({% link dev/table/common.md %}#register-a-tablesource) and how to [emit a Table through a TableSink]({% link dev/table/common.md %}#emit-a-table). See the [built-in sources, sinks, and formats]({% link dev/table/connect.md %}) page for examples how to use factories. + +* This will be replaced by the TOC +{:toc} + +Define a TableSource +-------------------- + +A `TableSource` is a generic interface that gives Table API and SQL queries access to data stored in an external system. It provides the schema of the table and the records that are mapped to rows with the table's schema. Depending on whether the `TableSource` is used in a streaming or batch query, the records are produced as a `DataSet` or `DataStream`. + +If a `TableSource` is used in a streaming query it must implement the `StreamTableSource` interface, if it is used in a batch query it must implement the `BatchTableSource` interface. A `TableSource` can also implement both interfaces and be used in streaming and batch queries. + +`StreamTableSource` and `BatchTableSource` extend the base interface `TableSource` that defines the following methods: + + ++ +* `getTableSchema()`: Returns the schema of the produced table, i.e., the names and types of the fields of the table. The field types are defined using Flink's `DataType` (see [Table API types]({% link dev/table/types.md %}) and [SQL types]({% link dev/table/sql/index.md %}#data-types)). Note that the returned `TableSchema` shouldn't contain computed columns to reflect the schema of the physical `TableSource`. + +* `getReturnType()`: Returns the physical type of the `DataStream` (`StreamTableSource`) or `DataSet` (`BatchTableSource`) and the records that are produced by the `TableSource`. + +* `explainSource()`: Returns a String that describes the `TableSource`. This method is optional and used for display purposes only. + +The `TableSource` interface separates the logical table schema from the physical type of the returned `DataStream` or `DataSet`. As a consequence, all fields of the table schema (`getTableSchema()`) must be mapped to a field with corresponding type of the physical return type (`getReturnType()`). By default, this mapping is done based on field names. For example, a `TableSource` that defines a table schema with two fields `[name: String, size: Integer]` requires a `TypeInformation` with at least two fields called `name` and `size` of type `String` and `Integer`, respectively. This could be a `PojoTypeInfo` or a `RowTypeInfo` that have two fields named `name` and `size` with matching types. + +However, some types, such as Tuple or CaseClass types, do support custom field names. If a `TableSource` returns a `DataStream` or `DataSet` of a type with fixed field names, it can implement the `DefinedFieldMapping` interface to map field names from the table schema to field names of the physical return type. + +### Defining a BatchTableSource + +The `BatchTableSource` interface extends the `TableSource` interface and defines one additional method: + ++{% highlight java %} +TableSource+ +{ + + public TableSchema getTableSchema(); + + public TypeInformation getReturnType(); + + public String explainSource(); +} +{% endhighlight %} + +{% highlight scala %} +TableSource[T] { + + def getTableSchema: TableSchema + + def getReturnType: TypeInformation[T] + + def explainSource: String + +} +{% endhighlight %} ++++ +* `getDataSet(execEnv)`: Returns a `DataSet` with the data of the table. The type of the `DataSet` must be identical to the return type defined by the `TableSource.getReturnType()` method. The `DataSet` can by created using a regular [data source]({% link dev/batch/index.md %}#data-sources) of the DataSet API. Commonly, a `BatchTableSource` is implemented by wrapping a `InputFormat` or [batch connector]({% link dev/batch/connectors.md %}). + +{% top %} + +### Defining a StreamTableSource + +The `StreamTableSource` interface extends the `TableSource` interface and defines one additional method: + ++{% highlight java %} +BatchTableSource+ +implements TableSource { + + public DataSet getDataSet(ExecutionEnvironment execEnv); +} +{% endhighlight %} + +{% highlight scala %} +BatchTableSource[T] extends TableSource[T] { + + def getDataSet(execEnv: ExecutionEnvironment): DataSet[T] +} +{% endhighlight %} ++++ +* `getDataStream(execEnv)`: Returns a `DataStream` with the data of the table. The type of the `DataStream` must be identical to the return type defined by the `TableSource.getReturnType()` method. The `DataStream` can by created using a regular [data source]({% link dev/datastream_api.md %}#data-sources) of the DataStream API. Commonly, a `StreamTableSource` is implemented by wrapping a `SourceFunction` or a [stream connector]({% link dev/connectors/index.md %}). + +{% top %} + +### Defining a TableSource with Time Attributes + +Time-based operations of streaming [Table API]({% link dev/table/tableApi.md %}#group-windows) and [SQL]({% link dev/table/sql/queries.md %}#group-windows) queries, such as windowed aggregations or joins, require explicitly specified [time attributes]({% link dev/table/streaming/time_attributes.md %}). + +A `TableSource` defines a time attribute as a field of type `Types.SQL_TIMESTAMP` in its table schema. In contrast to all regular fields in the schema, a time attribute must not be matched to a physical field in the return type of the table source. Instead, a `TableSource` defines a time attribute by implementing a certain interface. + +#### Defining a Processing Time Attribute + +[Processing time attributes]({% link dev/table/streaming/time_attributes.md %}#processing-time) are commonly used in streaming queries. A processing time attribute returns the current wall-clock time of the operator that accesses it. A `TableSource` defines a processing time attribute by implementing the `DefinedProctimeAttribute` interface. The interface looks as follows: + ++{% highlight java %} +StreamTableSource+ +implements TableSource { + + public DataStream getDataStream(StreamExecutionEnvironment execEnv); +} +{% endhighlight %} + +{% highlight scala %} +StreamTableSource[T] extends TableSource[T] { + + def getDataStream(execEnv: StreamExecutionEnvironment): DataStream[T] +} +{% endhighlight %} ++++ +* `getProctimeAttribute()`: Returns the name of the processing time attribute. The specified attribute must be defined of type `Types.SQL_TIMESTAMP` in the table schema and can be used in time-based operations. A `DefinedProctimeAttribute` table source can define no processing time attribute by returning `null`. + +Attention Both `StreamTableSource` and `BatchTableSource` can implement `DefinedProctimeAttribute` and define a processing time attribute. In case of a `BatchTableSource` the processing time field is initialized with the current timestamp during the table scan. + +#### Defining a Rowtime Attribute + +[Rowtime attributes]({% link dev/table/streaming/time_attributes.md %}#event-time) are attributes of type `TIMESTAMP` and handled in a unified way in stream and batch queries. + +A table schema field of type `SQL_TIMESTAMP` can be declared as rowtime attribute by specifying + +* the name of the field, +* a `TimestampExtractor` that computes the actual value for the attribute (usually from one or more other fields), and +* a `WatermarkStrategy` that specifies how watermarks are generated for the the rowtime attribute. + +A `TableSource` defines a rowtime attribute by implementing the `DefinedRowtimeAttributes` interface. The interface looks as follows: + ++{% highlight java %} +DefinedProctimeAttribute { + + public String getProctimeAttribute(); +} +{% endhighlight %} ++ ++{% highlight scala %} +DefinedProctimeAttribute { + + def getProctimeAttribute: String +} +{% endhighlight %} ++++ +* `getRowtimeAttributeDescriptors()`: Returns a list of `RowtimeAttributeDescriptor`. A `RowtimeAttributeDescriptor` describes a rowtime attribute with the following properties: + * `attributeName`: The name of the rowtime attribute in the table schema. The field must be defined with type `Types.SQL_TIMESTAMP`. + * `timestampExtractor`: The timestamp extractor extracts the timestamp from a record with the return type. For example, it can convert a Long field into a timestamp or parse a String-encoded timestamp. Flink comes with a set of built-in `TimestampExtractor` implementation for common use cases. It is also possible to provide a custom implementation. + * `watermarkStrategy`: The watermark strategy defines how watermarks are generated for the rowtime attribute. Flink comes with a set of built-in `WatermarkStrategy` implementations for common use cases. It is also possible to provide a custom implementation. + +Attention Although the `getRowtimeAttributeDescriptors()` method returns a list of descriptors, only a single rowtime attribute is support at the moment. We plan to remove this restriction in the future and support tables with more than one rowtime attribute. + +Attention Both, `StreamTableSource` and `BatchTableSource`, can implement `DefinedRowtimeAttributes` and define a rowtime attribute. In either case, the rowtime field is extracted using the `TimestampExtractor`. Hence, a `TableSource` that implements `StreamTableSource` and `BatchTableSource` and defines a rowtime attribute provides exactly the same data to streaming and batch queries. + +##### Provided Timestamp Extractors + +Flink provides `TimestampExtractor` implementations for common use cases. + +The following `TimestampExtractor` implementations are currently available: + +* `ExistingField(fieldName)`: Extracts the value of a rowtime attribute from an existing `LONG`, `SQL_TIMESTAMP`, or timestamp formatted `STRING` field. One example of such a string would be '2018-05-28 12:34:56.000'. +* `StreamRecordTimestamp()`: Extracts the value of a rowtime attribute from the timestamp of the `DataStream` `StreamRecord`. Note, this `TimestampExtractor` is not available for batch table sources. + +A custom `TimestampExtractor` can be defined by implementing the corresponding interface. + +##### Provided Watermark Strategies + +Flink provides `WatermarkStrategy` implementations for common use cases. + +The following `WatermarkStrategy` implementations are currently available: + +* `AscendingTimestamps`: A watermark strategy for ascending timestamps. Records with timestamps that are out-of-order will be considered late. +* `BoundedOutOfOrderTimestamps(delay)`: A watermark strategy for timestamps that are at most out-of-order by the specified delay. +* `PreserveWatermarks()`: A strategy which indicates the watermarks should be preserved from the underlying `DataStream`. + +A custom `WatermarkStrategy` can be defined by implementing the corresponding interface. + +{% top %} + +### Defining a TableSource with Projection Push-Down + +A `TableSource` supports projection push-down by implementing the `ProjectableTableSource` interface. The interface defines a single method: + ++{% highlight java %} +DefinedRowtimeAttribute { + + public List+ +getRowtimeAttributeDescriptors(); +} +{% endhighlight %} + +{% highlight scala %} +DefinedRowtimeAttributes { + + def getRowtimeAttributeDescriptors: util.List[RowtimeAttributeDescriptor] +} +{% endhighlight %} ++++ +* `projectFields(fields)`: Returns a *copy* of the `TableSource` with adjusted physical return type. The `fields` parameter provides the indexes of the fields that must be provided by the `TableSource`. The indexes relate to the `TypeInformation` of the physical return type, *not* to the logical table schema. The copied `TableSource` must adjust its return type and the returned `DataStream` or `DataSet`. The `TableSchema` of the copied `TableSource` must not be changed, i.e, it must be the same as the original `TableSource`. If the `TableSource` implements the `DefinedFieldMapping` interface, the field mapping must be adjusted to the new return type. + +Attention In order for Flink to distinguish a projection push-down table source from its original form, `explainSource` method must be override to include information regarding the projected fields. + +The `ProjectableTableSource` adds support to project flat fields. If the `TableSource` defines a table with nested schema, it can implement the `NestedFieldsProjectableTableSource` to extend the projection to nested fields. The `NestedFieldsProjectableTableSource` is defined as follows: + ++{% highlight java %} +ProjectableTableSource+ +{ + + public TableSource projectFields(int[] fields); +} +{% endhighlight %} + +{% highlight scala %} +ProjectableTableSource[T] { + + def projectFields(fields: Array[Int]): TableSource[T] +} +{% endhighlight %} ++++ +* `projectNestedField(fields, nestedFields)`: Returns a *copy* of the `TableSource` with adjusted physical return type. Fields of the physical return type may be removed or reordered but their type must not be changed. The contract of this method is essentially the same as for the `ProjectableTableSource.projectFields()` method. In addition, the `nestedFields` parameter contains for each field index in the `fields` list, a list of paths to all nested fields that are accessed by the query. All other nested fields do not need to be read, parsed, and set in the records that are produced by the `TableSource`. + +Attention the types of the projected fields must not be changed but unused fields may be set to null or to a default value. + +{% top %} + +### Defining a TableSource with Filter Push-Down + +The `FilterableTableSource` interface adds support for filter push-down to a `TableSource`. A `TableSource` extending this interface is able to filter records such that the returned `DataStream` or `DataSet` returns fewer records. + +The interface looks as follows: + ++{% highlight java %} +NestedFieldsProjectableTableSource+ +{ + + public TableSource projectNestedFields(int[] fields, String[][] nestedFields); +} +{% endhighlight %} + +{% highlight scala %} +NestedFieldsProjectableTableSource[T] { + + def projectNestedFields(fields: Array[Int], nestedFields: Array[Array[String]]): TableSource[T] +} +{% endhighlight %} ++++ +* `applyPredicate(predicates)`: Returns a *copy* of the `TableSource` with added predicates. The `predicates` parameter is a mutable list of conjunctive predicates that are "offered" to the `TableSource`. The `TableSource` accepts to evaluate a predicate by removing it from the list. Predicates that are left in the list will be evaluated by a subsequent filter operator. +* `isFilterPushedDown()`: Returns true if the `applyPredicate()` method was called before. Hence, `isFilterPushedDown()` must return true for all `TableSource` instances returned from a `applyPredicate()` call. + +Attention In order for Flink to distinguish a filter push-down table source from its original form, `explainSource` method must be override to include information regarding the push-down filters. + +{% top %} + +### Defining a TableSource for Lookups + +Attention This is an experimental feature. The interface may be changed in future versions. It's only supported in Blink planner. + +The `LookupableTableSource` interface adds support for the table to be accessed via key column(s) in a lookup fashion. This is very useful when used to join with a dimension table to enrich some information. If you want to use the `TableSource` in lookup mode, you should use the source in [temporal table join syntax]({% link dev/table/streaming/joins.md %}). + +The interface looks as follows: + ++{% highlight java %} +FilterableTableSource+ +{ + + public TableSource applyPredicate(List predicates); + + public boolean isFilterPushedDown(); +} +{% endhighlight %} + +{% highlight scala %} +FilterableTableSource[T] { + + def applyPredicate(predicates: java.util.List[Expression]): TableSource[T] + + def isFilterPushedDown: Boolean +} +{% endhighlight %} ++++ +* `getLookupFunction(lookupkeys)`: Returns a `TableFunction` which used to lookup the matched row(s) via lookup keys. The lookupkeys are the field names of `LookupableTableSource` in the join equal conditions. The eval method parameters of the returned `TableFunction`'s should be in the order which `lookupkeys` defined. It is recommended to define the parameters in varargs (e.g. `eval(Object... lookupkeys)` to match all the cases). The return type of the `TableFunction` must be identical to the return type defined by the `TableSource.getReturnType()` method. +* `getAsyncLookupFunction(lookupkeys)`: Optional. Similar to `getLookupFunction`, but the `AsyncLookupFunction` lookups the matched row(s) asynchronously. The underlying of `AsyncLookupFunction` will be called via [Async I/O]({% link dev/stream/operators/asyncio.md %}). The first argument of the eval method of the returned `AsyncTableFunction` should be defined as `java.util.concurrent.CompletableFuture` to collect results asynchronously (e.g. `eval(CompletableFuture+{% highlight java %} +LookupableTableSource+ +implements TableSource { + + public TableFunction getLookupFunction(String[] lookupkeys); + + public AsyncTableFunction getAsyncLookupFunction(String[] lookupkeys); + + public boolean isAsyncEnabled(); +} +{% endhighlight %} + +{% highlight scala %} +LookupableTableSource[T] extends TableSource[T] { + + def getLookupFunction(lookupKeys: Array[String]): TableFunction[T] + + def getAsyncLookupFunction(lookupKeys: Array[String]): AsyncTableFunction[T] + + def isAsyncEnabled: Boolean +} +{% endhighlight %} ++++ +The `TableSink#configure` method is called to pass the schema of the Table (field names and types) to emit to the `TableSink`. The method must return a new instance of the TableSink which is configured to emit the provided Table schema. Note that the provided `TableSchema` shouldn't contain computed columns to reflect the schema of the physical `TableSink`. + +### BatchTableSink + +Defines an external `TableSink` to emit a batch table. + +The interface looks as follows: + ++{% highlight java %} +TableSink+ +{ + + public TypeInformation getOutputType(); + + public String[] getFieldNames(); + + public TypeInformation[] getFieldTypes(); + + public TableSink configure(String[] fieldNames, TypeInformation[] fieldTypes); +} +{% endhighlight %} + +{% highlight scala %} +TableSink[T] { + + def getOutputType: TypeInformation++ + def getFieldNames: Array[String] + + def getFieldTypes: Array[TypeInformation] + + def configure(fieldNames: Array[String], fieldTypes: Array[TypeInformation]): TableSink[T] +} +{% endhighlight %} + ++ +{% top %} + +### AppendStreamTableSink + +Defines an external `TableSink` to emit a streaming table with only insert changes. + +The interface looks as follows: + ++{% highlight java %} +BatchTableSink+ +implements TableSink { + + public void emitDataSet(DataSet dataSet); +} +{% endhighlight %} + +{% highlight scala %} +BatchTableSink[T] extends TableSink[T] { + + def emitDataSet(dataSet: DataSet[T]): Unit +} +{% endhighlight %} ++++ +If the table is also modified by update or delete changes, a `TableException` will be thrown. + +{% top %} + +### RetractStreamTableSink + +Defines an external `TableSink` to emit a streaming table with insert, update, and delete changes. + +The interface looks as follows: + ++{% highlight java %} +AppendStreamTableSink+ +implements TableSink { + + public DataStreamSink consumeDataStream(DataStream dataStream); +} +{% endhighlight %} + +{% highlight scala %} +AppendStreamTableSink[T] extends TableSink[T] { + + def consumeDataStream(dataStream: DataStream[T]): DataStreamSink[_] +} +{% endhighlight %} ++++ +The table will be converted into a stream of accumulate and retraction messages which are encoded as Java `Tuple2`. The first field is a boolean flag to indicate the message type (`true` indicates insert, `false` indicates delete). The second field holds the record of the requested type `T`. + +{% top %} + +### UpsertStreamTableSink + +Defines an external `TableSink` to emit a streaming table with insert, update, and delete changes. + +The interface looks as follows: + ++{% highlight java %} +RetractStreamTableSink+ +implements TableSink getRecordType(); + + public DataStreamSink consumeDataStream(DataStream +{% highlight scala %} +RetractStreamTableSink[T] extends TableSink[Tuple2[Boolean, T]] { + + def getRecordType: TypeInformation[T] + + def consumeDataStream(dataStream: DataStream[Tuple2[Boolean, T]]): DataStreamSink[_] +} +{% endhighlight %} ++++ +The table must be have unique key fields (atomic or composite) or be append-only. If the table does not have a unique key and is not append-only, a `TableException` will be thrown. The unique key of the table is configured by the `UpsertStreamTableSink#setKeyFields()` method. + +The table will be converted into a stream of upsert and delete messages which are encoded as a Java `Tuple2`. The first field is a boolean flag to indicate the message type. The second field holds the record of the requested type `T`. + +A message with true boolean field is an upsert message for the configured key. A message with false flag is a delete message for the configured key. If the table is append-only, all messages will have a true flag and must be interpreted as insertions. + +{% top %} + +Define a TableFactory +--------------------- + +A `TableFactory` allows to create different table-related instances from string-based properties. All available factories are called for matching to the given set of properties and a corresponding factory class. + +Factories leverage Java's [Service Provider Interfaces (SPI)](https://docs.oracle.com/javase/tutorial/sound/SPI-intro.html) for discovering. This means that every dependency and JAR file should contain a file `org.apache.flink.table.factories.TableFactory` in the `META_INF/services` resource directory that lists all available table factories that it provides. + +Every table factory needs to implement the following interface: + ++{% highlight java %} +UpsertStreamTableSink+ +implements TableSink getRecordType(); + + public DataStreamSink consumeDataStream(DataStream +{% highlight scala %} +UpsertStreamTableSink[T] extends TableSink[Tuple2[Boolean, T]] { + + def setKeyFields(keys: Array[String]): Unit + + def setIsAppendOnly(isAppendOnly: Boolean): Unit + + def getRecordType: TypeInformation[T] + + def consumeDataStream(dataStream: DataStream[Tuple2[Boolean, T]]): DataStreamSink[_] +} +{% endhighlight %} ++++ +* `requiredContext()`: Specifies the context that this factory has been implemented for. The framework guarantees to only match for this factory if the specified set of properties and values are met. Typical properties might be `connector.type`, `format.type`, or `update-mode`. Property keys such as `connector.property-version` and `format.property-version` are reserved for future backwards compatibility cases. +* `supportedProperties()`: List of property keys that this factory can handle. This method will be used for validation. If a property is passed that this factory cannot handle, an exception will be thrown. The list must not contain the keys that are specified by the context. + +In order to create a specific instance, a factory class can implement one or more interfaces provided in `org.apache.flink.table.factories`: + +* `BatchTableSourceFactory`: Creates a batch table source. +* `BatchTableSinkFactory`: Creates a batch table sink. +* `StreamTableSourceFactory`: Creates a stream table source. +* `StreamTableSinkFactory`: Creates a stream table sink. +* `DeserializationSchemaFactory`: Creates a deserialization schema format. +* `SerializationSchemaFactory`: Creates a serialization schema format. + +The discovery of a factory happens in multiple stages: + +- Discover all available factories. +- Filter by factory class (e.g., `StreamTableSourceFactory`). +- Filter by matching context. +- Filter by supported properties. +- Verify that exactly one factory matches, otherwise throw an `AmbiguousTableFactoryException` or `NoMatchingTableFactoryException`. + +The following example shows how to provide a custom streaming source with an additional `connector.debug` property flag for parameterization. + ++{% highlight java %} +package org.apache.flink.table.factories; + +interface TableFactory { + + Map+ +supportedProperties(); +} +{% endhighlight %} + +{% highlight scala %} +package org.apache.flink.table.factories + +trait TableFactory { + + def requiredContext(): util.Map[String, String] + + def supportedProperties(): util.List[String] +} +{% endhighlight %} ++++ +{% top %} + +### Use a TableFactory in the SQL Client + +In a SQL Client environment file, the previously presented factory could be declared as: + +{% highlight yaml %} +tables: + - name: MySystemTable + type: source + update-mode: append + connector: + type: my-system + debug: true +{% endhighlight %} + +The YAML file is translated into flattened string properties and a table factory is called with those properties that describe the connection to the external system: + +{% highlight text %} +update-mode=append +connector.type=my-system +connector.debug=true +{% endhighlight %} + +Attention Properties such as `tables.#.name` or `tables.#.type` are SQL Client specifics and are not passed to any factory. The `type` property decides, depending on the execution environment, whether a `BatchTableSourceFactory`/`StreamTableSourceFactory` (for `source`), a `BatchTableSinkFactory`/`StreamTableSinkFactory` (for `sink`), or both (for `both`) need to discovered. + +{% top %} + +### Use a TableFactory in the Table & SQL API + +For a type-safe, programmatic approach with explanatory Scaladoc/Javadoc, the Table & SQL API offers descriptors in `org.apache.flink.table.descriptors` that translate into string-based properties. See the [built-in descriptors]({% link dev/table/connect.md %}) for sources, sinks, and formats as a reference. + ++{% highlight java %} +import org.apache.flink.table.sources.StreamTableSource; +import org.apache.flink.types.Row; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +class MySystemTableSourceFactory implements StreamTableSourceFactory+ +{ + + @Override + public Map
supportedProperties() { + List list = new ArrayList<>(); + list.add("connector.debug"); + return list; + } + + @Override + public StreamTableSource createStreamTableSource(Map
+{% highlight scala %} +import java.util +import org.apache.flink.table.sources.StreamTableSource +import org.apache.flink.types.Row + +class MySystemTableSourceFactory extends StreamTableSourceFactory[Row] { + + override def requiredContext(): util.Map[String, String] = { + val context = new util.HashMap[String, String]() + context.put("update-mode", "append") + context.put("connector.type", "my-system") + context + } + + override def supportedProperties(): util.List[String] = { + val properties = new util.ArrayList[String]() + properties.add("connector.debug") + properties + } + + override def createStreamTableSource(properties: util.Map[String, String]): StreamTableSource[Row] = { + val isDebug = java.lang.Boolean.valueOf(properties.get("connector.debug")) + + # additional validation of the passed properties can also happen here + + new MySystemAppendTableSource(isDebug) + } +} +{% endhighlight %} ++++ +{% top %} diff --git a/docs/dev/table/legacySourceSinks.zh.md b/docs/dev/table/legacySourceSinks.zh.md new file mode 100644 index 00000000000000..a410594654fbc6 --- /dev/null +++ b/docs/dev/table/legacySourceSinks.zh.md @@ -0,0 +1,815 @@ +--- +title: "Legacy Table Source & Sink Interfaces" +--- + + +A `TableSource` provides access to data which is stored in external systems (database, key-value store, message queue) or files. After a [TableSource is registered in a TableEnvironment]({% link dev/table/common.zh.md %}#register-a-tablesource) it can be accessed by [Table API]({% link dev/table/tableApi.zh.md %}) or [SQL]({% link dev/table/sql/queries.zh.md %}) queries. + +A `TableSink` [emits a Table]({% link dev/table/common.zh.md %}#emit-a-table) to an external storage system, such as a database, key-value store, message queue, or file system (in different encodings, e.g., CSV, Parquet, or ORC). + +A `TableFactory` allows for separating the declaration of a connection to an external system from the actual implementation. A table factory creates configured instances of table sources and sinks from normalized, string-based properties. The properties can be generated programmatically using a `Descriptor` or via YAML configuration files for the [SQL Client]({% link dev/table/sqlClient.zh.md %}). + +Have a look at the [common concepts and API]({% link dev/table/common.zh.md %}) page for details how to [register a TableSource]({% link dev/table/common.zh.md %}#register-a-tablesource) and how to [emit a Table through a TableSink]({% link dev/table/common.zh.md %}#emit-a-table). See the [built-in sources, sinks, and formats]({% link dev/table/connect.zh.md %}) page for examples how to use factories. + +* This will be replaced by the TOC +{:toc} + +Define a TableSource +-------------------- + +A `TableSource` is a generic interface that gives Table API and SQL queries access to data stored in an external system. It provides the schema of the table and the records that are mapped to rows with the table's schema. Depending on whether the `TableSource` is used in a streaming or batch query, the records are produced as a `DataSet` or `DataStream`. + +If a `TableSource` is used in a streaming query it must implement the `StreamTableSource` interface, if it is used in a batch query it must implement the `BatchTableSource` interface. A `TableSource` can also implement both interfaces and be used in streaming and batch queries. + +`StreamTableSource` and `BatchTableSource` extend the base interface `TableSource` that defines the following methods: + ++ +A custom descriptor can be defined by extending the `ConnectorDescriptor` class. + +{% highlight java %} +import org.apache.flink.table.descriptors.ConnectorDescriptor; +import java.util.HashMap; +import java.util.Map; + +/** + * Connector to MySystem with debug mode. + */ +public class MySystemConnector extends ConnectorDescriptor { + + public final boolean isDebug; + + public MySystemConnector(boolean isDebug) { + super("my-system", 1, false); + this.isDebug = isDebug; + } + + @Override + protected Map+ ++ +A custom descriptor can be defined by extending the `ConnectorDescriptor` class. + +{% highlight scala %} +import org.apache.flink.table.descriptors.ConnectorDescriptor +import java.util.HashMap +import java.util.Map + +/** + * Connector to MySystem with debug mode. + */ +class MySystemConnector(isDebug: Boolean) extends ConnectorDescriptor("my-system", 1, false) { + + override protected def toConnectorProperties(): Map[String, String] = { + val properties = new HashMap[String, String] + properties.put("connector.debug", isDebug.toString) + properties + } +} +{% endhighlight %} + +The descriptor can then be used to create a table with the table environment. + +{% highlight scala %} +val tableEnv: StreamTableEnvironment = // ... + +tableEnv + .connect(new MySystemConnector(isDebug = true)) + .withSchema(...) + .inAppendMode() + .createTemporaryTable("MySystemTable") +{% endhighlight %} + ++ ++ +You can use a Java `TableFactory` from Python using the `CustomConnectorDescriptor`. + +{% highlight python %} +s_env = StreamExecutionEnvironment.get_execution_environment() +st_env = StreamTableEnvironment.create(s_env) + +custom_connector = CustomConnectorDescriptor('my-system', 1, False) +st_env\ + .connect(custom_connector.property("connector.debug", "true")) \ + .with_schema(...) \ + .in_append_mode()\ + .create_temporary_table("MySystemTable") +{% endhighlight %} ++ +++ +* `getTableSchema()`: Returns the schema of the produced table, i.e., the names and types of the fields of the table. The field types are defined using Flink's `DataType` (see [Table API types]({% link dev/table/types.zh.md %}) and [SQL types]({% link dev/table/sql/index.zh.md %}#data-types)). Note that the returned `TableSchema` shouldn't contain computed columns to reflect the schema of the physical `TableSource`. + +* `getReturnType()`: Returns the physical type of the `DataStream` (`StreamTableSource`) or `DataSet` (`BatchTableSource`) and the records that are produced by the `TableSource`. + +* `explainSource()`: Returns a String that describes the `TableSource`. This method is optional and used for display purposes only. + +The `TableSource` interface separates the logical table schema from the physical type of the returned `DataStream` or `DataSet`. As a consequence, all fields of the table schema (`getTableSchema()`) must be mapped to a field with corresponding type of the physical return type (`getReturnType()`). By default, this mapping is done based on field names. For example, a `TableSource` that defines a table schema with two fields `[name: String, size: Integer]` requires a `TypeInformation` with at least two fields called `name` and `size` of type `String` and `Integer`, respectively. This could be a `PojoTypeInfo` or a `RowTypeInfo` that have two fields named `name` and `size` with matching types. + +However, some types, such as Tuple or CaseClass types, do support custom field names. If a `TableSource` returns a `DataStream` or `DataSet` of a type with fixed field names, it can implement the `DefinedFieldMapping` interface to map field names from the table schema to field names of the physical return type. + +### Defining a BatchTableSource + +The `BatchTableSource` interface extends the `TableSource` interface and defines one additional method: + ++{% highlight java %} +TableSource+ +{ + + public TableSchema getTableSchema(); + + public TypeInformation getReturnType(); + + public String explainSource(); +} +{% endhighlight %} + +{% highlight scala %} +TableSource[T] { + + def getTableSchema: TableSchema + + def getReturnType: TypeInformation[T] + + def explainSource: String + +} +{% endhighlight %} ++++ +* `getDataSet(execEnv)`: Returns a `DataSet` with the data of the table. The type of the `DataSet` must be identical to the return type defined by the `TableSource.getReturnType()` method. The `DataSet` can by created using a regular [data source]({% link dev/batch/index.zh.md %}#data-sources) of the DataSet API. Commonly, a `BatchTableSource` is implemented by wrapping a `InputFormat` or [batch connector]({% link dev/batch/connectors.zh.md %}). + +{% top %} + +### Defining a StreamTableSource + +The `StreamTableSource` interface extends the `TableSource` interface and defines one additional method: + ++{% highlight java %} +BatchTableSource+ +implements TableSource { + + public DataSet getDataSet(ExecutionEnvironment execEnv); +} +{% endhighlight %} + +{% highlight scala %} +BatchTableSource[T] extends TableSource[T] { + + def getDataSet(execEnv: ExecutionEnvironment): DataSet[T] +} +{% endhighlight %} ++++ +* `getDataStream(execEnv)`: Returns a `DataStream` with the data of the table. The type of the `DataStream` must be identical to the return type defined by the `TableSource.getReturnType()` method. The `DataStream` can by created using a regular [data source]({% link dev/datastream_api.zh.md %}#data-sources) of the DataStream API. Commonly, a `StreamTableSource` is implemented by wrapping a `SourceFunction` or a [stream connector]({% link dev/connectors/index.zh.md %}). + +{% top %} + +### Defining a TableSource with Time Attributes + +Time-based operations of streaming [Table API]({% link dev/table/tableApi.zh.md %}#group-windows) and [SQL]({% link dev/table/sql/queries.zh.md %}#group-windows) queries, such as windowed aggregations or joins, require explicitly specified [time attributes]({% link dev/table/streaming/time_attributes.zh.md %}). + +A `TableSource` defines a time attribute as a field of type `Types.SQL_TIMESTAMP` in its table schema. In contrast to all regular fields in the schema, a time attribute must not be matched to a physical field in the return type of the table source. Instead, a `TableSource` defines a time attribute by implementing a certain interface. + +#### Defining a Processing Time Attribute + +[Processing time attributes]({% link dev/table/streaming/time_attributes.zh.md %}#processing-time) are commonly used in streaming queries. A processing time attribute returns the current wall-clock time of the operator that accesses it. A `TableSource` defines a processing time attribute by implementing the `DefinedProctimeAttribute` interface. The interface looks as follows: + ++{% highlight java %} +StreamTableSource+ +implements TableSource { + + public DataStream getDataStream(StreamExecutionEnvironment execEnv); +} +{% endhighlight %} + +{% highlight scala %} +StreamTableSource[T] extends TableSource[T] { + + def getDataStream(execEnv: StreamExecutionEnvironment): DataStream[T] +} +{% endhighlight %} ++++ +* `getProctimeAttribute()`: Returns the name of the processing time attribute. The specified attribute must be defined of type `Types.SQL_TIMESTAMP` in the table schema and can be used in time-based operations. A `DefinedProctimeAttribute` table source can define no processing time attribute by returning `null`. + +Attention Both `StreamTableSource` and `BatchTableSource` can implement `DefinedProctimeAttribute` and define a processing time attribute. In case of a `BatchTableSource` the processing time field is initialized with the current timestamp during the table scan. + +#### Defining a Rowtime Attribute + +[Rowtime attributes]({% link dev/table/streaming/time_attributes.zh.md %}#event-time) are attributes of type `TIMESTAMP` and handled in a unified way in stream and batch queries. + +A table schema field of type `SQL_TIMESTAMP` can be declared as rowtime attribute by specifying + +* the name of the field, +* a `TimestampExtractor` that computes the actual value for the attribute (usually from one or more other fields), and +* a `WatermarkStrategy` that specifies how watermarks are generated for the the rowtime attribute. + +A `TableSource` defines a rowtime attribute by implementing the `DefinedRowtimeAttributes` interface. The interface looks as follows: + ++{% highlight java %} +DefinedProctimeAttribute { + + public String getProctimeAttribute(); +} +{% endhighlight %} ++ ++{% highlight scala %} +DefinedProctimeAttribute { + + def getProctimeAttribute: String +} +{% endhighlight %} ++++ +* `getRowtimeAttributeDescriptors()`: Returns a list of `RowtimeAttributeDescriptor`. A `RowtimeAttributeDescriptor` describes a rowtime attribute with the following properties: + * `attributeName`: The name of the rowtime attribute in the table schema. The field must be defined with type `Types.SQL_TIMESTAMP`. + * `timestampExtractor`: The timestamp extractor extracts the timestamp from a record with the return type. For example, it can convert a Long field into a timestamp or parse a String-encoded timestamp. Flink comes with a set of built-in `TimestampExtractor` implementation for common use cases. It is also possible to provide a custom implementation. + * `watermarkStrategy`: The watermark strategy defines how watermarks are generated for the rowtime attribute. Flink comes with a set of built-in `WatermarkStrategy` implementations for common use cases. It is also possible to provide a custom implementation. + +Attention Although the `getRowtimeAttributeDescriptors()` method returns a list of descriptors, only a single rowtime attribute is support at the moment. We plan to remove this restriction in the future and support tables with more than one rowtime attribute. + +Attention Both, `StreamTableSource` and `BatchTableSource`, can implement `DefinedRowtimeAttributes` and define a rowtime attribute. In either case, the rowtime field is extracted using the `TimestampExtractor`. Hence, a `TableSource` that implements `StreamTableSource` and `BatchTableSource` and defines a rowtime attribute provides exactly the same data to streaming and batch queries. + +##### Provided Timestamp Extractors + +Flink provides `TimestampExtractor` implementations for common use cases. + +The following `TimestampExtractor` implementations are currently available: + +* `ExistingField(fieldName)`: Extracts the value of a rowtime attribute from an existing `LONG`, `SQL_TIMESTAMP`, or timestamp formatted `STRING` field. One example of such a string would be '2018-05-28 12:34:56.000'. +* `StreamRecordTimestamp()`: Extracts the value of a rowtime attribute from the timestamp of the `DataStream` `StreamRecord`. Note, this `TimestampExtractor` is not available for batch table sources. + +A custom `TimestampExtractor` can be defined by implementing the corresponding interface. + +##### Provided Watermark Strategies + +Flink provides `WatermarkStrategy` implementations for common use cases. + +The following `WatermarkStrategy` implementations are currently available: + +* `AscendingTimestamps`: A watermark strategy for ascending timestamps. Records with timestamps that are out-of-order will be considered late. +* `BoundedOutOfOrderTimestamps(delay)`: A watermark strategy for timestamps that are at most out-of-order by the specified delay. +* `PreserveWatermarks()`: A strategy which indicates the watermarks should be preserved from the underlying `DataStream`. + +A custom `WatermarkStrategy` can be defined by implementing the corresponding interface. + +{% top %} + +### Defining a TableSource with Projection Push-Down + +A `TableSource` supports projection push-down by implementing the `ProjectableTableSource` interface. The interface defines a single method: + ++{% highlight java %} +DefinedRowtimeAttribute { + + public List+ +getRowtimeAttributeDescriptors(); +} +{% endhighlight %} + +{% highlight scala %} +DefinedRowtimeAttributes { + + def getRowtimeAttributeDescriptors: util.List[RowtimeAttributeDescriptor] +} +{% endhighlight %} ++++ +* `projectFields(fields)`: Returns a *copy* of the `TableSource` with adjusted physical return type. The `fields` parameter provides the indexes of the fields that must be provided by the `TableSource`. The indexes relate to the `TypeInformation` of the physical return type, *not* to the logical table schema. The copied `TableSource` must adjust its return type and the returned `DataStream` or `DataSet`. The `TableSchema` of the copied `TableSource` must not be changed, i.e, it must be the same as the original `TableSource`. If the `TableSource` implements the `DefinedFieldMapping` interface, the field mapping must be adjusted to the new return type. + +Attention In order for Flink to distinguish a projection push-down table source from its original form, `explainSource` method must be override to include information regarding the projected fields. + +The `ProjectableTableSource` adds support to project flat fields. If the `TableSource` defines a table with nested schema, it can implement the `NestedFieldsProjectableTableSource` to extend the projection to nested fields. The `NestedFieldsProjectableTableSource` is defined as follows: + ++{% highlight java %} +ProjectableTableSource+ +{ + + public TableSource projectFields(int[] fields); +} +{% endhighlight %} + +{% highlight scala %} +ProjectableTableSource[T] { + + def projectFields(fields: Array[Int]): TableSource[T] +} +{% endhighlight %} ++++ +* `projectNestedField(fields, nestedFields)`: Returns a *copy* of the `TableSource` with adjusted physical return type. Fields of the physical return type may be removed or reordered but their type must not be changed. The contract of this method is essentially the same as for the `ProjectableTableSource.projectFields()` method. In addition, the `nestedFields` parameter contains for each field index in the `fields` list, a list of paths to all nested fields that are accessed by the query. All other nested fields do not need to be read, parsed, and set in the records that are produced by the `TableSource`. + +Attention the types of the projected fields must not be changed but unused fields may be set to null or to a default value. + +{% top %} + +### Defining a TableSource with Filter Push-Down + +The `FilterableTableSource` interface adds support for filter push-down to a `TableSource`. A `TableSource` extending this interface is able to filter records such that the returned `DataStream` or `DataSet` returns fewer records. + +The interface looks as follows: + ++{% highlight java %} +NestedFieldsProjectableTableSource+ +{ + + public TableSource projectNestedFields(int[] fields, String[][] nestedFields); +} +{% endhighlight %} + +{% highlight scala %} +NestedFieldsProjectableTableSource[T] { + + def projectNestedFields(fields: Array[Int], nestedFields: Array[Array[String]]): TableSource[T] +} +{% endhighlight %} ++++ +* `applyPredicate(predicates)`: Returns a *copy* of the `TableSource` with added predicates. The `predicates` parameter is a mutable list of conjunctive predicates that are "offered" to the `TableSource`. The `TableSource` accepts to evaluate a predicate by removing it from the list. Predicates that are left in the list will be evaluated by a subsequent filter operator. +* `isFilterPushedDown()`: Returns true if the `applyPredicate()` method was called before. Hence, `isFilterPushedDown()` must return true for all `TableSource` instances returned from a `applyPredicate()` call. + +Attention In order for Flink to distinguish a filter push-down table source from its original form, `explainSource` method must be override to include information regarding the push-down filters. + +{% top %} + +### Defining a TableSource for Lookups + +Attention This is an experimental feature. The interface may be changed in future versions. It's only supported in Blink planner. + +The `LookupableTableSource` interface adds support for the table to be accessed via key column(s) in a lookup fashion. This is very useful when used to join with a dimension table to enrich some information. If you want to use the `TableSource` in lookup mode, you should use the source in [temporal table join syntax]({% link dev/table/streaming/joins.zh.md %}). + +The interface looks as follows: + ++{% highlight java %} +FilterableTableSource+ +{ + + public TableSource applyPredicate(List predicates); + + public boolean isFilterPushedDown(); +} +{% endhighlight %} + +{% highlight scala %} +FilterableTableSource[T] { + + def applyPredicate(predicates: java.util.List[Expression]): TableSource[T] + + def isFilterPushedDown: Boolean +} +{% endhighlight %} ++++ +* `getLookupFunction(lookupkeys)`: Returns a `TableFunction` which used to lookup the matched row(s) via lookup keys. The lookupkeys are the field names of `LookupableTableSource` in the join equal conditions. The eval method parameters of the returned `TableFunction`'s should be in the order which `lookupkeys` defined. It is recommended to define the parameters in varargs (e.g. `eval(Object... lookupkeys)` to match all the cases). The return type of the `TableFunction` must be identical to the return type defined by the `TableSource.getReturnType()` method. +* `getAsyncLookupFunction(lookupkeys)`: Optional. Similar to `getLookupFunction`, but the `AsyncLookupFunction` lookups the matched row(s) asynchronously. The underlying of `AsyncLookupFunction` will be called via [Async I/O]({% link dev/stream/operators/asyncio.zh.md %}). The first argument of the eval method of the returned `AsyncTableFunction` should be defined as `java.util.concurrent.CompletableFuture` to collect results asynchronously (e.g. `eval(CompletableFuture+{% highlight java %} +LookupableTableSource+ +implements TableSource { + + public TableFunction getLookupFunction(String[] lookupkeys); + + public AsyncTableFunction getAsyncLookupFunction(String[] lookupkeys); + + public boolean isAsyncEnabled(); +} +{% endhighlight %} + +{% highlight scala %} +LookupableTableSource[T] extends TableSource[T] { + + def getLookupFunction(lookupKeys: Array[String]): TableFunction[T] + + def getAsyncLookupFunction(lookupKeys: Array[String]): AsyncTableFunction[T] + + def isAsyncEnabled: Boolean +} +{% endhighlight %} ++++ +The `TableSink#configure` method is called to pass the schema of the Table (field names and types) to emit to the `TableSink`. The method must return a new instance of the TableSink which is configured to emit the provided Table schema. Note that the provided `TableSchema` shouldn't contain computed columns to reflect the schema of the physical `TableSink`. + +### BatchTableSink + +Defines an external `TableSink` to emit a batch table. + +The interface looks as follows: + ++{% highlight java %} +TableSink+ +{ + + public TypeInformation getOutputType(); + + public String[] getFieldNames(); + + public TypeInformation[] getFieldTypes(); + + public TableSink configure(String[] fieldNames, TypeInformation[] fieldTypes); +} +{% endhighlight %} + +{% highlight scala %} +TableSink[T] { + + def getOutputType: TypeInformation++ + def getFieldNames: Array[String] + + def getFieldTypes: Array[TypeInformation] + + def configure(fieldNames: Array[String], fieldTypes: Array[TypeInformation]): TableSink[T] +} +{% endhighlight %} + ++ +{% top %} + +### AppendStreamTableSink + +Defines an external `TableSink` to emit a streaming table with only insert changes. + +The interface looks as follows: + ++{% highlight java %} +BatchTableSink+ +implements TableSink { + + public void emitDataSet(DataSet dataSet); +} +{% endhighlight %} + +{% highlight scala %} +BatchTableSink[T] extends TableSink[T] { + + def emitDataSet(dataSet: DataSet[T]): Unit +} +{% endhighlight %} ++++ +If the table is also modified by update or delete changes, a `TableException` will be thrown. + +{% top %} + +### RetractStreamTableSink + +Defines an external `TableSink` to emit a streaming table with insert, update, and delete changes. + +The interface looks as follows: + ++{% highlight java %} +AppendStreamTableSink+ +implements TableSink { + + public DataStreamSink consumeDataStream(DataStream dataStream); +} +{% endhighlight %} + +{% highlight scala %} +AppendStreamTableSink[T] extends TableSink[T] { + + def consumeDataStream(dataStream: DataStream[T]): DataStreamSink[_] +} +{% endhighlight %} ++++ +The table will be converted into a stream of accumulate and retraction messages which are encoded as Java `Tuple2`. The first field is a boolean flag to indicate the message type (`true` indicates insert, `false` indicates delete). The second field holds the record of the requested type `T`. + +{% top %} + +### UpsertStreamTableSink + +Defines an external `TableSink` to emit a streaming table with insert, update, and delete changes. + +The interface looks as follows: + ++{% highlight java %} +RetractStreamTableSink+ +implements TableSink getRecordType(); + + public DataStreamSink consumeDataStream(DataStream +{% highlight scala %} +RetractStreamTableSink[T] extends TableSink[Tuple2[Boolean, T]] { + + def getRecordType: TypeInformation[T] + + def consumeDataStream(dataStream: DataStream[Tuple2[Boolean, T]]): DataStreamSink[_] +} +{% endhighlight %} ++++ +The table must be have unique key fields (atomic or composite) or be append-only. If the table does not have a unique key and is not append-only, a `TableException` will be thrown. The unique key of the table is configured by the `UpsertStreamTableSink#setKeyFields()` method. + +The table will be converted into a stream of upsert and delete messages which are encoded as a Java `Tuple2`. The first field is a boolean flag to indicate the message type. The second field holds the record of the requested type `T`. + +A message with true boolean field is an upsert message for the configured key. A message with false flag is a delete message for the configured key. If the table is append-only, all messages will have a true flag and must be interpreted as insertions. + +{% top %} + +Define a TableFactory +--------------------- + +A `TableFactory` allows to create different table-related instances from string-based properties. All available factories are called for matching to the given set of properties and a corresponding factory class. + +Factories leverage Java's [Service Provider Interfaces (SPI)](https://docs.oracle.com/javase/tutorial/sound/SPI-intro.html) for discovering. This means that every dependency and JAR file should contain a file `org.apache.flink.table.factories.TableFactory` in the `META_INF/services` resource directory that lists all available table factories that it provides. + +Every table factory needs to implement the following interface: + ++{% highlight java %} +UpsertStreamTableSink+ +implements TableSink getRecordType(); + + public DataStreamSink consumeDataStream(DataStream +{% highlight scala %} +UpsertStreamTableSink[T] extends TableSink[Tuple2[Boolean, T]] { + + def setKeyFields(keys: Array[String]): Unit + + def setIsAppendOnly(isAppendOnly: Boolean): Unit + + def getRecordType: TypeInformation[T] + + def consumeDataStream(dataStream: DataStream[Tuple2[Boolean, T]]): DataStreamSink[_] +} +{% endhighlight %} ++++ +* `requiredContext()`: Specifies the context that this factory has been implemented for. The framework guarantees to only match for this factory if the specified set of properties and values are met. Typical properties might be `connector.type`, `format.type`, or `update-mode`. Property keys such as `connector.property-version` and `format.property-version` are reserved for future backwards compatibility cases. +* `supportedProperties()`: List of property keys that this factory can handle. This method will be used for validation. If a property is passed that this factory cannot handle, an exception will be thrown. The list must not contain the keys that are specified by the context. + +In order to create a specific instance, a factory class can implement one or more interfaces provided in `org.apache.flink.table.factories`: + +* `BatchTableSourceFactory`: Creates a batch table source. +* `BatchTableSinkFactory`: Creates a batch table sink. +* `StreamTableSourceFactory`: Creates a stream table source. +* `StreamTableSinkFactory`: Creates a stream table sink. +* `DeserializationSchemaFactory`: Creates a deserialization schema format. +* `SerializationSchemaFactory`: Creates a serialization schema format. + +The discovery of a factory happens in multiple stages: + +- Discover all available factories. +- Filter by factory class (e.g., `StreamTableSourceFactory`). +- Filter by matching context. +- Filter by supported properties. +- Verify that exactly one factory matches, otherwise throw an `AmbiguousTableFactoryException` or `NoMatchingTableFactoryException`. + +The following example shows how to provide a custom streaming source with an additional `connector.debug` property flag for parameterization. + ++{% highlight java %} +package org.apache.flink.table.factories; + +interface TableFactory { + + Map+ +supportedProperties(); +} +{% endhighlight %} + +{% highlight scala %} +package org.apache.flink.table.factories + +trait TableFactory { + + def requiredContext(): util.Map[String, String] + + def supportedProperties(): util.List[String] +} +{% endhighlight %} ++++ +{% top %} + +### Use a TableFactory in the SQL Client + +In a SQL Client environment file, the previously presented factory could be declared as: + +{% highlight yaml %} +tables: + - name: MySystemTable + type: source + update-mode: append + connector: + type: my-system + debug: true +{% endhighlight %} + +The YAML file is translated into flattened string properties and a table factory is called with those properties that describe the connection to the external system: + +{% highlight text %} +update-mode=append +connector.type=my-system +connector.debug=true +{% endhighlight %} + +Attention Properties such as `tables.#.name` or `tables.#.type` are SQL Client specifics and are not passed to any factory. The `type` property decides, depending on the execution environment, whether a `BatchTableSourceFactory`/`StreamTableSourceFactory` (for `source`), a `BatchTableSinkFactory`/`StreamTableSinkFactory` (for `sink`), or both (for `both`) need to discovered. + +{% top %} + +### Use a TableFactory in the Table & SQL API + +For a type-safe, programmatic approach with explanatory Scaladoc/Javadoc, the Table & SQL API offers descriptors in `org.apache.flink.table.descriptors` that translate into string-based properties. See the [built-in descriptors]({% link dev/table/connect.zh.md %}) for sources, sinks, and formats as a reference. + ++{% highlight java %} +import org.apache.flink.table.sources.StreamTableSource; +import org.apache.flink.types.Row; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +class MySystemTableSourceFactory implements StreamTableSourceFactory+ +{ + + @Override + public Map
supportedProperties() { + List list = new ArrayList<>(); + list.add("connector.debug"); + return list; + } + + @Override + public StreamTableSource createStreamTableSource(Map
+{% highlight scala %} +import java.util +import org.apache.flink.table.sources.StreamTableSource +import org.apache.flink.types.Row + +class MySystemTableSourceFactory extends StreamTableSourceFactory[Row] { + + override def requiredContext(): util.Map[String, String] = { + val context = new util.HashMap[String, String]() + context.put("update-mode", "append") + context.put("connector.type", "my-system") + context + } + + override def supportedProperties(): util.List[String] = { + val properties = new util.ArrayList[String]() + properties.add("connector.debug") + properties + } + + override def createStreamTableSource(properties: util.Map[String, String]): StreamTableSource[Row] = { + val isDebug = java.lang.Boolean.valueOf(properties.get("connector.debug")) + + # additional validation of the passed properties can also happen here + + new MySystemAppendTableSource(isDebug) + } +} +{% endhighlight %} ++++ +{% top %} diff --git a/docs/dev/table/python/common_questions.md b/docs/dev/table/python/common_questions.md index 89da24cbb5855f..de6c75e5c2243b 100644 --- a/docs/dev/table/python/common_questions.md +++ b/docs/dev/table/python/common_questions.md @@ -34,9 +34,9 @@ You can specify the version parameter to generate a Python virtual environment r {% highlight bash %} {% if site.is_stable %} -$ setup-pyflink-virtual-env.sh {{ site.version }} +$ sh setup-pyflink-virtual-env.sh {{ site.version }} {% else %} -$ setup-pyflink-virtual-env.sh +$ sh setup-pyflink-virtual-env.sh {% endif %} {% endhighlight bash %} @@ -69,11 +69,11 @@ A PyFlink job may depend on jar files, i.e. connectors, Java UDFs, etc. You can specify the dependencies with the following Python Table APIs or through command-line arguments directly when submitting the job. {% highlight python %} -# NOTE: Only local file URLs (start with "file://") are supported. -table_env.get_config().set_configuration("pipeline.jars", "file:///my/jar/path/connector.jar;file:///my/jar/path/udf.jar") +# NOTE: Only local file URLs (start with "file:") are supported. +table_env.get_config().get_configuration().set_string("pipeline.jars", "file:///my/jar/path/connector.jar;file:///my/jar/path/udf.jar") -# NOTE: The Paths must specify a protocol (e.g. file://) and users should ensure that the URLs are accessible on both the client and the cluster. -table_env.get_config().set_configuration("pipeline.classpaths", "file:///my/jar/path/connector.jar;file:///my/jar/path/udf.jar") +# NOTE: The Paths must specify a protocol (e.g. "file") and users should ensure that the URLs are accessible on both the client and the cluster. +table_env.get_config().get_configuration().set_string("pipeline.classpaths", "file:///my/jar/path/connector.jar;file:///my/jar/path/udf.jar") {% endhighlight %} For details about the APIs of adding Java dependency, you can refer to [the relevant documentation]({{ site.baseurl }}/dev/table/python/dependency_management.html##java-dependency) diff --git a/docs/dev/table/python/common_questions.zh.md b/docs/dev/table/python/common_questions.zh.md index f11f83c2175e78..3821c214fffc84 100644 --- a/docs/dev/table/python/common_questions.zh.md +++ b/docs/dev/table/python/common_questions.zh.md @@ -34,9 +34,9 @@ You can specify the version parameter to generate a Python virtual environment r {% highlight bash %} {% if site.is_stable %} -$ setup-pyflink-virtual-env.sh {{ site.version }} +$ sh setup-pyflink-virtual-env.sh {{ site.version }} {% else %} -$ setup-pyflink-virtual-env.sh +$ sh setup-pyflink-virtual-env.sh {% endif %} {% endhighlight bash %} @@ -69,11 +69,11 @@ A PyFlink job may depend on jar files, i.e. connectors, Java UDFs, etc. You can specify the dependencies with the following Python Table APIs or through command-line arguments directly when submitting the job. {% highlight python %} -# NOTE: Only local file URLs (start with "file://") are supported. -table_env.get_config().set_configuration("pipeline.jars", "file:///my/jar/path/connector.jar;file:///my/jar/path/udf.jar") +# NOTE: Only local file URLs (start with "file:") are supported. +table_env.get_config().get_configuration().set_string("pipeline.jars", "file:///my/jar/path/connector.jar;file:///my/jar/path/udf.jar") -# NOTE: The Paths must specify a protocol (e.g. file://) and users should ensure that the URLs are accessible on both the client and the cluster. -table_env.get_config().set_configuration("pipeline.classpaths", "file:///my/jar/path/connector.jar;file:///my/jar/path/udf.jar") +# NOTE: The Paths must specify a protocol (e.g. "file") and users should ensure that the URLs are accessible on both the client and the cluster. +table_env.get_config().get_configuration().set_string("pipeline.classpaths", "file:///my/jar/path/connector.jar;file:///my/jar/path/udf.jar") {% endhighlight %} For details about the APIs of adding Java dependency, you can refer to [the relevant documentation]({{ site.baseurl }}/zh/dev/table/python/dependency_management.html##java-dependency) diff --git a/docs/dev/table/python/installation.md b/docs/dev/table/python/installation.md index 0010ca989d8ef1..c33d83c56308ba 100644 --- a/docs/dev/table/python/installation.md +++ b/docs/dev/table/python/installation.md @@ -42,3 +42,5 @@ $ python -m pip install apache-flink {% endhighlight %} You can also build PyFlink from source by following the [development guide]({{ site.baseurl }}/flinkDev/building.html#build-pyflink). + +Note Starting from Flink 1.11, it's also supported to run PyFlink jobs locally on Windows and so you could develop and debug PyFlink jobs on Windows. diff --git a/docs/dev/table/python/installation.zh.md b/docs/dev/table/python/installation.zh.md index 43ccca8967b972..6e4e45f8d9f10e 100644 --- a/docs/dev/table/python/installation.zh.md +++ b/docs/dev/table/python/installation.zh.md @@ -25,20 +25,22 @@ under the License. * This will be replaced by the TOC {:toc} -## Environment Requirements -Note Python version (3.5, 3.6 or 3.7) is required for PyFlink. Please run the following command to make sure that it meets the requirements: +## 环境要求 +注意 PyFlink 需要特定的Python 版本（3.5, 3.6 或 3.7）。请运行如下的命令确保版本满足要求。 {% highlight bash %} $ python --version # the version printed here must be 3.5, 3.6 or 3.7 {% endhighlight %} -## Installation of PyFlink +## PyFlink 安装 -PyFlink has already been deployed to PyPi and can be installed as following: +PyFlink 已经被部署到 PyPi，可以按如下方式安装： {% highlight bash %} $ python -m pip install apache-flink {% endhighlight %} -You can also build PyFlink from source by following the [development guide]({{ site.baseurl }}/zh/flinkDev/building.html#build-pyflink). +你也可以从源码手动构建 PyFlink，具体可以参见[开发指南]({{ site.baseurl }}/zh/flinkDev/building.html#build-pyflink). + +注意 从 Flink 1.11 版本开始, PyFlink 作业可以支持在 Windows 系统上本地运行，因此你也可以在 Windows 上开发和调试 PyFlink 作业了。 diff --git a/docs/dev/table/python/python_udfs.md b/docs/dev/table/python/python_udfs.md index ea01cd0ca3faba..98ce5f064f2af7 100644 --- a/docs/dev/table/python/python_udfs.md +++ b/docs/dev/table/python/python_udfs.md @@ -48,6 +48,9 @@ class HashCode(ScalarFunction): table_env = BatchTableEnvironment.create(env) +# configure the off-heap memory of current taskmanager to enable the python worker uses off-heap memory. +table_env.get_config().get_configuration().set_string("taskmanager.memory.task.off-heap.size", '80m') + # register the Python function table_env.register_function("hash_code", udf(HashCode(), DataTypes.BIGINT(), DataTypes.BIGINT())) @@ -58,6 +61,10 @@ my_table.select("string, bigint, bigint.hash_code(), hash_code(bigint)") table_env.sql_query("SELECT string, bigint, hash_code(bigint) FROM MyTable") {% endhighlight %} +Note If not using RocksDB as state backend, you can also configure the python +worker to use the managed memory of taskmanager by setting **python.fn-execution.memory.managed** to be **true**. +Then there is no need to set the the configuration **taskmanager.memory.task.off-heap.size**. + It also supports to use Java/Scala scalar functions in Python Table API programs. {% highlight python %} @@ -76,6 +83,9 @@ public class HashCode extends ScalarFunction { table_env = BatchTableEnvironment.create(env) +# configure the off-heap memory of current taskmanager to enable the python worker uses off-heap memory. +table_env.get_config().get_configuration().set_string("taskmanager.memory.task.off-heap.size", '80m') + # register the Java function table_env.register_java_function("hash_code", "my.java.function.HashCode") @@ -86,6 +96,10 @@ my_table.select("string.hash_code(), hash_code(string)") table_env.sql_query("SELECT string, bigint, hash_code(string) FROM MyTable") {% endhighlight %} +Note If not using RocksDB as state backend, you can also configure the python +worker to use the managed memory of taskmanager by setting **python.fn-execution.memory.managed** to be **true**. +Then there is no need to set the the configuration **taskmanager.memory.task.off-heap.size**. + There are many ways to define a Python scalar function besides extending the base class `ScalarFunction`. The following examples show the different ways to define a Python scalar function which takes two columns of bigint as the input parameters and returns the sum of them as the result. @@ -145,6 +159,9 @@ env = StreamExecutionEnvironment.get_execution_environment() table_env = StreamTableEnvironment.create(env) my_table = ... # type: Table, table schema: [a: String] +# configure the off-heap memory of current taskmanager to enable the python worker uses off-heap memory. +table_env.get_config().get_configuration().set_string("taskmanager.memory.task.off-heap.size", '80m') + # register the Python Table Function table_env.register_function("split", udtf(Split(), DataTypes.STRING(), [DataTypes.STRING(), DataTypes.INT()])) @@ -158,6 +175,9 @@ table_env.sql_query("SELECT a, word, length FROM MyTable LEFT JOIN LATERAL TABLE {% endhighlight %} +Note If not using RocksDB as state backend, you can also configure the python +worker to use the managed memory of taskmanager by setting **python.fn-execution.memory.managed** to be **true**. +Then there is no need to set the the configuration **taskmanager.memory.task.off-heap.size**. It also supports to use Java/Scala table functions in Python Table API programs. {% highlight python %} @@ -182,6 +202,9 @@ env = StreamExecutionEnvironment.get_execution_environment() table_env = StreamTableEnvironment.create(env) my_table = ... # type: Table, table schema: [a: String] +# configure the off-heap memory of current taskmanager to enable the python worker uses off-heap memory. +table_env.get_config().get_configuration().set_string("taskmanager.memory.task.off-heap.size", '80m') + # Register the java function. table_env.register_java_function("split", "my.java.function.Split") @@ -198,6 +221,10 @@ table_env.sql_query("SELECT a, word, length FROM MyTable, LATERAL TABLE(split(a) table_env.sql_query("SELECT a, word, length FROM MyTable LEFT JOIN LATERAL TABLE(split(a)) as T(word, length) ON TRUE") {% endhighlight %} +Note If not using RocksDB as state backend, you can also configure the python +worker to use the managed memory of taskmanager by setting **python.fn-execution.memory.managed** to be **true**. +Then there is no need to set the the configuration **taskmanager.memory.task.off-heap.size**. + Like Python scalar functions, you can use the above five ways to define Python TableFunctions. Note The only difference is that the return type of Python Table Functions needs to be an iterable, iterator or generator. diff --git a/docs/dev/table/python/python_udfs.zh.md b/docs/dev/table/python/python_udfs.zh.md index b19120e01a37d7..1d83da55134ea1 100644 --- a/docs/dev/table/python/python_udfs.zh.md +++ b/docs/dev/table/python/python_udfs.zh.md @@ -48,6 +48,9 @@ class HashCode(ScalarFunction): table_env = BatchTableEnvironment.create(env) +# configure the off-heap memory of current taskmanager to enable the python worker uses off-heap memory. +table_env.get_config().get_configuration().set_string("taskmanager.memory.task.off-heap.size", '80m') + # register the Python function table_env.register_function("hash_code", udf(HashCode(), DataTypes.BIGINT(), DataTypes.BIGINT())) @@ -58,6 +61,10 @@ my_table.select("string, bigint, bigint.hash_code(), hash_code(bigint)") table_env.sql_query("SELECT string, bigint, hash_code(bigint) FROM MyTable") {% endhighlight %} +Note If not using RocksDB as state backend, you can also configure the python +worker to use the managed memory of taskmanager by setting **python.fn-execution.memory.managed** to be **true**. +Then there is no need to set the the configuration **taskmanager.memory.task.off-heap.size**. + It also supports to use Java/Scala scalar functions in Python Table API programs. {% highlight python %} @@ -76,6 +83,9 @@ public class HashCode extends ScalarFunction { table_env = BatchTableEnvironment.create(env) +# configure the off-heap memory of current taskmanager to enable the python worker uses off-heap memory. +table_env.get_config().get_configuration().set_string("taskmanager.memory.task.off-heap.size", '80m') + # register the Java function table_env.register_java_function("hash_code", "my.java.function.HashCode") @@ -86,6 +96,10 @@ my_table.select("string.hash_code(), hash_code(string)") table_env.sql_query("SELECT string, bigint, hash_code(string) FROM MyTable") {% endhighlight %} +Note If not using RocksDB as state backend, you can also configure the python +worker to use the managed memory of taskmanager by setting **python.fn-execution.memory.managed** to be **true**. +Then there is no need to set the the configuration **taskmanager.memory.task.off-heap.size**. + There are many ways to define a Python scalar function besides extending the base class `ScalarFunction`. The following examples show the different ways to define a Python scalar function which takes two columns of bigint as the input parameters and returns the sum of them as the result. @@ -145,6 +159,9 @@ env = StreamExecutionEnvironment.get_execution_environment() table_env = StreamTableEnvironment.create(env) my_table = ... # type: Table, table schema: [a: String] +# configure the off-heap memory of current taskmanager to enable the python worker uses off-heap memory. +table_env.get_config().get_configuration().set_string("taskmanager.memory.task.off-heap.size", '80m') + # register the Python Table Function table_env.register_function("split", udtf(Split(), DataTypes.STRING(), [DataTypes.STRING(), DataTypes.INT()])) @@ -158,6 +175,9 @@ table_env.sql_query("SELECT a, word, length FROM MyTable LEFT JOIN LATERAL TABLE {% endhighlight %} +Note If not using RocksDB as state backend, you can also configure the python +worker to use the managed memory of taskmanager by setting **python.fn-execution.memory.managed** to be **true**. +Then there is no need to set the the configuration **taskmanager.memory.task.off-heap.size**. It also supports to use Java/Scala table functions in Python Table API programs. {% highlight python %} @@ -182,6 +202,9 @@ env = StreamExecutionEnvironment.get_execution_environment() table_env = StreamTableEnvironment.create(env) my_table = ... # type: Table, table schema: [a: String] +# configure the off-heap memory of current taskmanager to enable the python worker uses off-heap memory. +table_env.get_config().get_configuration().set_string("taskmanager.memory.task.off-heap.size", '80m') + # Register the java function. table_env.register_java_function("split", "my.java.function.Split") @@ -198,6 +221,10 @@ table_env.sql_query("SELECT a, word, length FROM MyTable, LATERAL TABLE(split(a) table_env.sql_query("SELECT a, word, length FROM MyTable LEFT JOIN LATERAL TABLE(split(a)) as T(word, length) ON TRUE") {% endhighlight %} +Note If not using RocksDB as state backend, you can also configure the python +worker to use the managed memory of taskmanager by setting **python.fn-execution.memory.managed** to be **true**. +Then there is no need to set the the configuration **taskmanager.memory.task.off-heap.size**. + Like Python scalar functions, you can use the above five ways to define Python TableFunctions. Note The only difference is that the return type of Python Table Functions needs to be an iterable, iterator or generator. diff --git a/docs/dev/table/python/vectorized_python_udfs.md b/docs/dev/table/python/vectorized_python_udfs.md index 8eec40bf497bc4..ee5f03f937c1ef 100644 --- a/docs/dev/table/python/vectorized_python_udfs.md +++ b/docs/dev/table/python/vectorized_python_udfs.md @@ -54,6 +54,9 @@ def add(i, j): table_env = BatchTableEnvironment.create(env) +# configure the off-heap memory of current taskmanager to enable the python worker uses off-heap memory. +table_env.get_config().get_configuration().set_string("taskmanager.memory.task.off-heap.size", '80m') + # register the vectorized Python scalar function table_env.register_function("add", add) @@ -63,3 +66,7 @@ my_table.select("add(bigint, bigint)") # use the vectorized Python scalar function in SQL API table_env.sql_query("SELECT add(bigint, bigint) FROM MyTable") {% endhighlight %} + +Note If not using RocksDB as state backend, you can also configure the python +worker to use the managed memory of taskmanager by setting **python.fn-execution.memory.managed** to be **true**. +Then there is no need to set the the configuration **taskmanager.memory.task.off-heap.size**. diff --git a/docs/dev/table/python/vectorized_python_udfs.zh.md b/docs/dev/table/python/vectorized_python_udfs.zh.md index c50bc5b0de1002..b2d2ed922f958f 100644 --- a/docs/dev/table/python/vectorized_python_udfs.zh.md +++ b/docs/dev/table/python/vectorized_python_udfs.zh.md @@ -54,6 +54,9 @@ def add(i, j): table_env = BatchTableEnvironment.create(env) +# configure the off-heap memory of current taskmanager to enable the python worker uses off-heap memory. +table_env.get_config().get_configuration().set_string("taskmanager.memory.task.off-heap.size", '80m') + # register the vectorized Python scalar function table_env.register_function("add", add) @@ -63,3 +66,7 @@ my_table.select("add(bigint, bigint)") # use the vectorized Python scalar function in SQL API table_env.sql_query("SELECT add(bigint, bigint) FROM MyTable") {% endhighlight %} + +Note If not using RocksDB as state backend, you can also configure the python +worker to use the managed memory of taskmanager by setting **python.fn-execution.memory.managed** to be **true**. +Then there is no need to set the the configuration **taskmanager.memory.task.off-heap.size**. diff --git a/docs/dev/table/sourceSinks.md b/docs/dev/table/sourceSinks.md index 7b3647f0673669..e747c6f42018e4 100644 --- a/docs/dev/table/sourceSinks.md +++ b/docs/dev/table/sourceSinks.md @@ -22,796 +22,725 @@ specific language governing permissions and limitations under the License. --> -A `TableSource` provides access to data which is stored in external systems (database, key-value store, message queue) or files. After a [TableSource is registered in a TableEnvironment](common.html#register-a-tablesource) it can be accessed by [Table API](tableApi.html) or [SQL]({{ site.baseurl }}/dev/table/sql/queries.html) queries. +_Dynamic tables_ are the core concept of Flink's Table & SQL API for processing both bounded and unbounded +data in a unified fashion. -A `TableSink` [emits a Table](common.html#emit-a-table) to an external storage system, such as a database, key-value store, message queue, or file system (in different encodings, e.g., CSV, Parquet, or ORC). +Because dynamic tables are only a logical concept, Flink does not own the data itself. Instead, the content +of a dynamic table is stored in external systems (such as databases, key-value stores, message queues) or files. -A `TableFactory` allows for separating the declaration of a connection to an external system from the actual implementation. A table factory creates configured instances of table sources and sinks from normalized, string-based properties. The properties can be generated programmatically using a `Descriptor` or via YAML configuration files for the [SQL Client](sqlClient.html). +_Dynamic sources_ and _dynamic sinks_ can be used to read and write data from and to an external system. In +the documentation, sources and sinks are often summarized under the term _connector_. -Have a look at the [common concepts and API](common.html) page for details how to [register a TableSource](common.html#register-a-tablesource) and how to [emit a Table through a TableSink](common.html#emit-a-table). See the [built-in sources, sinks, and formats](connect.html) page for examples how to use factories. +Flink provides pre-defined connectors for Kafka, Hive, and different file systems. See the [connector section]({% link dev/table/connectors/index.md %}) +for more information about built-in table sources and sinks. -* This will be replaced by the TOC -{:toc} +This page focuses on how to develop a custom, user-defined connector. -Define a TableSource --------------------- +Attention New table source and table sink interfaces have been +introduced in Flink 1.11 as part of [FLIP-95](https://cwiki.apache.org/confluence/display/FLINK/FLIP-95%3A+New+TableSource+and+TableSink+interfaces). +Also the factory interfaces have been reworked. FLIP-95 is not fully implemented yet. Many ability interfaces +are not supported yet (e.g. for filter or partition push down). If necessary, please also have a look +at the [old table sources and sinks page]({% link dev/table/legacySourceSinks.md %}). Those interfaces +are still supported for backwards compatibility. -A `TableSource` is a generic interface that gives Table API and SQL queries access to data stored in an external system. It provides the schema of the table and the records that are mapped to rows with the table's schema. Depending on whether the `TableSource` is used in a streaming or batch query, the records are produced as a `DataSet` or `DataStream`. +* This will be replaced by the TOC +{:toc} -If a `TableSource` is used in a streaming query it must implement the `StreamTableSource` interface, if it is used in a batch query it must implement the `BatchTableSource` interface. A `TableSource` can also implement both interfaces and be used in streaming and batch queries. +Overview +-------- -`StreamTableSource` and `BatchTableSource` extend the base interface `TableSource` that defines the following methods: +In many cases, implementers don't need to create a new connector from scratch but would like to slightly +modify existing connectors or hook into the existing stack. In other cases, implementers would like to +create specialized connectors. -+ +A custom descriptor can be defined by extending the `ConnectorDescriptor` class. + +{% highlight java %} +import org.apache.flink.table.descriptors.ConnectorDescriptor; +import java.util.HashMap; +import java.util.Map; + +/** + * Connector to MySystem with debug mode. + */ +public class MySystemConnector extends ConnectorDescriptor { + + public final boolean isDebug; + + public MySystemConnector(boolean isDebug) { + super("my-system", 1, false); + this.isDebug = isDebug; + } + + @Override + protected Map+ ++ +A custom descriptor can be defined by extending the `ConnectorDescriptor` class. + +{% highlight scala %} +import org.apache.flink.table.descriptors.ConnectorDescriptor +import java.util.HashMap +import java.util.Map + +/** + * Connector to MySystem with debug mode. + */ +class MySystemConnector(isDebug: Boolean) extends ConnectorDescriptor("my-system", 1, false) { + + override protected def toConnectorProperties(): Map[String, String] = { + val properties = new HashMap[String, String] + properties.put("connector.debug", isDebug.toString) + properties + } +} +{% endhighlight %} + +The descriptor can then be used to create a table with the table environment. + +{% highlight scala %} +val tableEnv: StreamTableEnvironment = // ... + +tableEnv + .connect(new MySystemConnector(isDebug = true)) + .withSchema(...) + .inAppendMode() + .createTemporaryTable("MySystemTable") +{% endhighlight %} + ++ ++ +You can use a Java `TableFactory` from Python using the `CustomConnectorDescriptor`. + +{% highlight python %} +s_env = StreamExecutionEnvironment.get_execution_environment() +st_env = StreamTableEnvironment.create(s_env) + +custom_connector = CustomConnectorDescriptor('my-system', 1, False) +st_env\ + .connect(custom_connector.property("connector.debug", "true")) \ + .with_schema(...) \ + .in_append_mode()\ + .create_temporary_table("MySystemTable") +{% endhighlight %} ++ +--{% highlight java %} -TableSource- -* `getTableSchema()`: Returns the schema of the produced table, i.e., the names and types of the fields of the table. The field types are defined using Flink's `DataType` (see [Table API types]({{ site.baseurl }}/dev/table/types.html) and [SQL types]({{ site.baseurl }}/dev/table/sql/index.html#data-types)). Note that the returned `TableSchema` shouldn't contain computed columns to reflect the schema of the physical `TableSource`. +The metadata for dynamic tables (created via DDL or provided by the catalog) is represented as instances +of `CatalogTable`. A table name will be resolved into a `CatalogTable` internally when necessary. -* `getReturnType()`: Returns the physical type of the `DataStream` (`StreamTableSource`) or `DataSet` (`BatchTableSource`) and the records that are produced by the `TableSource`. +### Planning -* `explainSource()`: Returns a String that describes the `TableSource`. This method is optional and used for display purposes only. +When it comes to planning and optimization of the table program, a `CatalogTable` needs to be resolved +into a `DynamicTableSource` (for reading in a `SELECT` query) and `DynamicTableSink` (for writing in +an `INSERT INTO` statement). -The `TableSource` interface separates the logical table schema from the physical type of the returned `DataStream` or `DataSet`. As a consequence, all fields of the table schema (`getTableSchema()`) must be mapped to a field with corresponding type of the physical return type (`getReturnType()`). By default, this mapping is done based on field names. For example, a `TableSource` that defines a table schema with two fields `[name: String, size: Integer]` requires a `TypeInformation` with at least two fields called `name` and `size` of type `String` and `Integer`, respectively. This could be a `PojoTypeInfo` or a `RowTypeInfo` that have two fields named `name` and `size` with matching types. +`DynamicTableSourceFactory` and `DynamicTableSinkFactory` provide connector-specific logic for translating +the metadata of a `CatalogTable` into instances of `DynamicTableSource` and `DynamicTableSink`. In most +of the cases, a factory's purpose is to validate options (such as `'port' = '5022'` in the example), +configure encoding/decoding formats (if required), and create a parameterized instance of the table +connector. -However, some types, such as Tuple or CaseClass types, do support custom field names. If a `TableSource` returns a `DataStream` or `DataSet` of a type with fixed field names, it can implement the `DefinedFieldMapping` interface to map field names from the table schema to field names of the physical return type. +By default, instances of `DynamicTableSourceFactory` and `DynamicTableSinkFactory` are discovered using +Java's [Service Provider Interfaces (SPI)](https://docs.oracle.com/javase/tutorial/sound/SPI-intro.html). The +`connector` option (such as `'connector' = 'custom'` in the example) must correspond to a valid factory +identifier. -### Defining a BatchTableSource +Although it might not be apparent in the class naming, `DynamicTableSource` and `DynamicTableSink` +can also be seen as stateful factories that eventually produce concrete runtime implementation for reading/writing +the actual data. -The `BatchTableSource` interface extends the `TableSource` interface and defines one additional method: +The planner uses the source and sink instances to perform connector-specific bidirectional comunication +until an optimal logical plan could be found. Depending on the optionally declared ability interfaces (e.g. +`SupportsProjectionPushDown` or `SupportsOverwrite`), the planner might apply changes to an instance and +thus mutate the produced runtime implementation. -{ +This section helps for both kinds of use cases. It explains the general architecture of table connectors +from pure declaration in the API to runtime code that will be executed on the cluster. - public TableSchema getTableSchema(); +The filled arrows show how objects are transformed to other objects from one stage to the next stage during +the translation process. - public TypeInformation getReturnType(); - - public String explainSource(); -} -{% endhighlight %} + +-
-{% highlight scala %} -TableSource[T] { - - def getTableSchema: TableSchema +### Metadata - def getReturnType: TypeInformation[T] +Both Table API and SQL are declarative APIs. This includes the declaration of tables. Thus, executing +a `CREATE TABLE` statement results in updated metadata in the target catalog. - def explainSource: String +For most catalog implementations, physical data in the external system is not modified for such an +operation. Connector-specific dependencies don't have to be present in the classpath yet. The options declared +in the `WITH` clause are neither validated nor otherwise interpreted. -} -{% endhighlight %} ---+Those interfaces are grouped by another level of abstraction as subclasses of `ScanRuntimeProvider`, +`LookupRuntimeProvider`, and `SinkRuntimeProvider`. -* `getDataSet(execEnv)`: Returns a `DataSet` with the data of the table. The type of the `DataSet` must be identical to the return type defined by the `TableSource.getReturnType()` method. The `DataSet` can by created using a regular [data source]({{ site.baseurl }}/dev/batch/#data-sources) of the DataSet API. Commonly, a `BatchTableSource` is implemented by wrapping a `InputFormat` or [batch connector]({{ site.baseurl }}/dev/batch/connectors.html). +For example, both `OutputFormatProvider` (providing `org.apache.flink.api.common.io.OutputFormat`) and `SinkFunctionProvider` (providing `org.apache.flink.streaming.api.functions.sink.SinkFunction`) are concrete instances of `SinkRuntimeProvider` +that the planner can handle. {% top %} -### Defining a StreamTableSource +Extension Points +---------------- -The `StreamTableSource` interface extends the `TableSource` interface and defines one additional method: +This section explains the available interfaces for extending Flink's table connectors. --{% highlight java %} -BatchTableSource+### Runtime -implements TableSource { - - public DataSet getDataSet(ExecutionEnvironment execEnv); -} -{% endhighlight %} - -{% highlight scala %} -BatchTableSource[T] extends TableSource[T] { +Once the logical planning is complete, the planner will obtain the _runtime implementation_ from the table +connector. Runtime logic is implemented in Flink's core connector interfaces such as `InputFormat` or `SourceFunction`. - def getDataSet(execEnv: ExecutionEnvironment): DataSet[T] -} -{% endhighlight %} ---+`org.apache.flink.table.factories.DynamicTableSinkFactory` can be implemented to construct a `DynamicTableSink`. -* `getDataStream(execEnv)`: Returns a `DataStream` with the data of the table. The type of the `DataStream` must be identical to the return type defined by the `TableSource.getReturnType()` method. The `DataStream` can by created using a regular [data source]({{ site.baseurl }}/dev/datastream_api.html#data-sources) of the DataStream API. Commonly, a `StreamTableSource` is implemented by wrapping a `SourceFunction` or a [stream connector]({{ site.baseurl }}/dev/connectors/). +By default, the factory is discovered using the value of the `connector` option as the factory identifier +and Java's Service Provider Interface. -{% top %} +In JAR files, references to new implementations can be added to the service file: -### Defining a TableSource with Time Attributes +`META-INF/services/org.apache.flink.table.factories.Factory` -Time-based operations of streaming [Table API](tableApi.html#group-windows) and [SQL]({{ site.baseurl }}/dev/table/sql/queries.html#group-windows) queries, such as windowed aggregations or joins, require explicitly specified [time attributes](streaming/time_attributes.html). +The framework will check for a single matching factory that is uniquely identified by factory identifier +and requested base class (e.g. `DynamicTableSourceFactory`). -A `TableSource` defines a time attribute as a field of type `Types.SQL_TIMESTAMP` in its table schema. In contrast to all regular fields in the schema, a time attribute must not be matched to a physical field in the return type of the table source. Instead, a `TableSource` defines a time attribute by implementing a certain interface. +The factory discovery process can be bypassed by the catalog implementation if necessary. For this, a +catalog needs to return an instance that implements the requested base class in `org.apache.flink.table.catalog.Catalog#getFactory`. -#### Defining a Processing Time Attribute +### Dynamic Table Source -[Processing time attributes](streaming/time_attributes.html#processing-time) are commonly used in streaming queries. A processing time attribute returns the current wall-clock time of the operator that accesses it. A `TableSource` defines a processing time attribute by implementing the `DefinedProctimeAttribute` interface. The interface looks as follows: +By definition, a dynamic table can change over time. --{% highlight java %} -StreamTableSource+Dynamic table factories are used to configure a dynamic table connector for an external storage system from catalog +and session information. -implements TableSource { +### Dynamic Table Factories - public DataStream getDataStream(StreamExecutionEnvironment execEnv); -} -{% endhighlight %} - -{% highlight scala %} -StreamTableSource[T] extends TableSource[T] { +`org.apache.flink.table.factories.DynamicTableSourceFactory` can be implemented to construct a `DynamicTableSource`. - def getDataStream(execEnv: StreamExecutionEnvironment): DataStream[T] -} -{% endhighlight %} ---+A `ScanTableSource` scans all rows from an external storage system during runtime. -* `getProctimeAttribute()`: Returns the name of the processing time attribute. The specified attribute must be defined of type `Types.SQL_TIMESTAMP` in the table schema and can be used in time-based operations. A `DefinedProctimeAttribute` table source can define no processing time attribute by returning `null`. +The scanned rows don't have to contain only insertions but can also contain updates and deletions. Thus, +the table source can be used to read a (finite or infinite) changelog. The returned _changelog mode_ indicates +the set of changes that the planner can expect during runtime. -Attention Both `StreamTableSource` and `BatchTableSource` can implement `DefinedProctimeAttribute` and define a processing time attribute. In case of a `BatchTableSource` the processing time field is initialized with the current timestamp during the table scan. +For regular batch scenarios, the source can emit a bounded stream of insert-only rows. -#### Defining a Rowtime Attribute +For regular streaming scenarios, the source can emit an unbounded stream of insert-only rows. -[Rowtime attributes](streaming/time_attributes.html#event-time) are attributes of type `TIMESTAMP` and handled in a unified way in stream and batch queries. +For change data capture (CDC) scenarios, the source can emit bounded or unbounded streams with insert, +update, and delete rows. -A table schema field of type `SQL_TIMESTAMP` can be declared as rowtime attribute by specifying +A table source can implement further abilitiy interfaces such as `SupportsProjectionPushDown` that might +mutate an instance during planning. All abilities are listed in the `org.apache.flink.table.connector.source.abilities` +package and in the documentation of `org.apache.flink.table.connector.source.ScanTableSource`. -* the name of the field, -* a `TimestampExtractor` that computes the actual value for the attribute (usually from one or more other fields), and -* a `WatermarkStrategy` that specifies how watermarks are generated for the the rowtime attribute. +The runtime implementation of a `ScanTableSource` must produce internal data structures. Thus, records +must be emitted as `org.apache.flink.table.data.RowData`. The framework provides runtime converters such +that a source can still work on common data structures and perform a conversion at the end. -A `TableSource` defines a rowtime attribute by implementing the `DefinedRowtimeAttributes` interface. The interface looks as follows: +#### Lookup Table Source --{% highlight java %} -DefinedProctimeAttribute { +When reading a dynamic table, the content can either be considered as: +- A changelog (finite or infinite) for which all changes are consumed continuously until the changelog + is exhausted. This is represented by the `ScanTableSource` interface. +- A continuously changing or very large external table whose content is usually never read entirely + but queried for individual values when necessary. This is represented by the `LookupTableSource` + interface. - public String getProctimeAttribute(); -} -{% endhighlight %} -+A class can implement both of these interfaces at the same time. The planner decides about their usage depending +on the specified query. --{% highlight scala %} -DefinedProctimeAttribute { +#### Scan Table Source - def getProctimeAttribute: String -} -{% endhighlight %} ---+Compared to `ScanTableSource`, a `LookupTableSource` does only support emitting insert-only changes currently. -* `getRowtimeAttributeDescriptors()`: Returns a list of `RowtimeAttributeDescriptor`. A `RowtimeAttributeDescriptor` describes a rowtime attribute with the following properties: - * `attributeName`: The name of the rowtime attribute in the table schema. The field must be defined with type `Types.SQL_TIMESTAMP`. - * `timestampExtractor`: The timestamp extractor extracts the timestamp from a record with the return type. For example, it can convert a Long field into a timestamp or parse a String-encoded timestamp. Flink comes with a set of built-in `TimestampExtractor` implementation for common use cases. It is also possible to provide a custom implementation. - * `watermarkStrategy`: The watermark strategy defines how watermarks are generated for the rowtime attribute. Flink comes with a set of built-in `WatermarkStrategy` implementations for common use cases. It is also possible to provide a custom implementation. +Further abilities are not supported. See the documentation of `org.apache.flink.table.connector.source.LookupTableSource` +for more information. -Attention Although the `getRowtimeAttributeDescriptors()` method returns a list of descriptors, only a single rowtime attribute is support at the moment. We plan to remove this restriction in the future and support tables with more than one rowtime attribute. +The runtime implementation of a `LookupTableSource` is a `TableFunction` or `AsyncTableFunction`. The function +will be called with values for the given lookup keys during runtime. -Attention Both, `StreamTableSource` and `BatchTableSource`, can implement `DefinedRowtimeAttributes` and define a rowtime attribute. In either case, the rowtime field is extracted using the `TimestampExtractor`. Hence, a `TableSource` that implements `StreamTableSource` and `BatchTableSource` and defines a rowtime attribute provides exactly the same data to streaming and batch queries. +### Dynamic Table Sink -##### Provided Timestamp Extractors +By definition, a dynamic table can change over time. -Flink provides `TimestampExtractor` implementations for common use cases. +When writing a dynamic table, the content can always be considered as a changelog (finite or infinite) +for which all changes are written out continuously until the changelog is exhausted. The returned _changelog mode_ +indicates the set of changes that the sink accepts during runtime. -The following `TimestampExtractor` implementations are currently available: +For regular batch scenarios, the sink can solely accept insert-only rows and write out bounded streams. -* `ExistingField(fieldName)`: Extracts the value of a rowtime attribute from an existing `LONG`, `SQL_TIMESTAMP`, or timestamp formatted `STRING` field. One example of such a string would be '2018-05-28 12:34:56.000'. -* `StreamRecordTimestamp()`: Extracts the value of a rowtime attribute from the timestamp of the `DataStream` `StreamRecord`. Note, this `TimestampExtractor` is not available for batch table sources. +For regular streaming scenarios, the sink can solely accept insert-only rows and can write out unbounded streams. -A custom `TimestampExtractor` can be defined by implementing the corresponding interface. +For change data capture (CDC) scenarios, the sink can write out bounded or unbounded streams with insert, +update, and delete rows. -##### Provided Watermark Strategies +A table sink can implement further abilitiy interfaces such as `SupportsOverwrite` that might mutate an +instance during planning. All abilities are listed in the `org.apache.flink.table.connector.sink.abilities` +package and in the documentation of `org.apache.flink.table.connector.sink.DynamicTableSink`. -Flink provides `WatermarkStrategy` implementations for common use cases. +The runtime implementation of a `DynamicTableSink` must consume internal data structures. Thus, records +must be accepted as `org.apache.flink.table.data.RowData`. The framework provides runtime converters such +that a sink can still work on common data structures and perform a conversion at the beginning. -The following `WatermarkStrategy` implementations are currently available: +### Encoding / Decoding Formats -* `AscendingTimestamps`: A watermark strategy for ascending timestamps. Records with timestamps that are out-of-order will be considered late. -* `BoundedOutOfOrderTimestamps(delay)`: A watermark strategy for timestamps that are at most out-of-order by the specified delay. -* `PreserveWatermarks()`: A strategy which indicates the watermarks should be preserved from the underlying `DataStream`. +Some table connectors accept different formats that encode and decode keys and/or values. -A custom `WatermarkStrategy` can be defined by implementing the corresponding interface. +Formats work similar to the pattern `DynamicTableSourceFactory -> DynamicTableSource -> ScanRuntimeProvider`, +where the factory is responsible for translating options and the source is responsible for creating runtime logic. -{% top %} +Because formats might be located in different modules, they are discovered using Java's Service Provider +Interface similar to [table factories](#dynamic-table-factories). In order to discover a format factory, +the dynamic table factory searches for a factory that corresponds to a factory identifier and connector-specific +base class. -### Defining a TableSource with Projection Push-Down +For example, the Kafka table source requires a `DeserializationSchema` as runtime interface for a decoding +format. Therefore, the Kafka table source factory uses the value of the `value.format` option to discover +a `DeserializationFormatFactory`. -A `TableSource` supports projection push-down by implementing the `ProjectableTableSource` interface. The interface defines a single method: +The following format factories are currently supported: --{% highlight java %} -DefinedRowtimeAttribute { - - public List+A `LookupTableSource` looks up rows of an external storage system by one or more keys during runtime. -getRowtimeAttributeDescriptors(); -} -{% endhighlight %} - -{% highlight scala %} -DefinedRowtimeAttributes { +Compared to `ScanTableSource`, the source does not have to read the entire table and can lazily fetch individual +values from a (possibly continuously changing) external table when necessary. - def getRowtimeAttributeDescriptors: util.List[RowtimeAttributeDescriptor] -} -{% endhighlight %} ---- -* `projectFields(fields)`: Returns a *copy* of the `TableSource` with adjusted physical return type. The `fields` parameter provides the indexes of the fields that must be provided by the `TableSource`. The indexes relate to the `TypeInformation` of the physical return type, *not* to the logical table schema. The copied `TableSource` must adjust its return type and the returned `DataStream` or `DataSet`. The `TableSchema` of the copied `TableSource` must not be changed, i.e, it must be the same as the original `TableSource`. If the `TableSource` implements the `DefinedFieldMapping` interface, the field mapping must be adjusted to the new return type. - -Attention In order for Flink to distinguish a projection push-down table source from its original form, `explainSource` method must be override to include information regarding the projected fields. - -The `ProjectableTableSource` adds support to project flat fields. If the `TableSource` defines a table with nested schema, it can implement the `NestedFieldsProjectableTableSource` to extend the projection to nested fields. The `NestedFieldsProjectableTableSource` is defined as follows: - --{% highlight java %} -ProjectableTableSource+The format factory translates the options into an `EncodingFormat` or a `DecodingFormat`. Those interfaces are +another kind of factory that produce specialized format runtime logic for the given data type. -{ +``` +org.apache.flink.table.factories.DeserializationFormatFactory +org.apache.flink.table.factories.SerializationFormatFactory +``` - public TableSource projectFields(int[] fields); -} -{% endhighlight %} - -{% highlight scala %} -ProjectableTableSource[T] { - - def projectFields(fields: Array[Int]): TableSource[T] -} -{% endhighlight %} ---- -* `projectNestedField(fields, nestedFields)`: Returns a *copy* of the `TableSource` with adjusted physical return type. Fields of the physical return type may be removed or reordered but their type must not be changed. The contract of this method is essentially the same as for the `ProjectableTableSource.projectFields()` method. In addition, the `nestedFields` parameter contains for each field index in the `fields` list, a list of paths to all nested fields that are accessed by the query. All other nested fields do not need to be read, parsed, and set in the records that are produced by the `TableSource`. - -Attention the types of the projected fields must not be changed but unused fields may be set to null or to a default value. - -{% top %} - -### Defining a TableSource with Filter Push-Down - -The `FilterableTableSource` interface adds support for filter push-down to a `TableSource`. A `TableSource` extending this interface is able to filter records such that the returned `DataStream` or `DataSet` returns fewer records. - -The interface looks as follows: - --{% highlight java %} -NestedFieldsProjectableTableSource- -{ - - public TableSource projectNestedFields(int[] fields, String[][] nestedFields); -} -{% endhighlight %} - -{% highlight scala %} -NestedFieldsProjectableTableSource[T] { - - def projectNestedFields(fields: Array[Int], nestedFields: Array[Array[String]]): TableSource[T] -} -{% endhighlight %} ---- -* `applyPredicate(predicates)`: Returns a *copy* of the `TableSource` with added predicates. The `predicates` parameter is a mutable list of conjunctive predicates that are "offered" to the `TableSource`. The `TableSource` accepts to evaluate a predicate by removing it from the list. Predicates that are left in the list will be evaluated by a subsequent filter operator. -* `isFilterPushedDown()`: Returns true if the `applyPredicate()` method was called before. Hence, `isFilterPushedDown()` must return true for all `TableSource` instances returned from a `applyPredicate()` call. - -Attention In order for Flink to distinguish a filter push-down table source from its original form, `explainSource` method must be override to include information regarding the push-down filters. - -{% top %} - -### Defining a TableSource for Lookups - -Attention This is an experimental feature. The interface may be changed in future versions. It's only supported in Blink planner. - -The `LookupableTableSource` interface adds support for the table to be accessed via key column(s) in a lookup fashion. This is very useful when used to join with a dimension table to enrich some information. If you want to use the `TableSource` in lookup mode, you should use the source in [temporal table join syntax](streaming/joins.html). - -The interface looks as follows: - --{% highlight java %} -FilterableTableSource- -{ - - public TableSource applyPredicate(List predicates); - - public boolean isFilterPushedDown(); -} -{% endhighlight %} - -{% highlight scala %} -FilterableTableSource[T] { - - def applyPredicate(predicates: java.util.List[Expression]): TableSource[T] - - def isFilterPushedDown: Boolean -} -{% endhighlight %} ---- -* `getLookupFunction(lookupkeys)`: Returns a `TableFunction` which used to lookup the matched row(s) via lookup keys. The lookupkeys are the field names of `LookupableTableSource` in the join equal conditions. The eval method parameters of the returned `TableFunction`'s should be in the order which `lookupkeys` defined. It is recommended to define the parameters in varargs (e.g. `eval(Object... lookupkeys)` to match all the cases). The return type of the `TableFunction` must be identical to the return type defined by the `TableSource.getReturnType()` method. -* `getAsyncLookupFunction(lookupkeys)`: Optional. Similar to `getLookupFunction`, but the `AsyncLookupFunction` lookups the matched row(s) asynchronously. The underlying of `AsyncLookupFunction` will be called via [Async I/O]({{ site.baseurl }}/dev/stream/operators/asyncio.html). The first argument of the eval method of the returned `AsyncTableFunction` should be defined as `java.util.concurrent.CompletableFuture` to collect results asynchronously (e.g. `eval(CompletableFuture-{% highlight java %} -LookupableTableSource- -implements TableSource { - - public TableFunction getLookupFunction(String[] lookupkeys); - - public AsyncTableFunction getAsyncLookupFunction(String[] lookupkeys); - - public boolean isAsyncEnabled(); -} -{% endhighlight %} - -{% highlight scala %} -LookupableTableSource[T] extends TableSource[T] { - - def getLookupFunction(lookupKeys: Array[String]): TableFunction[T] - - def getAsyncLookupFunction(lookupKeys: Array[String]): AsyncTableFunction[T] - - def isAsyncEnabled: Boolean -} -{% endhighlight %} --` +that can be passed into the Kafka table source. {% top %} -Define a TableSink +Full Stack Example ------------------ -A `TableSink` specifies how to emit a `Table` to an external system or location. The interface is generic such that it can support different storage locations and formats. There are different table sinks for batch tables and streaming tables. +This section sketches how to implement a scan table source with a decoding format that supports changelog +semantics. The example illustrates how all of the mentioned components play together. It can serve as +a reference implementation. -The general interface looks as follows: +In particular, it shows how to +- create factories that parse and validate options, +- implement table connectors, +- implement and discover custom formats, +- and use provided utilities such as data structure converters and the `FactoryUtil`. - -- -The `TableSink#configure` method is called to pass the schema of the Table (field names and types) to emit to the `TableSink`. The method must return a new instance of the TableSink which is configured to emit the provided Table schema. Note that the provided `TableSchema` shouldn't contain computed columns to reflect the schema of the physical `TableSink`. - -### BatchTableSink -Defines an external `TableSink` to emit a batch table. - -The interface looks as follows: - --{% highlight java %} -TableSink- -{ - - public TypeInformation getOutputType(); +The table source uses a simple single-threaded `SourceFunction` to open a socket that listens for incoming +bytes. The raw bytes are decoded into rows by a pluggable format. The format expects a changelog flag +as the first column. - public String[] getFieldNames(); +We will use most of the interfaces metioned above to enable the following DDL: - public TypeInformation[] getFieldTypes(); - - public TableSink configure(String[] fieldNames, TypeInformation[] fieldTypes); -} +{% highlight sql %} +CREATE TABLE UserScores (name STRING, score INT) +WITH ( + 'connector' = 'socket', + 'hostname' = 'localhost', + 'port' = '9999', + 'byte-delimiter' = '10', + 'format' = 'changelog-csv', + 'changelog-csv.column-delimiter' = '|' +); {% endhighlight %} - -{% highlight scala %} -TableSink[T] { - - def getOutputType: TypeInformation-- def getFieldNames: Array[String] +Because the format supports changelog semantics, we are able to ingest updates during runtime and create +an updating view that can continuously evaluate changing data: - def getFieldTypes: Array[TypeInformation] - - def configure(fieldNames: Array[String], fieldTypes: Array[TypeInformation]): TableSink[T] -} +{% highlight sql %} +SELECT name, SUM(score) FROM UserScores GROUP BY name; {% endhighlight %} - -- -{% top %} +This section illustrates how to translate metadata coming from the catalog to concrete connector instances. -### AppendStreamTableSink +Both factories have been added to the `META-INF/services` directory. -Defines an external `TableSink` to emit a streaming table with only insert changes. +**`SocketDynamicTableFactory`** -The interface looks as follows: +The `SocketDynamicTableFactory` translates the catalog table to a table source. Because the table source +requires a decoding format, we are discovering the format using the provided `FactoryUtil` for convenience. --{% highlight java %} -BatchTableSink-implements TableSink { - - public void emitDataSet(DataSet dataSet); -} +Use the following command to ingest data in a terminal: +{% highlight text %} +> nc -lk 9999 +INSERT|Alice|12 +INSERT|Bob|5 +DELETE|Alice|12 +INSERT|Alice|18 {% endhighlight %} - -{% highlight scala %} -BatchTableSink[T] extends TableSink[T] { +### Factories - def emitDataSet(dataSet: DataSet[T]): Unit -} -{% endhighlight %} ---- -If the table is also modified by update or delete changes, a `TableException` will be thrown. -{% top %} +**`ChangelogCsvFormatFactory`** -### RetractStreamTableSink +The `ChangelogCsvFormatFactory` translates format-specific options to a format. The `FactoryUtil` in `SocketDynamicTableFactory` +takes care of adapting the option keys accordingly and handles the prefixing like `changelog-csv.column-delimiter`. -Defines an external `TableSink` to emit a streaming table with insert, update, and delete changes. +Because this factory implements `DeserializationFormatFactory`, it could also be used for other connectors +that support deserialization formats such as the Kafka connector. -The interface looks as follows: - -{% highlight java %} -AppendStreamTableSink+ @Override + public String factoryIdentifier() { + return "socket"; // used for matching to `connector = '...'` + } -implements TableSink { +import org.apache.flink.api.common.serialization.DeserializationSchema; +import org.apache.flink.configuration.ConfigOption; +import org.apache.flink.configuration.ConfigOptions; +import org.apache.flink.configuration.ReadableConfig; +import org.apache.flink.table.connector.format.DecodingFormat; +import org.apache.flink.table.connector.source.DynamicTableSource; +import org.apache.flink.table.data.RowData; +import org.apache.flink.table.factories.DeserializationFormatFactory; +import org.apache.flink.table.factories.DynamicTableSourceFactory; +import org.apache.flink.table.factories.FactoryUtil; +import org.apache.flink.table.types.DataType; + +public class SocketDynamicTableFactory implements DynamicTableSourceFactory { + + // define all options statically + public static final ConfigOption HOSTNAME = ConfigOptions.key("hostname") + .stringType() + .noDefaultValue(); + + public static final ConfigOption PORT = ConfigOptions.key("port") + .intType() + .noDefaultValue(); + + public static final ConfigOption BYTE_DELIMITER = ConfigOptions.key("byte-delimiter") + .intType() + .defaultValue(10); // corresponds to '\n' - public DataStreamSink consumeDataStream(DataStream dataStream); -} -{% endhighlight %} - -{% highlight scala %} -AppendStreamTableSink[T] extends TableSink[T] { + @Override + public Set--- -The table will be converted into a stream of accumulate and retraction messages which are encoded as Java `Tuple2`. The first field is a boolean flag to indicate the message type (`true` indicates insert, `false` indicates delete). The second field holds the record of the requested type `T`. -{% top %} +### Table Source and Decoding Format -### UpsertStreamTableSink +This section illustrates how to translate from instances of the planning layer to runtime instances that +are shipped to the cluster. -Defines an external `TableSink` to emit a streaming table with insert, update, and delete changes. +**`SocketDynamicTableSource`** -The interface looks as follows: +The `SocketDynamicTableSource` is used during planning. In our example, we don't implement any of the +available ability interfaces. Therefore, the main logic can be found in `getScanRuntimeProvider(...)` +where we instantiate the required `SourceFunction` and its `DeserializationSchema` for runtime. Both +instances are parameterized to return internal data structures (i.e. `RowData`). -{% highlight java %} -RetractStreamTableSink+ @Override + public String factoryIdentifier() { + return "changelog-csv"; + } -implements TableSink getRecordType(); +import org.apache.flink.api.common.serialization.DeserializationSchema; +import org.apache.flink.configuration.ConfigOption; +import org.apache.flink.configuration.ConfigOptions; +import org.apache.flink.configuration.ReadableConfig; +import org.apache.flink.table.connector.format.DecodingFormat; +import org.apache.flink.table.data.RowData; +import org.apache.flink.table.factories.FactoryUtil; +import org.apache.flink.table.factories.DeserializationFormatFactory; +import org.apache.flink.table.factories.DynamicTableFactory; + +public class ChangelogCsvFormatFactory implements DeserializationFormatFactory { + + // define all options statically + public static final ConfigOption COLUMN_DELIMITER = ConfigOptions.key("column-delimiter") + .stringType() + .defaultValue("|"); - public DataStreamSink consumeDataStream(DataStream -{% highlight scala %} -RetractStreamTableSink[T] extends TableSink[Tuple2[Boolean, T]] { + @Override + public Set---The table must be have unique key fields (atomic or composite) or be append-only. If the table does not have a unique key and is not append-only, a `TableException` will be thrown. The unique key of the table is configured by the `UpsertStreamTableSink#setKeyFields()` method. +**`ChangelogCsvFormat`** -The table will be converted into a stream of upsert and delete messages which are encoded as a Java `Tuple2`. The first field is a boolean flag to indicate the message type. The second field holds the record of the requested type `T`. +The `ChangelogCsvFormat` is a decoding format that uses a `DeserializationSchema` during runtime. It +supports emitting `INSERT` and `DELETE` changes. -A message with true boolean field is an upsert message for the configured key. A message with false flag is a delete message for the configured key. If the table is append-only, all messages will have a true flag and must be interpreted as insertions. - -{% top %} - -Define a TableFactory ---------------------- - -A `TableFactory` allows to create different table-related instances from string-based properties. All available factories are called for matching to the given set of properties and a corresponding factory class. - -Factories leverage Java's [Service Provider Interfaces (SPI)](https://docs.oracle.com/javase/tutorial/sound/SPI-intro.html) for discovering. This means that every dependency and JAR file should contain a file `org.apache.flink.table.factories.TableFactory` in the `META_INF/services` resource directory that lists all available table factories that it provides. - -Every table factory needs to implement the following interface: - -{% highlight java %} -UpsertStreamTableSink+ // create runtime classes that are shipped to the cluster -implements TableSink getRecordType(); + @Override + public ScanRuntimeProvider getScanRuntimeProvider(ScanContext runtimeProviderContext) { - public DataStreamSink consumeDataStream(DataStream -{% highlight scala %} -UpsertStreamTableSink[T] extends TableSink[Tuple2[Boolean, T]] { + final DeserializationSchema-deserializer = decodingFormat.createRuntimeDecoder( + runtimeProviderContext, + producedDataType); - def setKeyFields(keys: Array[String]): Unit + final SourceFunction sourceFunction = new SocketSourceFunction( + hostname, + port, + byteDelimiter, + deserializer); - def setIsAppendOnly(isAppendOnly: Boolean): Unit + return SourceFunctionProvider.of(sourceFunction, false); + } - def getRecordType: TypeInformation[T] + @Override + public DynamicTableSource copy() { + return new SocketDynamicTableSource(hostname, port, byteDelimiter, decodingFormat, producedDataType); + } - def consumeDataStream(dataStream: DataStream[Tuple2[Boolean, T]]): DataStreamSink[_] + @Override + public String asSummaryString() { + return "Socket Table Source"; + } } {% endhighlight %} - --* `requiredContext()`: Specifies the context that this factory has been implemented for. The framework guarantees to only match for this factory if the specified set of properties and values are met. Typical properties might be `connector.type`, `format.type`, or `update-mode`. Property keys such as `connector.property-version` and `format.property-version` are reserved for future backwards compatibility cases. -* `supportedProperties()`: List of property keys that this factory can handle. This method will be used for validation. If a property is passed that this factory cannot handle, an exception will be thrown. The list must not contain the keys that are specified by the context. +### Runtime -In order to create a specific instance, a factory class can implement one or more interfaces provided in `org.apache.flink.table.factories`: +For completeness, this section illustrates the runtime logic for both `SourceFunction` and `DeserializationSchema`. -* `BatchTableSourceFactory`: Creates a batch table source. -* `BatchTableSinkFactory`: Creates a batch table sink. -* `StreamTableSourceFactory`: Creates a stream table source. -* `StreamTableSinkFactory`: Creates a stream table sink. -* `DeserializationSchemaFactory`: Creates a deserialization schema format. -* `SerializationSchemaFactory`: Creates a serialization schema format. +**ChangelogCsvDeserializer** -The discovery of a factory happens in multiple stages: +The `ChangelogCsvDeserializer` contains a simple parsing logic for converting bytes into `Row` of `Integer` +and `String` with a row kind. The final conversion step converts those into internal data structures. -- Discover all available factories. -- Filter by factory class (e.g., `StreamTableSourceFactory`). -- Filter by matching context. -- Filter by supported properties. -- Verify that exactly one factory matches, otherwise throw an `AmbiguousTableFactoryException` or `NoMatchingTableFactoryException`. - -The following example shows how to provide a custom streaming source with an additional `connector.debug` property flag for parameterization. - -{% highlight java %} -package org.apache.flink.table.factories; - -interface TableFactory { - - Map- -supportedProperties(); -} -{% endhighlight %} - -{% highlight scala %} -package org.apache.flink.table.factories - -trait TableFactory { +import org.apache.flink.api.common.serialization.DeserializationSchema; +import org.apache.flink.api.common.typeinfo.TypeInformation; +import org.apache.flink.table.connector.ChangelogMode; +import org.apache.flink.table.connector.format.DecodingFormat; +import org.apache.flink.table.connector.source.DynamicTableSource; +import org.apache.flink.table.connector.source.DynamicTableSource.DataStructureConverter; +import org.apache.flink.table.data.RowData; +import org.apache.flink.table.types.DataType; +import org.apache.flink.table.types.logical.LogicalType; +import org.apache.flink.types.RowKind; + +public class ChangelogCsvFormat implements DecodingFormat-createRuntimeDecoder( + DynamicTableSource.Context context, + DataType producedDataType) { + // create type information for the DeserializationSchema + final TypeInformation producedTypeInfo = (TypeInformation ) context.createTypeInformation( + producedDataType); + + // most of the code in DeserializationSchema will not work on internal data structures + // create a converter for conversion at the end + final DataStructureConverter converter = context.createDataStructureConverter(producedDataType); + + // use logical types during runtime for parsing + final List parsingTypes = producedDataType.getLogicalType().getChildren(); + + // create runtime class + return new ChangelogCsvDeserializer(parsingTypes, converter, producedTypeInfo, columnDelimiter); + } - def supportedProperties(): util.List[String] + @Override + public ChangelogMode getChangelogMode() { + // define that this format can produce INSERT and DELETE rows + return ChangelogMode.newBuilder() + .addContainedKind(RowKind.INSERT) + .addContainedKind(RowKind.DELETE) + .build(); + } } {% endhighlight %} - -- -{% top %} - -### Use a TableFactory in the SQL Client - -In a SQL Client environment file, the previously presented factory could be declared as: - -{% highlight yaml %} -tables: - - name: MySystemTable - type: source - update-mode: append - connector: - type: my-system - debug: true -{% endhighlight %} - -The YAML file is translated into flattened string properties and a table factory is called with those properties that describe the connection to the external system: - -{% highlight text %} -update-mode=append -connector.type=my-system -connector.debug=true -{% endhighlight %} - -Attention Properties such as `tables.#.name` or `tables.#.type` are SQL Client specifics and are not passed to any factory. The `type` property decides, depending on the execution environment, whether a `BatchTableSourceFactory`/`StreamTableSourceFactory` (for `source`), a `BatchTableSinkFactory`/`StreamTableSinkFactory` (for `sink`), or both (for `both`) need to discovered. - -{% top %} - -### Use a TableFactory in the Table & SQL API - -For a type-safe, programmatic approach with explanatory Scaladoc/Javadoc, the Table & SQL API offers descriptors in `org.apache.flink.table.descriptors` that translate into string-based properties. See the [built-in descriptors](connect.html) for sources, sinks, and formats as a reference. -{% highlight java %} -import org.apache.flink.table.sources.StreamTableSource; +import org.apache.flink.api.common.serialization.DeserializationSchema; +import org.apache.flink.api.common.typeinfo.TypeInformation; +import org.apache.flink.table.connector.RuntimeConverter.Context; +import org.apache.flink.table.connector.source.DynamicTableSource.DataStructureConverter; +import org.apache.flink.table.data.RowData; +import org.apache.flink.table.types.logical.LogicalType; +import org.apache.flink.table.types.logical.LogicalTypeRoot; import org.apache.flink.types.Row; -import java.util.ArrayList; -import java.util.HashMap; -import java.util.List; -import java.util.Map; - -class MySystemTableSourceFactory implements StreamTableSourceFactory- -{ - - @Override - public Map
{ + + private final List parsingTypes; + private final DataStructureConverter converter; + private final TypeInformation producedTypeInfo; + private final String columnDelimiter; + + public ChangelogCsvDeserializer( + List parsingTypes, + DataStructureConverter converter, + TypeInformation producedTypeInfo, + String columnDelimiter) { + this.parsingTypes = parsingTypes; + this.converter = converter; + this.producedTypeInfo = producedTypeInfo; + this.columnDelimiter = columnDelimiter; } @Override - public List supportedProperties() { - List list = new ArrayList<>(); - list.add("connector.debug"); - return list; + public TypeInformation getProducedType() { + // return the type information required by Flink's core interfaces + return producedTypeInfo; } @Override - public StreamTableSource createStreamTableSource(Map
-{% highlight scala %} -import java.util -import org.apache.flink.table.sources.StreamTableSource -import org.apache.flink.types.Row -class MySystemTableSourceFactory extends StreamTableSourceFactory[Row] { - - override def requiredContext(): util.Map[String, String] = { - val context = new util.HashMap[String, String]() - context.put("update-mode", "append") - context.put("connector.type", "my-system") - context + @Override + public RowData deserialize(byte[] message) { + // parse the columns including a changelog flag + final String[] columns = new String(message).split(Pattern.quote(columnDelimiter)); + final RowKind kind = RowKind.valueOf(columns[0]); + final Row row = new Row(kind, parsingTypes.size()); + for (int i = 0; i < parsingTypes.size(); i++) { + row.setField(i, parse(parsingTypes.get(i).getTypeRoot(), columns[i + 1])); + } + // convert to internal data structure + return (RowData) converter.toInternal(row); } - override def supportedProperties(): util.List[String] = { - val properties = new util.ArrayList[String]() - properties.add("connector.debug") - properties + private static Object parse(LogicalTypeRoot root, String value) { + switch (root) { + case INTEGER: + return Integer.parseInt(value); + case VARCHAR: + return value; + default: + throw new IllegalArgumentException(); + } } - override def createStreamTableSource(properties: util.Map[String, String]): StreamTableSource[Row] = { - val isDebug = java.lang.Boolean.valueOf(properties.get("connector.debug")) - - # additional validation of the passed properties can also happen here - - new MySystemAppendTableSource(isDebug) + @Override + public boolean isEndOfStream(RowData nextElement) { + return false; } } {% endhighlight %} ---- {% top %} diff --git a/docs/dev/table/sourceSinks.zh.md b/docs/dev/table/sourceSinks.zh.md index e9f2c775959871..27f7b8e56c661b 100644 --- a/docs/dev/table/sourceSinks.zh.md +++ b/docs/dev/table/sourceSinks.zh.md @@ -22,796 +22,725 @@ specific language governing permissions and limitations under the License. --> -A `TableSource` provides access to data which is stored in external systems (database, key-value store, message queue) or files. After a [TableSource is registered in a TableEnvironment](common.html#register-a-tablesource) it can be accessed by [Table API](tableApi.html) or [SQL]({{ site.baseurl }}/dev/table/sql/queries.html) queries. +_Dynamic tables_ are the core concept of Flink's Table & SQL API for processing both bounded and unbounded +data in a unified fashion. -A `TableSink` [emits a Table](common.html#emit-a-table) to an external storage system, such as a database, key-value store, message queue, or file system (in different encodings, e.g., CSV, Parquet, or ORC). +Because dynamic tables are only a logical concept, Flink does not own the data itself. Instead, the content +of a dynamic table is stored in external systems (such as databases, key-value stores, message queues) or files. -A `TableFactory` allows for separating the declaration of a connection to an external system from the actual implementation. A table factory creates configured instances of table sources and sinks from normalized, string-based properties. The properties can be generated programmatically using a `Descriptor` or via YAML configuration files for the [SQL Client](sqlClient.html). +_Dynamic sources_ and _dynamic sinks_ can be used to read and write data from and to an external system. In +the documentation, sources and sinks are often summarized under the term _connector_. -Have a look at the [common concepts and API](common.html) page for details how to [register a TableSource](common.html#register-a-tablesource) and how to [emit a Table through a TableSink](common.html#emit-a-table). See the [built-in sources, sinks, and formats](connect.html) page for examples how to use factories. +Flink provides pre-defined connectors for Kafka, Hive, and different file systems. See the [connector section]({% link dev/table/connectors/index.zh.md %}) +for more information about built-in table sources and sinks. -* This will be replaced by the TOC -{:toc} +This page focuses on how to develop a custom, user-defined connector. -Define a TableSource --------------------- +Attention New table source and table sink interfaces have been +introduced in Flink 1.11 as part of [FLIP-95](https://cwiki.apache.org/confluence/display/FLINK/FLIP-95%3A+New+TableSource+and+TableSink+interfaces). +Also the factory interfaces have been reworked. FLIP-95 is not fully implemented yet. Many ability interfaces +are not supported yet (e.g. for filter or partition push down). If necessary, please also have a look +at the [old table sources and sinks page]({% link dev/table/legacySourceSinks.zh.md %}). Those interfaces +are still supported for backwards compatibility. -A `TableSource` is a generic interface that gives Table API and SQL queries access to data stored in an external system. It provides the schema of the table and the records that are mapped to rows with the table's schema. Depending on whether the `TableSource` is used in a streaming or batch query, the records are produced as a `DataSet` or `DataStream`. +* This will be replaced by the TOC +{:toc} -If a `TableSource` is used in a streaming query it must implement the `StreamTableSource` interface, if it is used in a batch query it must implement the `BatchTableSource` interface. A `TableSource` can also implement both interfaces and be used in streaming and batch queries. +Overview +-------- -`StreamTableSource` and `BatchTableSource` extend the base interface `TableSource` that defines the following methods: +In many cases, implementers don't need to create a new connector from scratch but would like to slightly +modify existing connectors or hook into the existing stack. In other cases, implementers would like to +create specialized connectors. -+**SocketSourceFunction** -A custom descriptor can be defined by extending the `ConnectorDescriptor` class. +The `SocketSourceFunction` opens a socket and consumes bytes. It splits records by the given byte +delimiter (`\n` by default) and delegates the decoding to a pluggable `DeserializationSchema`. The +source function can only work with a parallelism of 1. {% highlight java %} -import org.apache.flink.table.descriptors.ConnectorDescriptor; -import java.util.HashMap; -import java.util.Map; - -/** - * Connector to MySystem with debug mode. - */ -public class MySystemConnector extends ConnectorDescriptor { - - public final boolean isDebug; - - public MySystemConnector(boolean isDebug) { - super("my-system", 1, false); - this.isDebug = isDebug; +import org.apache.flink.api.common.serialization.DeserializationSchema; +import org.apache.flink.api.common.typeinfo.TypeInformation; +import org.apache.flink.api.java.typeutils.ResultTypeQueryable; +import org.apache.flink.configuration.Configuration; +import org.apache.flink.streaming.api.functions.source.RichSourceFunction; +import org.apache.flink.table.data.RowData; + +public class SocketSourceFunction extends RichSourceFunction- -implements ResultTypeQueryable { + + private final String hostname; + private final int port; + private final byte byteDelimiter; + private final DeserializationSchema deserializer; + + private volatile boolean isRunning = true; + private Socket currentSocket; + + public SocketSourceFunction(String hostname, int port, byte byteDelimiter, DeserializationSchema deserializer) { + this.hostname = hostname; + this.port = port; + this.byteDelimiter = byteDelimiter; + this.deserializer = deserializer; } @Override - protected Map getProducedType() { + return deserializer.getProducedType(); } -} -{% endhighlight %} - -The descriptor can then be used to create a table with the table environment. -{% highlight java %} -StreamTableEnvironment tableEnv = // ... - -tableEnv - .connect(new MySystemConnector(true)) - .withSchema(...) - .inAppendMode() - .createTemporaryTable("MySystemTable"); -{% endhighlight %} - - - -A custom descriptor can be defined by extending the `ConnectorDescriptor` class. + @Override + public void open(Configuration parameters) throws Exception { + deserializer.open(() -> getRuntimeContext().getMetricGroup()); + } -{% highlight scala %} -import org.apache.flink.table.descriptors.ConnectorDescriptor -import java.util.HashMap -import java.util.Map + @Override + public void run(SourceContext- -ctx) throws Exception { + while (isRunning) { + // open and consume from socket + try (final Socket socket = new Socket()) { + currentSocket = socket; + socket.connect(new InetSocketAddress(hostname, port), 0); + try (InputStream stream = socket.getInputStream()) { + ByteArrayOutputStream buffer = new ByteArrayOutputStream(); + int b; + while ((b = stream.read()) >= 0) { + // buffer until delimiter + if (b != byteDelimiter) { + buffer.write(b); + } + // decode and emit record + else { + ctx.collect(deserializer.deserialize(buffer.toByteArray())); + buffer.reset(); + } + } + } + } catch (Throwable t) { + t.printStackTrace(); // print and continue + } + Thread.sleep(1000); + } + } -/** - * Connector to MySystem with debug mode. - */ -class MySystemConnector(isDebug: Boolean) extends ConnectorDescriptor("my-system", 1, false) { - - override protected def toConnectorProperties(): Map[String, String] = { - val properties = new HashMap[String, String] - properties.put("connector.debug", isDebug.toString) - properties + @Override + public void cancel() { + isRunning = false; + try { + currentSocket.close(); + } catch (Throwable t) { + // ignore + } } } {% endhighlight %} -The descriptor can then be used to create a table with the table environment. - -{% highlight scala %} -val tableEnv: StreamTableEnvironment = // ... - -tableEnv - .connect(new MySystemConnector(isDebug = true)) - .withSchema(...) - .inAppendMode() - .createTemporaryTable("MySystemTable") -{% endhighlight %} - - - -You can use a Java `TableFactory` from Python using the `CustomConnectorDescriptor`. - -{% highlight python %} -s_env = StreamExecutionEnvironment.get_execution_environment() -st_env = StreamTableEnvironment.create(s_env) - -custom_connector = CustomConnectorDescriptor('my-system', 1, False) -st_env\ - .connect(custom_connector.property("connector.debug", "true")) \ - .with_schema(...) \ - .in_append_mode()\ - .create_temporary_table("MySystemTable") -{% endhighlight %} -- --@@ -65,32 +67,36 @@ val settings = EnvironmentSettings.newInstance()... val tableEnv = TableEnvironment.create(settings) // register a table named "Orders" -tableEnv.sqlUpdate("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)"); +tableEnv.executeSql("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)"); // a string array: ["Orders"] -val tables = tableEnv.listTable() +val tables = tableEnv.listTables() +// or tableEnv.executeSql("SHOW TABLES").print() // rename "Orders" to "NewOrders" -tableEnv.sqlUpdate("ALTER TABLE Orders RENAME TO NewOrders;") +tableEnv.executeSql("ALTER TABLE Orders RENAME TO NewOrders;") // a string array: ["NewOrders"] -val tables = tableEnv.listTable() +val tables = tableEnv.listTables() +// or tableEnv.executeSql("SHOW TABLES").print() {% endhighlight %}-{% highlight java %} -TableSource- -* `getTableSchema()`: Returns the schema of the produced table, i.e., the names and types of the fields of the table. The field types are defined using Flink's `DataType` (see [Table API types]({{ site.baseurl }}/dev/table/types.html) and [SQL types]({{ site.baseurl }}/dev/table/sql/index.html#data-types)). Note that the returned `TableSchema` shouldn't contain computed columns to reflect the schema of the physical `TableSource`. +The metadata for dynamic tables (created via DDL or provided by the catalog) is represented as instances +of `CatalogTable`. A table name will be resolved into a `CatalogTable` internally when necessary. -* `getReturnType()`: Returns the physical type of the `DataStream` (`StreamTableSource`) or `DataSet` (`BatchTableSource`) and the records that are produced by the `TableSource`. +### Planning -* `explainSource()`: Returns a String that describes the `TableSource`. This method is optional and used for display purposes only. +When it comes to planning and optimization of the table program, a `CatalogTable` needs to be resolved +into a `DynamicTableSource` (for reading in a `SELECT` query) and `DynamicTableSink` (for writing in +an `INSERT INTO` statement). -The `TableSource` interface separates the logical table schema from the physical type of the returned `DataStream` or `DataSet`. As a consequence, all fields of the table schema (`getTableSchema()`) must be mapped to a field with corresponding type of the physical return type (`getReturnType()`). By default, this mapping is done based on field names. For example, a `TableSource` that defines a table schema with two fields `[name: String, size: Integer]` requires a `TypeInformation` with at least two fields called `name` and `size` of type `String` and `Integer`, respectively. This could be a `PojoTypeInfo` or a `RowTypeInfo` that have two fields named `name` and `size` with matching types. +`DynamicTableSourceFactory` and `DynamicTableSinkFactory` provide connector-specific logic for translating +the metadata of a `CatalogTable` into instances of `DynamicTableSource` and `DynamicTableSink`. In most +of the cases, a factory's purpose is to validate options (such as `'port' = '5022'` in the example), +configure encoding/decoding formats (if required), and create a parameterized instance of the table +connector. -However, some types, such as Tuple or CaseClass types, do support custom field names. If a `TableSource` returns a `DataStream` or `DataSet` of a type with fixed field names, it can implement the `DefinedFieldMapping` interface to map field names from the table schema to field names of the physical return type. +By default, instances of `DynamicTableSourceFactory` and `DynamicTableSinkFactory` are discovered using +Java's [Service Provider Interfaces (SPI)](https://docs.oracle.com/javase/tutorial/sound/SPI-intro.html). The +`connector` option (such as `'connector' = 'custom'` in the example) must correspond to a valid factory +identifier. -### Defining a BatchTableSource +Although it might not be apparent in the class naming, `DynamicTableSource` and `DynamicTableSink` +can also be seen as stateful factories that eventually produce concrete runtime implementation for reading/writing +the actual data. -The `BatchTableSource` interface extends the `TableSource` interface and defines one additional method: +The planner uses the source and sink instances to perform connector-specific bidirectional comunication +until an optimal logical plan could be found. Depending on the optionally declared ability interfaces (e.g. +`SupportsProjectionPushDown` or `SupportsOverwrite`), the planner might apply changes to an instance and +thus mutate the produced runtime implementation. -{ +This section helps for both kinds of use cases. It explains the general architecture of table connectors +from pure declaration in the API to runtime code that will be executed on the cluster. - public TableSchema getTableSchema(); +The filled arrows show how objects are transformed to other objects from one stage to the next stage during +the translation process. - public TypeInformation getReturnType(); - - public String explainSource(); -} -{% endhighlight %} + +-
-{% highlight scala %} -TableSource[T] { - - def getTableSchema: TableSchema +### Metadata - def getReturnType: TypeInformation[T] +Both Table API and SQL are declarative APIs. This includes the declaration of tables. Thus, executing +a `CREATE TABLE` statement results in updated metadata in the target catalog. - def explainSource: String +For most catalog implementations, physical data in the external system is not modified for such an +operation. Connector-specific dependencies don't have to be present in the classpath yet. The options declared +in the `WITH` clause are neither validated nor otherwise interpreted. -} -{% endhighlight %} ---+Those interfaces are grouped by another level of abstraction as subclasses of `ScanRuntimeProvider`, +`LookupRuntimeProvider`, and `SinkRuntimeProvider`. -* `getDataSet(execEnv)`: Returns a `DataSet` with the data of the table. The type of the `DataSet` must be identical to the return type defined by the `TableSource.getReturnType()` method. The `DataSet` can by created using a regular [data source]({{ site.baseurl }}/dev/batch/#data-sources) of the DataSet API. Commonly, a `BatchTableSource` is implemented by wrapping a `InputFormat` or [batch connector]({{ site.baseurl }}/dev/batch/connectors.html). +For example, both `OutputFormatProvider` (providing `org.apache.flink.api.common.io.OutputFormat`) and `SinkFunctionProvider` (providing `org.apache.flink.streaming.api.functions.sink.SinkFunction`) are concrete instances of `SinkRuntimeProvider` +that the planner can handle. {% top %} -### Defining a StreamTableSource +Extension Points +---------------- -The `StreamTableSource` interface extends the `TableSource` interface and defines one additional method: +This section explains the available interfaces for extending Flink's table connectors. --{% highlight java %} -BatchTableSource+### Runtime -implements TableSource { - - public DataSet getDataSet(ExecutionEnvironment execEnv); -} -{% endhighlight %} - -{% highlight scala %} -BatchTableSource[T] extends TableSource[T] { +Once the logical planning is complete, the planner will obtain the _runtime implementation_ from the table +connector. Runtime logic is implemented in Flink's core connector interfaces such as `InputFormat` or `SourceFunction`. - def getDataSet(execEnv: ExecutionEnvironment): DataSet[T] -} -{% endhighlight %} ---+`org.apache.flink.table.factories.DynamicTableSinkFactory` can be implemented to construct a `DynamicTableSink`. -* `getDataStream(execEnv)`: Returns a `DataStream` with the data of the table. The type of the `DataStream` must be identical to the return type defined by the `TableSource.getReturnType()` method. The `DataStream` can by created using a regular [data source]({{ site.baseurl }}/dev/datastream_api.html#data-sources) of the DataStream API. Commonly, a `StreamTableSource` is implemented by wrapping a `SourceFunction` or a [stream connector]({{ site.baseurl }}/dev/connectors/). +By default, the factory is discovered using the value of the `connector` option as the factory identifier +and Java's Service Provider Interface. -{% top %} +In JAR files, references to new implementations can be added to the service file: -### Defining a TableSource with Time Attributes +`META-INF/services/org.apache.flink.table.factories.Factory` -Time-based operations of streaming [Table API](tableApi.html#group-windows) and [SQL]({{ site.baseurl }}/dev/table/sql/queries.html#group-windows) queries, such as windowed aggregations or joins, require explicitly specified [time attributes](streaming/time_attributes.html). +The framework will check for a single matching factory that is uniquely identified by factory identifier +and requested base class (e.g. `DynamicTableSourceFactory`). -A `TableSource` defines a time attribute as a field of type `Types.SQL_TIMESTAMP` in its table schema. In contrast to all regular fields in the schema, a time attribute must not be matched to a physical field in the return type of the table source. Instead, a `TableSource` defines a time attribute by implementing a certain interface. +The factory discovery process can be bypassed by the catalog implementation if necessary. For this, a +catalog needs to return an instance that implements the requested base class in `org.apache.flink.table.catalog.Catalog#getFactory`. -#### Defining a Processing Time Attribute +### Dynamic Table Source -[Processing time attributes](streaming/time_attributes.html#processing-time) are commonly used in streaming queries. A processing time attribute returns the current wall-clock time of the operator that accesses it. A `TableSource` defines a processing time attribute by implementing the `DefinedProctimeAttribute` interface. The interface looks as follows: +By definition, a dynamic table can change over time. --{% highlight java %} -StreamTableSource+Dynamic table factories are used to configure a dynamic table connector for an external storage system from catalog +and session information. -implements TableSource { +### Dynamic Table Factories - public DataStream getDataStream(StreamExecutionEnvironment execEnv); -} -{% endhighlight %} - -{% highlight scala %} -StreamTableSource[T] extends TableSource[T] { +`org.apache.flink.table.factories.DynamicTableSourceFactory` can be implemented to construct a `DynamicTableSource`. - def getDataStream(execEnv: StreamExecutionEnvironment): DataStream[T] -} -{% endhighlight %} ---+A `ScanTableSource` scans all rows from an external storage system during runtime. -* `getProctimeAttribute()`: Returns the name of the processing time attribute. The specified attribute must be defined of type `Types.SQL_TIMESTAMP` in the table schema and can be used in time-based operations. A `DefinedProctimeAttribute` table source can define no processing time attribute by returning `null`. +The scanned rows don't have to contain only insertions but can also contain updates and deletions. Thus, +the table source can be used to read a (finite or infinite) changelog. The returned _changelog mode_ indicates +the set of changes that the planner can expect during runtime. -Attention Both `StreamTableSource` and `BatchTableSource` can implement `DefinedProctimeAttribute` and define a processing time attribute. In case of a `BatchTableSource` the processing time field is initialized with the current timestamp during the table scan. +For regular batch scenarios, the source can emit a bounded stream of insert-only rows. -#### Defining a Rowtime Attribute +For regular streaming scenarios, the source can emit an unbounded stream of insert-only rows. -[Rowtime attributes](streaming/time_attributes.html#event-time) are attributes of type `TIMESTAMP` and handled in a unified way in stream and batch queries. +For change data capture (CDC) scenarios, the source can emit bounded or unbounded streams with insert, +update, and delete rows. -A table schema field of type `SQL_TIMESTAMP` can be declared as rowtime attribute by specifying +A table source can implement further abilitiy interfaces such as `SupportsProjectionPushDown` that might +mutate an instance during planning. All abilities are listed in the `org.apache.flink.table.connector.source.abilities` +package and in the documentation of `org.apache.flink.table.connector.source.ScanTableSource`. -* the name of the field, -* a `TimestampExtractor` that computes the actual value for the attribute (usually from one or more other fields), and -* a `WatermarkStrategy` that specifies how watermarks are generated for the the rowtime attribute. +The runtime implementation of a `ScanTableSource` must produce internal data structures. Thus, records +must be emitted as `org.apache.flink.table.data.RowData`. The framework provides runtime converters such +that a source can still work on common data structures and perform a conversion at the end. -A `TableSource` defines a rowtime attribute by implementing the `DefinedRowtimeAttributes` interface. The interface looks as follows: +#### Lookup Table Source --{% highlight java %} -DefinedProctimeAttribute { +When reading a dynamic table, the content can either be considered as: +- A changelog (finite or infinite) for which all changes are consumed continuously until the changelog + is exhausted. This is represented by the `ScanTableSource` interface. +- A continuously changing or very large external table whose content is usually never read entirely + but queried for individual values when necessary. This is represented by the `LookupTableSource` + interface. - public String getProctimeAttribute(); -} -{% endhighlight %} -+A class can implement both of these interfaces at the same time. The planner decides about their usage depending +on the specified query. --{% highlight scala %} -DefinedProctimeAttribute { +#### Scan Table Source - def getProctimeAttribute: String -} -{% endhighlight %} ---+Compared to `ScanTableSource`, a `LookupTableSource` does only support emitting insert-only changes currently. -* `getRowtimeAttributeDescriptors()`: Returns a list of `RowtimeAttributeDescriptor`. A `RowtimeAttributeDescriptor` describes a rowtime attribute with the following properties: - * `attributeName`: The name of the rowtime attribute in the table schema. The field must be defined with type `Types.SQL_TIMESTAMP`. - * `timestampExtractor`: The timestamp extractor extracts the timestamp from a record with the return type. For example, it can convert a Long field into a timestamp or parse a String-encoded timestamp. Flink comes with a set of built-in `TimestampExtractor` implementation for common use cases. It is also possible to provide a custom implementation. - * `watermarkStrategy`: The watermark strategy defines how watermarks are generated for the rowtime attribute. Flink comes with a set of built-in `WatermarkStrategy` implementations for common use cases. It is also possible to provide a custom implementation. +Further abilities are not supported. See the documentation of `org.apache.flink.table.connector.source.LookupTableSource` +for more information. -Attention Although the `getRowtimeAttributeDescriptors()` method returns a list of descriptors, only a single rowtime attribute is support at the moment. We plan to remove this restriction in the future and support tables with more than one rowtime attribute. +The runtime implementation of a `LookupTableSource` is a `TableFunction` or `AsyncTableFunction`. The function +will be called with values for the given lookup keys during runtime. -Attention Both, `StreamTableSource` and `BatchTableSource`, can implement `DefinedRowtimeAttributes` and define a rowtime attribute. In either case, the rowtime field is extracted using the `TimestampExtractor`. Hence, a `TableSource` that implements `StreamTableSource` and `BatchTableSource` and defines a rowtime attribute provides exactly the same data to streaming and batch queries. +### Dynamic Table Sink -##### Provided Timestamp Extractors +By definition, a dynamic table can change over time. -Flink provides `TimestampExtractor` implementations for common use cases. +When writing a dynamic table, the content can always be considered as a changelog (finite or infinite) +for which all changes are written out continuously until the changelog is exhausted. The returned _changelog mode_ +indicates the set of changes that the sink accepts during runtime. -The following `TimestampExtractor` implementations are currently available: +For regular batch scenarios, the sink can solely accept insert-only rows and write out bounded streams. -* `ExistingField(fieldName)`: Extracts the value of a rowtime attribute from an existing `LONG`, `SQL_TIMESTAMP`, or timestamp formatted `STRING` field. One example of such a string would be '2018-05-28 12:34:56.000'. -* `StreamRecordTimestamp()`: Extracts the value of a rowtime attribute from the timestamp of the `DataStream` `StreamRecord`. Note, this `TimestampExtractor` is not available for batch table sources. +For regular streaming scenarios, the sink can solely accept insert-only rows and can write out unbounded streams. -A custom `TimestampExtractor` can be defined by implementing the corresponding interface. +For change data capture (CDC) scenarios, the sink can write out bounded or unbounded streams with insert, +update, and delete rows. -##### Provided Watermark Strategies +A table sink can implement further abilitiy interfaces such as `SupportsOverwrite` that might mutate an +instance during planning. All abilities are listed in the `org.apache.flink.table.connector.sink.abilities` +package and in the documentation of `org.apache.flink.table.connector.sink.DynamicTableSink`. -Flink provides `WatermarkStrategy` implementations for common use cases. +The runtime implementation of a `DynamicTableSink` must consume internal data structures. Thus, records +must be accepted as `org.apache.flink.table.data.RowData`. The framework provides runtime converters such +that a sink can still work on common data structures and perform a conversion at the beginning. -The following `WatermarkStrategy` implementations are currently available: +### Encoding / Decoding Formats -* `AscendingTimestamps`: A watermark strategy for ascending timestamps. Records with timestamps that are out-of-order will be considered late. -* `BoundedOutOfOrderTimestamps(delay)`: A watermark strategy for timestamps that are at most out-of-order by the specified delay. -* `PreserveWatermarks()`: A strategy which indicates the watermarks should be preserved from the underlying `DataStream`. +Some table connectors accept different formats that encode and decode keys and/or values. -A custom `WatermarkStrategy` can be defined by implementing the corresponding interface. +Formats work similar to the pattern `DynamicTableSourceFactory -> DynamicTableSource -> ScanRuntimeProvider`, +where the factory is responsible for translating options and the source is responsible for creating runtime logic. -{% top %} +Because formats might be located in different modules, they are discovered using Java's Service Provider +Interface similar to [table factories](#dynamic-table-factories). In order to discover a format factory, +the dynamic table factory searches for a factory that corresponds to a factory identifier and connector-specific +base class. -### Defining a TableSource with Projection Push-Down +For example, the Kafka table source requires a `DeserializationSchema` as runtime interface for a decoding +format. Therefore, the Kafka table source factory uses the value of the `value.format` option to discover +a `DeserializationFormatFactory`. -A `TableSource` supports projection push-down by implementing the `ProjectableTableSource` interface. The interface defines a single method: +The following format factories are currently supported: --{% highlight java %} -DefinedRowtimeAttribute { - - public List+A `LookupTableSource` looks up rows of an external storage system by one or more keys during runtime. -getRowtimeAttributeDescriptors(); -} -{% endhighlight %} - -{% highlight scala %} -DefinedRowtimeAttributes { +Compared to `ScanTableSource`, the source does not have to read the entire table and can lazily fetch individual +values from a (possibly continuously changing) external table when necessary. - def getRowtimeAttributeDescriptors: util.List[RowtimeAttributeDescriptor] -} -{% endhighlight %} ---- -* `projectFields(fields)`: Returns a *copy* of the `TableSource` with adjusted physical return type. The `fields` parameter provides the indexes of the fields that must be provided by the `TableSource`. The indexes relate to the `TypeInformation` of the physical return type, *not* to the logical table schema. The copied `TableSource` must adjust its return type and the returned `DataStream` or `DataSet`. The `TableSchema` of the copied `TableSource` must not be changed, i.e, it must be the same as the original `TableSource`. If the `TableSource` implements the `DefinedFieldMapping` interface, the field mapping must be adjusted to the new return type. - -Attention In order for Flink to distinguish a projection push-down table source from its original form, `explainSource` method must be override to include information regarding the projected fields. - -The `ProjectableTableSource` adds support to project flat fields. If the `TableSource` defines a table with nested schema, it can implement the `NestedFieldsProjectableTableSource` to extend the projection to nested fields. The `NestedFieldsProjectableTableSource` is defined as follows: - --{% highlight java %} -ProjectableTableSource+The format factory translates the options into an `EncodingFormat` or a `DecodingFormat`. Those interfaces are +another kind of factory that produce specialized format runtime logic for the given data type. -{ +``` +org.apache.flink.table.factories.DeserializationFormatFactory +org.apache.flink.table.factories.SerializationFormatFactory +``` - public TableSource projectFields(int[] fields); -} -{% endhighlight %} - -{% highlight scala %} -ProjectableTableSource[T] { - - def projectFields(fields: Array[Int]): TableSource[T] -} -{% endhighlight %} ---- -* `projectNestedField(fields, nestedFields)`: Returns a *copy* of the `TableSource` with adjusted physical return type. Fields of the physical return type may be removed or reordered but their type must not be changed. The contract of this method is essentially the same as for the `ProjectableTableSource.projectFields()` method. In addition, the `nestedFields` parameter contains for each field index in the `fields` list, a list of paths to all nested fields that are accessed by the query. All other nested fields do not need to be read, parsed, and set in the records that are produced by the `TableSource`. - -Attention the types of the projected fields must not be changed but unused fields may be set to null or to a default value. - -{% top %} - -### Defining a TableSource with Filter Push-Down - -The `FilterableTableSource` interface adds support for filter push-down to a `TableSource`. A `TableSource` extending this interface is able to filter records such that the returned `DataStream` or `DataSet` returns fewer records. - -The interface looks as follows: - --{% highlight java %} -NestedFieldsProjectableTableSource- -{ - - public TableSource projectNestedFields(int[] fields, String[][] nestedFields); -} -{% endhighlight %} - -{% highlight scala %} -NestedFieldsProjectableTableSource[T] { - - def projectNestedFields(fields: Array[Int], nestedFields: Array[Array[String]]): TableSource[T] -} -{% endhighlight %} ---- -* `applyPredicate(predicates)`: Returns a *copy* of the `TableSource` with added predicates. The `predicates` parameter is a mutable list of conjunctive predicates that are "offered" to the `TableSource`. The `TableSource` accepts to evaluate a predicate by removing it from the list. Predicates that are left in the list will be evaluated by a subsequent filter operator. -* `isFilterPushedDown()`: Returns true if the `applyPredicate()` method was called before. Hence, `isFilterPushedDown()` must return true for all `TableSource` instances returned from a `applyPredicate()` call. - -Attention In order for Flink to distinguish a filter push-down table source from its original form, `explainSource` method must be override to include information regarding the push-down filters. - -{% top %} - -### Defining a TableSource for Lookups - -Attention This is an experimental feature. The interface may be changed in future versions. It's only supported in Blink planner. - -The `LookupableTableSource` interface adds support for the table to be accessed via key column(s) in a lookup fashion. This is very useful when used to join with a dimension table to enrich some information. If you want to use the `TableSource` in lookup mode, you should use the source in [temporal table join syntax](streaming/joins.html). - -The interface looks as follows: - --{% highlight java %} -FilterableTableSource- -{ - - public TableSource applyPredicate(List predicates); - - public boolean isFilterPushedDown(); -} -{% endhighlight %} - -{% highlight scala %} -FilterableTableSource[T] { - - def applyPredicate(predicates: java.util.List[Expression]): TableSource[T] - - def isFilterPushedDown: Boolean -} -{% endhighlight %} ---- -* `getLookupFunction(lookupkeys)`: Returns a `TableFunction` which used to lookup the matched row(s) via lookup keys. The lookupkeys are the field names of `LookupableTableSource` in the join equal conditions. The eval method parameters of the returned `TableFunction`'s should be in the order which `lookupkeys` defined. It is recommended to define the parameters in varargs (e.g. `eval(Object... lookupkeys)` to match all the cases). The return type of the `TableFunction` must be identical to the return type defined by the `TableSource.getReturnType()` method. -* `getAsyncLookupFunction(lookupkeys)`: Optional. Similar to `getLookupFunction`, but the `AsyncLookupFunction` lookups the matched row(s) asynchronously. The underlying of `AsyncLookupFunction` will be called via [Async I/O]({{ site.baseurl }}/dev/stream/operators/asyncio.html). The first argument of the eval method of the returned `AsyncTableFunction` should be defined as `java.util.concurrent.CompletableFuture` to collect results asynchronously (e.g. `eval(CompletableFuture-{% highlight java %} -LookupableTableSource- -implements TableSource { - - public TableFunction getLookupFunction(String[] lookupkeys); - - public AsyncTableFunction getAsyncLookupFunction(String[] lookupkeys); - - public boolean isAsyncEnabled(); -} -{% endhighlight %} - -{% highlight scala %} -LookupableTableSource[T] extends TableSource[T] { - - def getLookupFunction(lookupKeys: Array[String]): TableFunction[T] - - def getAsyncLookupFunction(lookupKeys: Array[String]): AsyncTableFunction[T] - - def isAsyncEnabled: Boolean -} -{% endhighlight %} --` +that can be passed into the Kafka table source. {% top %} -Define a TableSink +Full Stack Example ------------------ -A `TableSink` specifies how to emit a `Table` to an external system or location. The interface is generic such that it can support different storage locations and formats. There are different table sinks for batch tables and streaming tables. +This section sketches how to implement a scan table source with a decoding format that supports changelog +semantics. The example illustrates how all of the mentioned components play together. It can serve as +a reference implementation. -The general interface looks as follows: +In particular, it shows how to +- create factories that parse and validate options, +- implement table connectors, +- implement and discover custom formats, +- and use provided utilities such as data structure converters and the `FactoryUtil`. - -- -The `TableSink#configure` method is called to pass the schema of the Table (field names and types) to emit to the `TableSink`. The method must return a new instance of the TableSink which is configured to emit the provided Table schema. Note that the provided `TableSchema` shouldn't contain computed columns to reflect the schema of the physical `TableSink`. - -### BatchTableSink -Defines an external `TableSink` to emit a batch table. - -The interface looks as follows: - --{% highlight java %} -TableSink- -{ - - public TypeInformation getOutputType(); +The table source uses a simple single-threaded `SourceFunction` to open a socket that listens for incoming +bytes. The raw bytes are decoded into rows by a pluggable format. The format expects a changelog flag +as the first column. - public String[] getFieldNames(); +We will use most of the interfaces metioned above to enable the following DDL: - public TypeInformation[] getFieldTypes(); - - public TableSink configure(String[] fieldNames, TypeInformation[] fieldTypes); -} +{% highlight sql %} +CREATE TABLE UserScores (name STRING, score INT) +WITH ( + 'connector' = 'socket', + 'hostname' = 'localhost', + 'port' = '9999', + 'byte-delimiter' = '10', + 'format' = 'changelog-csv', + 'changelog-csv.column-delimiter' = '|' +); {% endhighlight %} - -{% highlight scala %} -TableSink[T] { - - def getOutputType: TypeInformation-- def getFieldNames: Array[String] +Because the format supports changelog semantics, we are able to ingest updates during runtime and create +an updating view that can continuously evaluate changing data: - def getFieldTypes: Array[TypeInformation] - - def configure(fieldNames: Array[String], fieldTypes: Array[TypeInformation]): TableSink[T] -} +{% highlight sql %} +SELECT name, SUM(score) FROM UserScores GROUP BY name; {% endhighlight %} - -+### Factories -{% top %} +This section illustrates how to translate metadata coming from the catalog to concrete connector instances. -### AppendStreamTableSink +Both factories have been added to the `META-INF/services` directory. -Defines an external `TableSink` to emit a streaming table with only insert changes. +**`SocketDynamicTableFactory`** -The interface looks as follows: +The `SocketDynamicTableFactory` translates the catalog table to a table source. Because the table source +requires a decoding format, we are discovering the format using the provided `FactoryUtil` for convenience. --{% highlight java %} -BatchTableSink- -implements TableSink { - - public void emitDataSet(DataSet dataSet); -} +Use the following command to ingest data in a terminal: +{% highlight text %} +> nc -lk 9999 +INSERT|Alice|12 +INSERT|Bob|5 +DELETE|Alice|12 +INSERT|Alice|18 {% endhighlight %} - -{% highlight scala %} -BatchTableSink[T] extends TableSink[T] { - def emitDataSet(dataSet: DataSet[T]): Unit -} -{% endhighlight %} ---- -If the table is also modified by update or delete changes, a `TableException` will be thrown. -{% top %} +**`ChangelogCsvFormatFactory`** -### RetractStreamTableSink +The `ChangelogCsvFormatFactory` translates format-specific options to a format. The `FactoryUtil` in `SocketDynamicTableFactory` +takes care of adapting the option keys accordingly and handles the prefixing like `changelog-csv.column-delimiter`. -Defines an external `TableSink` to emit a streaming table with insert, update, and delete changes. +Because this factory implements `DeserializationFormatFactory`, it could also be used for other connectors +that support deserialization formats such as the Kafka connector. -The interface looks as follows: - -{% highlight java %} -AppendStreamTableSink+ @Override + public String factoryIdentifier() { + return "socket"; // used for matching to `connector = '...'` + } -implements TableSink { +import org.apache.flink.api.common.serialization.DeserializationSchema; +import org.apache.flink.configuration.ConfigOption; +import org.apache.flink.configuration.ConfigOptions; +import org.apache.flink.configuration.ReadableConfig; +import org.apache.flink.table.connector.format.DecodingFormat; +import org.apache.flink.table.connector.source.DynamicTableSource; +import org.apache.flink.table.data.RowData; +import org.apache.flink.table.factories.DeserializationFormatFactory; +import org.apache.flink.table.factories.DynamicTableSourceFactory; +import org.apache.flink.table.factories.FactoryUtil; +import org.apache.flink.table.types.DataType; + +public class SocketDynamicTableFactory implements DynamicTableSourceFactory { + + // define all options statically + public static final ConfigOption HOSTNAME = ConfigOptions.key("hostname") + .stringType() + .noDefaultValue(); + + public static final ConfigOption PORT = ConfigOptions.key("port") + .intType() + .noDefaultValue(); + + public static final ConfigOption BYTE_DELIMITER = ConfigOptions.key("byte-delimiter") + .intType() + .defaultValue(10); // corresponds to '\n' - public DataStreamSink consumeDataStream(DataStream dataStream); -} -{% endhighlight %} - -{% highlight scala %} -AppendStreamTableSink[T] extends TableSink[T] { + @Override + public Set---The table will be converted into a stream of accumulate and retraction messages which are encoded as Java `Tuple2`. The first field is a boolean flag to indicate the message type (`true` indicates insert, `false` indicates delete). The second field holds the record of the requested type `T`. +### Table Source and Decoding Format -{% top %} - -### UpsertStreamTableSink +This section illustrates how to translate from instances of the planning layer to runtime instances that +are shipped to the cluster. -Defines an external `TableSink` to emit a streaming table with insert, update, and delete changes. +**`SocketDynamicTableSource`** -The interface looks as follows: +The `SocketDynamicTableSource` is used during planning. In our example, we don't implement any of the +available ability interfaces. Therefore, the main logic can be found in `getScanRuntimeProvider(...)` +where we instantiate the required `SourceFunction` and its `DeserializationSchema` for runtime. Both +instances are parameterized to return internal data structures (i.e. `RowData`). -{% highlight java %} -RetractStreamTableSink+ @Override + public String factoryIdentifier() { + return "changelog-csv"; + } -implements TableSink getRecordType(); +import org.apache.flink.api.common.serialization.DeserializationSchema; +import org.apache.flink.configuration.ConfigOption; +import org.apache.flink.configuration.ConfigOptions; +import org.apache.flink.configuration.ReadableConfig; +import org.apache.flink.table.connector.format.DecodingFormat; +import org.apache.flink.table.data.RowData; +import org.apache.flink.table.factories.FactoryUtil; +import org.apache.flink.table.factories.DeserializationFormatFactory; +import org.apache.flink.table.factories.DynamicTableFactory; + +public class ChangelogCsvFormatFactory implements DeserializationFormatFactory { + + // define all options statically + public static final ConfigOption COLUMN_DELIMITER = ConfigOptions.key("column-delimiter") + .stringType() + .defaultValue("|"); - public DataStreamSink consumeDataStream(DataStream -{% highlight scala %} -RetractStreamTableSink[T] extends TableSink[Tuple2[Boolean, T]] { + @Override + public Set--- -The table must be have unique key fields (atomic or composite) or be append-only. If the table does not have a unique key and is not append-only, a `TableException` will be thrown. The unique key of the table is configured by the `UpsertStreamTableSink#setKeyFields()` method. -The table will be converted into a stream of upsert and delete messages which are encoded as a Java `Tuple2`. The first field is a boolean flag to indicate the message type. The second field holds the record of the requested type `T`. +**`ChangelogCsvFormat`** -A message with true boolean field is an upsert message for the configured key. A message with false flag is a delete message for the configured key. If the table is append-only, all messages will have a true flag and must be interpreted as insertions. +The `ChangelogCsvFormat` is a decoding format that uses a `DeserializationSchema` during runtime. It +supports emitting `INSERT` and `DELETE` changes. -{% top %} - -Define a TableFactory ---------------------- - -A `TableFactory` allows to create different table-related instances from string-based properties. All available factories are called for matching to the given set of properties and a corresponding factory class. - -Factories leverage Java's [Service Provider Interfaces (SPI)](https://docs.oracle.com/javase/tutorial/sound/SPI-intro.html) for discovering. This means that every dependency and JAR file should contain a file `org.apache.flink.table.factories.TableFactory` in the `META_INF/services` resource directory that lists all available table factories that it provides. - -Every table factory needs to implement the following interface: - -{% highlight java %} -UpsertStreamTableSink+ // create runtime classes that are shipped to the cluster -implements TableSink getRecordType(); + @Override + public ScanRuntimeProvider getScanRuntimeProvider(ScanContext runtimeProviderContext) { - public DataStreamSink consumeDataStream(DataStream -{% highlight scala %} -UpsertStreamTableSink[T] extends TableSink[Tuple2[Boolean, T]] { + final DeserializationSchema-deserializer = decodingFormat.createRuntimeDecoder( + runtimeProviderContext, + producedDataType); - def setKeyFields(keys: Array[String]): Unit + final SourceFunction sourceFunction = new SocketSourceFunction( + hostname, + port, + byteDelimiter, + deserializer); - def setIsAppendOnly(isAppendOnly: Boolean): Unit + return SourceFunctionProvider.of(sourceFunction, false); + } - def getRecordType: TypeInformation[T] + @Override + public DynamicTableSource copy() { + return new SocketDynamicTableSource(hostname, port, byteDelimiter, decodingFormat, producedDataType); + } - def consumeDataStream(dataStream: DataStream[Tuple2[Boolean, T]]): DataStreamSink[_] + @Override + public String asSummaryString() { + return "Socket Table Source"; + } } {% endhighlight %} - -- -* `requiredContext()`: Specifies the context that this factory has been implemented for. The framework guarantees to only match for this factory if the specified set of properties and values are met. Typical properties might be `connector.type`, `format.type`, or `update-mode`. Property keys such as `connector.property-version` and `format.property-version` are reserved for future backwards compatibility cases. -* `supportedProperties()`: List of property keys that this factory can handle. This method will be used for validation. If a property is passed that this factory cannot handle, an exception will be thrown. The list must not contain the keys that are specified by the context. -In order to create a specific instance, a factory class can implement one or more interfaces provided in `org.apache.flink.table.factories`: +### Runtime -* `BatchTableSourceFactory`: Creates a batch table source. -* `BatchTableSinkFactory`: Creates a batch table sink. -* `StreamTableSourceFactory`: Creates a stream table source. -* `StreamTableSinkFactory`: Creates a stream table sink. -* `DeserializationSchemaFactory`: Creates a deserialization schema format. -* `SerializationSchemaFactory`: Creates a serialization schema format. +For completeness, this section illustrates the runtime logic for both `SourceFunction` and `DeserializationSchema`. -The discovery of a factory happens in multiple stages: +**ChangelogCsvDeserializer** -- Discover all available factories. -- Filter by factory class (e.g., `StreamTableSourceFactory`). -- Filter by matching context. -- Filter by supported properties. -- Verify that exactly one factory matches, otherwise throw an `AmbiguousTableFactoryException` or `NoMatchingTableFactoryException`. +The `ChangelogCsvDeserializer` contains a simple parsing logic for converting bytes into `Row` of `Integer` +and `String` with a row kind. The final conversion step converts those into internal data structures. -The following example shows how to provide a custom streaming source with an additional `connector.debug` property flag for parameterization. - -{% highlight java %} -package org.apache.flink.table.factories; - -interface TableFactory { - - Map- -supportedProperties(); -} -{% endhighlight %} - -{% highlight scala %} -package org.apache.flink.table.factories - -trait TableFactory { +import org.apache.flink.api.common.serialization.DeserializationSchema; +import org.apache.flink.api.common.typeinfo.TypeInformation; +import org.apache.flink.table.connector.ChangelogMode; +import org.apache.flink.table.connector.format.DecodingFormat; +import org.apache.flink.table.connector.source.DynamicTableSource; +import org.apache.flink.table.connector.source.DynamicTableSource.DataStructureConverter; +import org.apache.flink.table.data.RowData; +import org.apache.flink.table.types.DataType; +import org.apache.flink.table.types.logical.LogicalType; +import org.apache.flink.types.RowKind; + +public class ChangelogCsvFormat implements DecodingFormat-createRuntimeDecoder( + DynamicTableSource.Context context, + DataType producedDataType) { + // create type information for the DeserializationSchema + final TypeInformation producedTypeInfo = (TypeInformation ) context.createTypeInformation( + producedDataType); + + // most of the code in DeserializationSchema will not work on internal data structures + // create a converter for conversion at the end + final DataStructureConverter converter = context.createDataStructureConverter(producedDataType); + + // use logical types during runtime for parsing + final List parsingTypes = producedDataType.getLogicalType().getChildren(); + + // create runtime class + return new ChangelogCsvDeserializer(parsingTypes, converter, producedTypeInfo, columnDelimiter); + } - def supportedProperties(): util.List[String] + @Override + public ChangelogMode getChangelogMode() { + // define that this format can produce INSERT and DELETE rows + return ChangelogMode.newBuilder() + .addContainedKind(RowKind.INSERT) + .addContainedKind(RowKind.DELETE) + .build(); + } } {% endhighlight %} - -- -{% top %} - -### Use a TableFactory in the SQL Client - -In a SQL Client environment file, the previously presented factory could be declared as: - -{% highlight yaml %} -tables: - - name: MySystemTable - type: source - update-mode: append - connector: - type: my-system - debug: true -{% endhighlight %} - -The YAML file is translated into flattened string properties and a table factory is called with those properties that describe the connection to the external system: - -{% highlight text %} -update-mode=append -connector.type=my-system -connector.debug=true -{% endhighlight %} - -Attention Properties such as `tables.#.name` or `tables.#.type` are SQL Client specifics and are not passed to any factory. The `type` property decides, depending on the execution environment, whether a `BatchTableSourceFactory`/`StreamTableSourceFactory` (for `source`), a `BatchTableSinkFactory`/`StreamTableSinkFactory` (for `sink`), or both (for `both`) need to discovered. - -{% top %} -### Use a TableFactory in the Table & SQL API +**SocketSourceFunction** -For a type-safe, programmatic approach with explanatory Scaladoc/Javadoc, the Table & SQL API offers descriptors in `org.apache.flink.table.descriptors` that translate into string-based properties. See the [built-in descriptors](connect.html) for sources, sinks, and formats as a reference. - -{% highlight java %} -import org.apache.flink.table.sources.StreamTableSource; +import org.apache.flink.api.common.serialization.DeserializationSchema; +import org.apache.flink.api.common.typeinfo.TypeInformation; +import org.apache.flink.table.connector.RuntimeConverter.Context; +import org.apache.flink.table.connector.source.DynamicTableSource.DataStructureConverter; +import org.apache.flink.table.data.RowData; +import org.apache.flink.table.types.logical.LogicalType; +import org.apache.flink.table.types.logical.LogicalTypeRoot; import org.apache.flink.types.Row; -import java.util.ArrayList; -import java.util.HashMap; -import java.util.List; -import java.util.Map; - -class MySystemTableSourceFactory implements StreamTableSourceFactory-{ - - @Override - public Map
{ + + private final List parsingTypes; + private final DataStructureConverter converter; + private final TypeInformation producedTypeInfo; + private final String columnDelimiter; + + public ChangelogCsvDeserializer( + List parsingTypes, + DataStructureConverter converter, + TypeInformation producedTypeInfo, + String columnDelimiter) { + this.parsingTypes = parsingTypes; + this.converter = converter; + this.producedTypeInfo = producedTypeInfo; + this.columnDelimiter = columnDelimiter; } @Override - public List supportedProperties() { - List list = new ArrayList<>(); - list.add("connector.debug"); - return list; + public TypeInformation getProducedType() { + // return the type information required by Flink's core interfaces + return producedTypeInfo; } @Override - public StreamTableSource createStreamTableSource(Map
-{% highlight scala %} -import java.util -import org.apache.flink.table.sources.StreamTableSource -import org.apache.flink.types.Row - -class MySystemTableSourceFactory extends StreamTableSourceFactory[Row] { - - override def requiredContext(): util.Map[String, String] = { - val context = new util.HashMap[String, String]() - context.put("update-mode", "append") - context.put("connector.type", "my-system") - context + @Override + public RowData deserialize(byte[] message) { + // parse the columns including a changelog flag + final String[] columns = new String(message).split(Pattern.quote(columnDelimiter)); + final RowKind kind = RowKind.valueOf(columns[0]); + final Row row = new Row(kind, parsingTypes.size()); + for (int i = 0; i < parsingTypes.size(); i++) { + row.setField(i, parse(parsingTypes.get(i).getTypeRoot(), columns[i + 1])); + } + // convert to internal data structure + return (RowData) converter.toInternal(row); } - override def supportedProperties(): util.List[String] = { - val properties = new util.ArrayList[String]() - properties.add("connector.debug") - properties + private static Object parse(LogicalTypeRoot root, String value) { + switch (root) { + case INTEGER: + return Integer.parseInt(value); + case VARCHAR: + return value; + default: + throw new IllegalArgumentException(); + } } - override def createStreamTableSource(properties: util.Map[String, String]): StreamTableSource[Row] = { - val isDebug = java.lang.Boolean.valueOf(properties.get("connector.debug")) - - # additional validation of the passed properties can also happen here - - new MySystemAppendTableSource(isDebug) + @Override + public boolean isEndOfStream(RowData nextElement) { + return false; } } {% endhighlight %} ---- {% top %} diff --git a/docs/dev/table/sql/alter.md b/docs/dev/table/sql/alter.md index 4cba0828e1d345..61d90981300b74 100644 --- a/docs/dev/table/sql/alter.md +++ b/docs/dev/table/sql/alter.md @@ -35,7 +35,7 @@ Flink SQL supports the following ALTER statements for now: ## Run an ALTER statement -ALTER statements can be executed with the `sqlUpdate()` method of the `TableEnvironment`, or executed in [SQL CLI]({{ site.baseurl }}/dev/table/sqlClient.html). The `sqlUpdate()` method returns nothing for a successful ALTER operation, otherwise will throw an exception. +ALTER statements can be executed with the `executeSql()` method of the `TableEnvironment`, or executed in [SQL CLI]({{ site.baseurl }}/dev/table/sqlClient.html). The `executeSql()` method returns 'OK' for a successful ALTER operation, otherwise will throw an exception. The following examples show how to run an ALTER statement in `TableEnvironment` and in SQL CLI. @@ -46,16 +46,18 @@ EnvironmentSettings settings = EnvironmentSettings.newInstance()... TableEnvironment tableEnv = TableEnvironment.create(settings); // register a table named "Orders" -tableEnv.sqlUpdate("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)"); +tableEnv.executeSql("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)"); // a string array: ["Orders"] -String[] tables = tableEnv.listTable(); +String[] tables = tableEnv.listTables(); +// or tableEnv.executeSql("SHOW TABLES").print(); // rename "Orders" to "NewOrders" -tableEnv.sqlUpdate("ALTER TABLE Orders RENAME TO NewOrders;"); +tableEnv.executeSql("ALTER TABLE Orders RENAME TO NewOrders;"); // a string array: ["NewOrders"] -String[] tables = tableEnv.listTable(); +String[] tables = tableEnv.listTables(); +// or tableEnv.executeSql("SHOW TABLES").print(); {% endhighlight %}- -A custom descriptor can be defined by extending the `ConnectorDescriptor` class. +The `SocketSourceFunction` opens a socket and consumes bytes. It splits records by the given byte +delimiter (`\n` by default) and delegates the decoding to a pluggable `DeserializationSchema`. The +source function can only work with a parallelism of 1. {% highlight java %} -import org.apache.flink.table.descriptors.ConnectorDescriptor; -import java.util.HashMap; -import java.util.Map; - -/** - * Connector to MySystem with debug mode. - */ -public class MySystemConnector extends ConnectorDescriptor { - - public final boolean isDebug; - - public MySystemConnector(boolean isDebug) { - super("my-system", 1, false); - this.isDebug = isDebug; +import org.apache.flink.api.common.serialization.DeserializationSchema; +import org.apache.flink.api.common.typeinfo.TypeInformation; +import org.apache.flink.api.java.typeutils.ResultTypeQueryable; +import org.apache.flink.configuration.Configuration; +import org.apache.flink.streaming.api.functions.source.RichSourceFunction; +import org.apache.flink.table.data.RowData; + +public class SocketSourceFunction extends RichSourceFunction-implements ResultTypeQueryable { + + private final String hostname; + private final int port; + private final byte byteDelimiter; + private final DeserializationSchema deserializer; + + private volatile boolean isRunning = true; + private Socket currentSocket; + + public SocketSourceFunction(String hostname, int port, byte byteDelimiter, DeserializationSchema deserializer) { + this.hostname = hostname; + this.port = port; + this.byteDelimiter = byteDelimiter; + this.deserializer = deserializer; } @Override - protected Map getProducedType() { + return deserializer.getProducedType(); } -} -{% endhighlight %} - -The descriptor can then be used to create a table with the table environment. - -{% highlight java %} -StreamTableEnvironment tableEnv = // ... - -tableEnv - .connect(new MySystemConnector(true)) - .withSchema(...) - .inAppendMode() - .createTemporaryTable("MySystemTable"); -{% endhighlight %} - - - -A custom descriptor can be defined by extending the `ConnectorDescriptor` class. - -{% highlight scala %} -import org.apache.flink.table.descriptors.ConnectorDescriptor -import java.util.HashMap -import java.util.Map + @Override + public void open(Configuration parameters) throws Exception { + deserializer.open(() -> getRuntimeContext().getMetricGroup()); + } -/** - * Connector to MySystem with debug mode. - */ -class MySystemConnector(isDebug: Boolean) extends ConnectorDescriptor("my-system", 1, false) { + @Override + public void run(SourceContext- -ctx) throws Exception { + while (isRunning) { + // open and consume from socket + try (final Socket socket = new Socket()) { + currentSocket = socket; + socket.connect(new InetSocketAddress(hostname, port), 0); + try (InputStream stream = socket.getInputStream()) { + ByteArrayOutputStream buffer = new ByteArrayOutputStream(); + int b; + while ((b = stream.read()) >= 0) { + // buffer until delimiter + if (b != byteDelimiter) { + buffer.write(b); + } + // decode and emit record + else { + ctx.collect(deserializer.deserialize(buffer.toByteArray())); + buffer.reset(); + } + } + } + } catch (Throwable t) { + t.printStackTrace(); // print and continue + } + Thread.sleep(1000); + } + } - override protected def toConnectorProperties(): Map[String, String] = { - val properties = new HashMap[String, String] - properties.put("connector.debug", isDebug.toString) - properties + @Override + public void cancel() { + isRunning = false; + try { + currentSocket.close(); + } catch (Throwable t) { + // ignore + } } } {% endhighlight %} -The descriptor can then be used to create a table with the table environment. - -{% highlight scala %} -val tableEnv: StreamTableEnvironment = // ... - -tableEnv - .connect(new MySystemConnector(isDebug = true)) - .withSchema(...) - .inAppendMode() - .createTemporaryTable("MySystemTable") -{% endhighlight %} - - - -You can use a Java `TableFactory` from Python using the `CustomConnectorDescriptor`. - -{% highlight python %} -s_env = StreamExecutionEnvironment.get_execution_environment() -st_env = StreamTableEnvironment.create(s_env) - -custom_connector = CustomConnectorDescriptor('my-system', 1, False) -st_env\ - .connect(custom_connector.property("connector.debug", "true")) \ - .with_schema(...) \ - .in_append_mode()\ - .create_temporary_table("MySystemTable") -{% endhighlight %} -- -{% highlight python %} -settings = EnvironmentSettings.newInstance()... -table_env = TableEnvironment.create(settings) +settings = EnvironmentSettings.new_instance()... +table_env = StreamTableEnvironment.create(env, settings) # a string array: ["Orders"] -tables = tableEnv.listTable() +tables = table_env.list_tables() +# or table_env.execute_sql("SHOW TABLES").print() # rename "Orders" to "NewOrders" -tableEnv.sqlUpdate("ALTER TABLE Orders RENAME TO NewOrders;") +table_env.execute_sql("ALTER TABLE Orders RENAME TO NewOrders;") # a string array: ["NewOrders"] -tables = tableEnv.listTable() +tables = table_env.list_tables() +# or table_env.execute_sql("SHOW TABLES").print() {% endhighlight %}@@ -142,10 +148,14 @@ Set one or more properties in the specified database. If a particular property i {% highlight sql%} ALTER [TEMPORARY|TEMPORARY SYSTEM] FUNCTION [IF EXISTS] [catalog_name.][db_name.]function_name - AS identifier [LANGUAGE JAVA|SCALA| + AS identifier [LANGUAGE JAVA|SCALA|PYTHON] {% endhighlight %} -Alter a catalog function with the new identifier which is full classpath for JAVA/SCALA and optional language tag. If a function doesn't exist in the catalog, an exception is thrown. +Alter a catalog function with the new identifier and optional language tag. If a function doesn't exist in the catalog, an exception is thrown. + +If the language tag is JAVA/SCALA, the identifier is the full classpath of the UDF. For the implementation of Java/Scala UDF, please refer to [User-defined Functions]({{ site.baseurl }}/dev/table/functions/udfs.html) for more details. + +If the language tag is PYTHON, the identifier is the fully qualified name of the UDF, e.g. `pyflink.table.tests.test_udf.add`. For the implementation of Python UDF, please refer to [Python UDFs]({{ site.baseurl }}/dev/table/python/python_udfs.html) for more details. **TEMPORARY** @@ -159,7 +169,7 @@ Alter temporary system function that has no namespace and overrides built-in fun If the function doesn't exist, nothing happens. -**LANGUAGE JAVA\|SCALA** +**LANGUAGE JAVA\|SCALA\|PYTHON** -Language tag to instruct flink runtime how to execute the function. Currently only JAVA and SCALA are supported, the default language for a function is JAVA. +Language tag to instruct flink runtime how to execute the function. Currently only JAVA, SCALA and PYTHON are supported, the default language for a function is JAVA. diff --git a/docs/dev/table/sql/alter.zh.md b/docs/dev/table/sql/alter.zh.md index def2862c2f03c1..7d3eed1b4306e4 100644 --- a/docs/dev/table/sql/alter.zh.md +++ b/docs/dev/table/sql/alter.zh.md @@ -35,7 +35,7 @@ Flink SQL 目前支持以下 ALTER 语句： ## 执行 ALTER 语句 -可以使用 `TableEnvironment` 中的 `sqlUpdate()` 方法执行 ALTER 语句，也可以在 [SQL CLI]({{ site.baseurl }}/zh/dev/table/sqlClient.html) 中执行 ALTER 语句。 若 ALTER 操作执行成功，`sqlUpdate()` 方法不返回任何内容，否则会抛出异常。 +可以使用 `TableEnvironment` 中的 `executeSql()` 方法执行 ALTER 语句，也可以在 [SQL CLI]({{ site.baseurl }}/zh/dev/table/sqlClient.html) 中执行 ALTER 语句。 若 ALTER 操作执行成功，`executeSql()` 方法返回 'OK'，否则会抛出异常。 以下的例子展示了如何在 `TableEnvironment` 和 SQL CLI 中执行一个 ALTER 语句。 @@ -46,16 +46,18 @@ EnvironmentSettings settings = EnvironmentSettings.newInstance()... TableEnvironment tableEnv = TableEnvironment.create(settings); // 注册名为 “Orders” 的表 -tableEnv.sqlUpdate("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)"); +tableEnv.executeSql("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)"); // 字符串数组： ["Orders"] -String[] tables = tableEnv.listTable(); +String[] tables = tableEnv.listTables(); +// or tableEnv.executeSql("SHOW TABLES").print(); // 把 “Orders” 的表名改为 “NewOrders” -tableEnv.sqlUpdate("ALTER TABLE Orders RENAME TO NewOrders;"); +tableEnv.executeSql("ALTER TABLE Orders RENAME TO NewOrders;"); // 字符串数组：["NewOrders"] -String[] tables = tableEnv.listTable(); +String[] tables = tableEnv.listTables(); +// or tableEnv.executeSql("SHOW TABLES").print(); {% endhighlight %}{% highlight python %} -settings = EnvironmentSettings.newInstance()... -table_env = TableEnvironment.create(settings) +settings = EnvironmentSettings.new_instance()... +table_env = StreamTableEnvironment.create(env, settings) # 字符串数组： ["Orders"] -tables = tableEnv.listTable() +tables = table_env.list_tables() +# or table_env.execute_sql("SHOW TABLES").print() # 把 “Orders” 的表名改为 “NewOrders” -tableEnv.sqlUpdate("ALTER TABLE Orders RENAME TO NewOrders;") +table_env.execute_sql("ALTER TABLE Orders RENAME TO NewOrders;") # 字符串数组：["NewOrders"] -tables = tableEnv.listTable() +tables = table_env.list_tables() +# or table_env.execute_sql("SHOW TABLES").print() {% endhighlight %}@@ -142,10 +148,14 @@ ALTER DATABASE [catalog_name.]db_name SET (key1=val1, key2=val2, ...) {% highlight sql%} ALTER [TEMPORARY|TEMPORARY SYSTEM] FUNCTION [IF EXISTS] [catalog_name.][db_name.]function_name - AS identifier [LANGUAGE JAVA|SCALA| + AS identifier [LANGUAGE JAVA|SCALA|PYTHON] {% endhighlight %} -修改一个有 catalog 和数据库命名空间的 catalog function ，其需要指定 JAVA / SCALA 或其他 language tag 完整的 classpath。若函数不存在，删除会抛出异常。 +修改一个有 catalog 和数据库命名空间的 catalog function ，需要指定一个新的 identifier ，可指定 language tag 。若函数不存在，删除会抛出异常。 + +如果 language tag 是 JAVA 或者 SCALA ，则 identifier 是 UDF 实现类的全限定名。关于 JAVA/SCALA UDF 的实现，请参考 [自定义函数]({{ site.baseurl }}/zh/dev/table/functions/udfs.html)。 + +如果 language tag 是 PYTHON ， 则 identifier 是 UDF 对象的全限定名，例如 `pyflink.table.tests.test_udf.add`。关于 PYTHON UDF 的实现，请参考 [Python UDFs]({{ site.baseurl }}/zh/dev/table/python/python_udfs.html)。 **TEMPORARY** @@ -159,6 +169,6 @@ ALTER [TEMPORARY|TEMPORARY SYSTEM] FUNCTION 若函数不存在，则不进行任何操作。 -**LANGUAGE JAVA\|SCALA** +**LANGUAGE JAVA\|SCALA\|PYTHON** -Language tag 用于指定 Flink runtime 如何执行这个函数。目前，只支持 JAVA 和 SCALA，且函数的默认语言为 JAVA。 \ No newline at end of file +Language tag 用于指定 Flink runtime 如何执行这个函数。目前，只支持 JAVA，SCALA 和 PYTHON，且函数的默认语言为 JAVA。 diff --git a/docs/dev/table/sql/create.md b/docs/dev/table/sql/create.md index e44e2bc74e0bde..4bc398c5d79002 100644 --- a/docs/dev/table/sql/create.md +++ b/docs/dev/table/sql/create.md @@ -31,11 +31,12 @@ Flink SQL supports the following CREATE statements for now: - CREATE TABLE - CREATE DATABASE +- CREATE VIEW - CREATE FUNCTION ## Run a CREATE statement -CREATE statements can be executed with the `sqlUpdate()` method of the `TableEnvironment`, or executed in [SQL CLI]({{ site.baseurl }}/dev/table/sqlClient.html). The `sqlUpdate()` method returns nothing for a successful CREATE operation, otherwise will throw an exception. +CREATE statements can be executed with the `executeSql()` method of the `TableEnvironment`, or executed in [SQL CLI]({{ site.baseurl }}/dev/table/sqlClient.html). The `executeSql()` method returns 'OK' for a successful CREATE operation, otherwise will throw an exception. The following examples show how to run a CREATE statement in `TableEnvironment` and in SQL CLI. @@ -47,16 +48,16 @@ TableEnvironment tableEnv = TableEnvironment.create(settings); // SQL query with a registered table // register a table named "Orders" -tableEnv.sqlUpdate("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)"); +tableEnv.executeSql("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)"); // run a SQL query on the Table and retrieve the result as a new Table Table result = tableEnv.sqlQuery( "SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'"); -// SQL update with a registered table +// Execute insert SQL with a registered table // register a TableSink -tableEnv.sqlUpdate("CREATE TABLE RubberOrders(product STRING, amount INT) WITH (...)"); -// run a SQL update query on the Table and emit the result to the TableSink -tableEnv.sqlUpdate( +tableEnv.executeSql("CREATE TABLE RubberOrders(product STRING, amount INT) WITH (...)"); +// run an insert SQL on the Table and emit the result to the TableSink +tableEnv.executeSql( "INSERT INTO RubberOrders SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'"); {% endhighlight %}{% highlight python %} -settings = EnvironmentSettings.newInstance()... -table_env = TableEnvironment.create(settings) +settings = EnvironmentSettings.new_instance()... +table_env = StreamTableEnvironment.create(env, settings) # SQL query with a registered table # register a table named "Orders" -tableEnv.sqlUpdate("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)"); +table_env.execute_sql("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)"); # run a SQL query on the Table and retrieve the result as a new Table -result = tableEnv.sqlQuery( +result = table_env.sql_query( "SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'"); -# SQL update with a registered table +# Execute an INSERT SQL with a registered table # register a TableSink -table_env.sql_update("CREATE TABLE RubberOrders(product STRING, amount INT) WITH (...)") -# run a SQL update query on the Table and emit the result to the TableSink +table_env.execute_sql("CREATE TABLE RubberOrders(product STRING, amount INT) WITH (...)") +# run an INSERT SQL on the Table and emit the result to the TableSink table_env \ - .sql_update("INSERT INTO RubberOrders SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'") + .execute_sql("INSERT INTO RubberOrders SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'") {% endhighlight %}@@ -126,19 +127,33 @@ CREATE TABLE [catalog_name.][db_name.]table_name ( {| }[ , ...n] [ ] + [ ][ , ...n] ) [COMMENT table_comment] [PARTITIONED BY (partition_column_name1, partition_column_name2, ...)] WITH (key1=val1, key2=val2, ...) - + [ LIKE source_table [( )] ] + : - column_name column_type [COMMENT column_comment] + column_name column_type [ ] [COMMENT column_comment] + + : + [CONSTRAINT constraint_name] PRIMARY KEY NOT ENFORCED + + : + [CONSTRAINT constraint_name] PRIMARY KEY (column_name, ...) NOT ENFORCED : column_name AS computed_column_expression [COMMENT column_comment] : WATERMARK FOR rowtime_column_name AS watermark_strategy_expression + + : +{ + { INCLUDING | EXCLUDING } { ALL | CONSTRAINTS | PARTITIONS } + | { INCLUDING | EXCLUDING | OVERWRITING } { GENERATED | OPTIONS | WATERMARKS } +}[, ...] {% endhighlight %} @@ -194,6 +209,24 @@ CREATE TABLE Orders ( ) WITH ( . . . ); {% endhighlight %} +**PRIMARY KEY** + +Primary key constraint is a hint for Flink to leverage for optimizations. It tells that a column or a set of columns of a table or a view are unique and they **do not** contain null. +Neither of columns in a primary can be nullable. Primary key therefore uniquely identify a row in a table. + +Primary key constraint can be either declared along with a column definition (a column constraint) or as a single line (a table constraint). +For both cases, it should only be declared as a singleton. If you define multiple primary key constraints at the same time, an exception would be thrown. + +##### Validity Check + +SQL standard specifies that a constraint can either be `ENFORCED` or `NOT ENFORCED`. This controls if the constraint checks are performed on the incoming/outgoing data. +Flink does not own the data therefore the only mode we want to support is the `NOT ENFORCED` mode. +It is up to the user to ensure that the query enforces key integrity. + +Flink will assume correctness of the primary key by assuming that the columns nullability is aligned with the columns in primary key. Connectors should ensure those are aligned. + +**Notes:** In a CREATE TABLE statement, creating a primary key constraint will alter the columns nullability, that means, a column with primary key constraint is not nullable. + **PARTITIONED BY** Partition the created table by the specified columns. A directory is created for each partition if this table is used as a filesystem sink. @@ -208,6 +241,101 @@ The key and value of expression `key1=val1` should both be string literal. See d **Notes:** The table registered with `CREATE TABLE` statement can be used as both table source and table sink, we can not decide if it is used as a source or sink until it is referenced in the DMLs. +**LIKE clause** + +The `LIKE` clause is a variant/combination of SQL features (Feature T171, “LIKE clause in table definition” and Feature T173, “Extended LIKE clause in table definition”). The clause can be used to create a table based on a definition of an existing table. Additionally, users +can extend the original table or exclude certain parts of it. In contrast to the SQL standard the clause must be defined at the top-level of a CREATE statement. That is because the clause applies to multiple parts of the definition and not only to the schema part. + +You can use the clause to reuse (and potentially overwrite) certain connector properties or add watermarks to tables defined externally. For example, you can add a watermark to a table defined in Apache Hive. + +Consider the example statement below: +{% highlight sql %} +CREATE TABLE Orders ( + user BIGINT, + product STRING, + order_time TIMESTAMP(3) +) WITH ( + 'connector' = 'kafka', + 'scan.startup.mode' = 'earliest-offset' +); + +CREATE TABLE Orders_with_watermark ( + -- Add watermark definition + WATERMARK FOR order_time AS order_time - INTERVAL '5' SECOND +) WITH ( + -- Overwrite the startup-mode + 'scan.startup.mode' = 'latest-offset' +) +LIKE Orders; +{% endhighlight %} + +The resulting table `Orders_with_watermark` will be equivalent to a table created with a following statement: +{% highlight sql %} +CREATE TABLE Orders_with_watermark ( + user BIGINT, + product STRING, + order_time TIMESTAMP(3), + WATERMARK FOR order_time AS order_time - INTERVAL '5' SECOND +) WITH ( + 'connector' = 'kafka', + 'scan.startup.mode' = 'latest-offset' +); +{% endhighlight %} + +The merging logic of table features can be controlled with `like options`. + +You can control the merging behavior of: + +* CONSTRAINTS - constraints such as primary and unique keys +* GENERATED - computed columns +* OPTIONS - connector options that describe connector and format properties +* PARTITIONS - partition of the tables +* WATERMARKS - watermark declarations + +with three different merging strategies: + +* INCLUDING - Includes the feature of the source table, fails on duplicate entries, e.g. if an option with the same key exists in both tables. +* EXCLUDING - Does not include the given feature of the source table. +* OVERWRITING - Includes the feature of the source table, overwrites duplicate entries of the source table with properties of the new table, e.g. if an option with the same key exists in both tables, the one from the current statement will be used. + +Additionally, you can use the `INCLUDING/EXCLUDING ALL` option to specify what should be the strategy if there was no specific strategy defined, i.e. if you use `EXCLUDING ALL INCLUDING WATERMARKS` only the watermarks will be included from the source table. + +Example: +{% highlight sql %} +-- A source table stored in a filesystem +CREATE TABLE Orders_in_file ( + user BIGINT, + product STRING, + order_time_string STRING, + order_time AS to_timestamp(order_time) + +) +PARTITIONED BY user +WITH ( + 'connector' = 'filesystem' + 'path' = '...' +); + +-- A corresponding table we want to store in kafka +CREATE TABLE Orders_in_kafka ( + -- Add watermark definition + WATERMARK FOR order_time AS order_time - INTERVAL '5' SECOND +) WITH ( + 'connector': 'kafka' + ... +) +LIKE Orders_in_file ( + -- Exclude everything besides the computed columns which we need to generate the watermark for. + -- We do not want to have the partitions or filesystem options as those do not apply to kafka. + EXCLUDING ALL + INCLUDING GENERATED +); +{% endhighlight %} + +If you provide no like options, `INCLUDING ALL OVERWRITING OPTIONS` will be used as a default. + +**NOTE** You cannot control the behavior of merging physical fields. Those will be merged as if you applied the `INCLUDING` strategy. + {% top %} ## CREATE CATALOG @@ -249,14 +377,37 @@ The key and value of expression `key1=val1` should both be string literal. {% top %} +## CREATE VIEW +{% highlight sql %} +CREATE [TEMPORARY] VIEW [IF NOT EXISTS] [catalog_name.][db_name.]view_name + [{columnName [, columnName ]* }] [COMMENT view_comment] + AS query_expression +{% endhighlight %} + +Create a view with the given query expression. If a view with the same name already exists in the catalog, an exception is thrown. + +**TEMPORARY** + +Create temporary view that has catalog and database namespaces and overrides views. + +**IF NOT EXISTS** + +If the view already exists, nothing happens. + +{% top %} + ## CREATE FUNCTION {% highlight sql%} CREATE [TEMPORARY|TEMPORARY SYSTEM] FUNCTION [IF NOT EXISTS] [catalog_name.][db_name.]function_name - AS identifier [LANGUAGE JAVA|SCALA] + AS identifier [LANGUAGE JAVA|SCALA|PYTHON] {% endhighlight %} -Create a catalog function that has catalog and database namespaces with the identifier which is full classpath for JAVA/SCALA and optional language tag. If a function with the same name already exists in the catalog, an exception is thrown. +Create a catalog function that has catalog and database namespaces with the identifier and optional language tag. If a function with the same name already exists in the catalog, an exception is thrown. + +If the language tag is JAVA/SCALA, the identifier is the full classpath of the UDF. For the implementation of Java/Scala UDF, please refer to [User-defined Functions]({{ site.baseurl }}/dev/table/functions/udfs.html) for more details. + +If the language tag is PYTHON, the identifier is the fully qualified name of the UDF, e.g. `pyflink.table.tests.test_udf.add`. For the implementation of Python UDF, please refer to [Python UDFs]({{ site.baseurl }}/dev/table/python/python_udfs.html) for more details. **TEMPORARY** @@ -270,6 +421,6 @@ Create temporary system function that has no namespace and overrides built-in fu If the function already exists, nothing happens. -**LANGUAGE JAVA\|SCALA** +**LANGUAGE JAVA\|SCALA\|PYTHON** -Language tag to instruct Flink runtime how to execute the function. Currently only JAVA and SCALA are supported, the default language for a function is JAVA. +Language tag to instruct Flink runtime how to execute the function. Currently only JAVA, SCALA and PYTHON are supported, the default language for a function is JAVA. diff --git a/docs/dev/table/sql/create.zh.md b/docs/dev/table/sql/create.zh.md index 5c6099ae82ba53..027debfe09ec1c 100644 --- a/docs/dev/table/sql/create.zh.md +++ b/docs/dev/table/sql/create.zh.md @@ -31,11 +31,12 @@ CREATE 语句用于向当前或指定的 [Catalog]({{ site.baseurl }}/zh/dev/tab - CREATE TABLE - CREATE DATABASE +- CREATE VIEW - CREATE FUNCTION ## 执行 CREATE 语句 -可以使用 `TableEnvironment` 中的 `sqlUpdate()` 方法执行 CREATE 语句，也可以在 [SQL CLI]({{ site.baseurl }}/zh/dev/table/sqlClient.html) 中执行 CREATE 语句。 若 CREATE 操作执行成功，`sqlUpdate()` 方法不返回任何内容，否则会抛出异常。 +可以使用 `TableEnvironment` 中的 `executeSql()` 方法执行 CREATE 语句，也可以在 [SQL CLI]({{ site.baseurl }}/zh/dev/table/sqlClient.html) 中执行 CREATE 语句。 若 CREATE 操作执行成功，`executeSql()` 方法返回 'OK'，否则会抛出异常。 以下的例子展示了如何在 `TableEnvironment` 和 SQL CLI 中执行一个 CREATE 语句。 @@ -45,18 +46,18 @@ CREATE 语句用于向当前或指定的 [Catalog]({{ site.baseurl }}/zh/dev/tab EnvironmentSettings settings = EnvironmentSettings.newInstance()... TableEnvironment tableEnv = TableEnvironment.create(settings); -// 对已经已经注册的表进行 SQL 查询 +// 对已注册的表进行 SQL 查询 // 注册名为 “Orders” 的表 -tableEnv.sqlUpdate("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)"); +tableEnv.executeSql("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)"); // 在表上执行 SQL 查询，并把得到的结果作为一个新的表 Table result = tableEnv.sqlQuery( "SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'"); -// SQL 对已注册的表进行 update 操作 +// 对已注册的表进行 INSERT 操作 // 注册 TableSink -tableEnv.sqlUpdate("CREATE TABLE RubberOrders(product STRING, amount INT) WITH (...)"); -// 在表上执行 SQL 更新查询并向 TableSink 发出结果 -tableEnv.sqlUpdate( +tableEnv.executeSql("CREATE TABLE RubberOrders(product STRING, amount INT) WITH (...)"); +// 在表上执行 INSERT 语句并向 TableSink 发出结果 +tableEnv.executeSql( "INSERT INTO RubberOrders SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'"); {% endhighlight %} {% highlight python %} -settings = EnvironmentSettings.newInstance()... -table_env = TableEnvironment.create(settings) +settings = EnvironmentSettings.new_instance()... +table_env = StreamTableEnvironment.create(env, settings) -# 对已经已经注册的表进行 SQL 查询 +# 对已经注册的表进行 SQL 查询 # 注册名为 “Orders” 的表 -tableEnv.sqlUpdate("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)"); +table_env.execute_sql("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)"); # 在表上执行 SQL 查询，并把得到的结果作为一个新的表 -result = tableEnv.sqlQuery( +result = table_env.sql_query( "SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'"); -# SQL 对已注册的表进行 update 操作 +# 对已注册的表进行 INSERT 操作 # 注册 TableSink -table_env.sql_update("CREATE TABLE RubberOrders(product STRING, amount INT) WITH (...)") -# 在表上执行 SQL 更新查询并向 TableSink 发出结果 +table_env.execute_sql("CREATE TABLE RubberOrders(product STRING, amount INT) WITH (...)") +# 在表上执行 INSERT 语句并向 TableSink 发出结果 table_env \ - .sql_update("INSERT INTO RubberOrders SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'") + .execute_sql("INSERT INTO RubberOrders SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'") {% endhighlight %}@@ -126,13 +127,21 @@ CREATE TABLE [catalog_name.][db_name.]table_name ( {| }[ , ...n] [ ] + [ ][ , ...n] ) [COMMENT table_comment] [PARTITIONED BY (partition_column_name1, partition_column_name2, ...)] WITH (key1=val1, key2=val2, ...) + [ LIKE source_table [( )] ] : - column_name column_type [COMMENT column_comment] + column_name column_type [ ] [COMMENT column_comment] + + : + [CONSTRAINT constraint_name] PRIMARY KEY NOT ENFORCED + + : + [CONSTRAINT constraint_name] PRIMARY KEY (column_name, ...) NOT ENFORCED : column_name AS computed_column_expression [COMMENT column_comment] @@ -140,6 +149,12 @@ CREATE TABLE [catalog_name.][db_name.]table_name : WATERMARK FOR rowtime_column_name AS watermark_strategy_expression + : +{ + { INCLUDING | EXCLUDING } { ALL | CONSTRAINTS | PARTITIONS } + | { INCLUDING | EXCLUDING | OVERWRITING } { GENERATED | OPTIONS | WATERMARKS } +}[, ...] + {% endhighlight %} 根据指定的表名创建一个表，如果同名表已经在 catalog 中存在了，则无法注册。 @@ -194,6 +209,22 @@ CREATE TABLE Orders ( ) WITH ( . . . ); {% endhighlight %} + +**PRIMARY KEY** + +主键用作 Flink 优化的一种提示信息。主键限制表明一张表或视图的某个（些）列是唯一的并且不包含 Null 值。 +主键声明的列都是非 nullable 的。因此主键可以被用作表行级别的唯一标识。 + +主键可以和列的定义一起声明，也可以独立声明为表的限制属性，不管是哪种方式，主键都不可以重复定义，否则 Flink 会报错。 + +##### 有效性检查 + +SQL 标准主键限制可以有两种模式：`ENFORCED` 或者 `NOT ENFORCED`。 它申明了是否输入/出数据会做合法性检查（是否唯一）。Flink 不存储数据因此只支持 `NOT ENFORCED` 模式，即不做检查，用户需要自己保证唯一性。 + +Flink 假设声明了主键的列都是不包含 Null 值的，Connector 在处理数据时需要自己保证语义正确。 + +**Notes:** 在 CREATE TABLE 语句中，创建主键会修改列的 nullable 属性，主键声明的列默认都是非 Nullable 的。 + **PARTITIONED BY** 根据指定的列对已经创建的表进行分区。若表使用 filesystem sink ，则将会为每个分区创建一个目录。 @@ -208,6 +239,103 @@ CREATE TABLE Orders ( **注意：** 使用 `CREATE TABLE` 语句注册的表均可用作 table source 和 table sink。 在被 DML 语句引用前，我们无法决定其实际用于 source 抑或是 sink。 +**LIKE** + +`LIKE` 子句来源于两种 SQL 特性的变体/组合（Feature T171，“表定义中的 LIKE 语法” 和 Feature T173，“表定义中的 LIKE 语法扩展”）。LIKE 子句可以基于现有表的定义去创建新表，并且可以扩展或排除原始表中的某些部分。与 SQL 标准相反，LIKE 子句必须在 CREATE 语句中定义，并且是基于 CREATE 语句的更上层定义，这是因为 LIKE 子句可以用于定义表的多个部分，而不仅仅是 schema 部分。 + +你可以使用该子句，重用（或改写）指定的连接器配置属性或者可以向外部表添加 watermark 定义，例如可以向 Apache Hive 中定义的表添加 watermark 定义。 + +示例如下： + +{% highlight sql %} +CREATE TABLE Orders ( + user BIGINT, + product STRING, + order_time TIMESTAMP(3) +) WITH ( + 'connector' = 'kafka', + 'scan.startup.mode' = 'earliest-offset' +); + +CREATE TABLE Orders_with_watermark ( + -- 添加 watermark 定义 + WATERMARK FOR order_time AS order_time - INTERVAL '5' SECOND +) WITH ( + -- 改写 startup-mode 属性 + 'scan.startup.mode' = 'latest-offset' +) +LIKE Orders; +{% endhighlight %} + +结果表 `Orders_with_watermark` 等效于使用以下语句创建的表： + +{% highlight sql %} +CREATE TABLE Orders_with_watermark ( + user BIGINT, + product STRING, + order_time TIMESTAMP(3), + WATERMARK FOR order_time AS order_time - INTERVAL '5' SECOND +) WITH ( + 'connector' = 'kafka', + 'scan.startup.mode' = 'latest-offset' +); +{% endhighlight %} + +表属性的合并逻辑可以用 `like options` 来控制。 + +可以控制合并的表属性如下： + +* CONSTRAINTS - 主键和唯一键约束 +* GENERATED - 计算列 +* OPTIONS - 连接器信息、格式化方式等配置项 +* PARTITIONS - 表分区信息 +* WATERMARKS - watermark 定义 + +并且有三种不同的表属性合并策略： + +* INCLUDING - 新表包含源表（source table）所有的表属性，如果和源表的表属性重复则会直接失败，例如新表和源表存在相同 key 的属性。 +* EXCLUDING - 新表不包含源表指定的任何表属性。 +* OVERWRITING - 新表包含源表的表属性，但如果出现重复项，则会用新表的表属性覆盖源表中的重复表属性，例如，两个表中都存在相同 key 的属性，则会使用当前语句中定义的 key 的属性值。 + +并且你可以使用 `INCLUDING/EXCLUDING ALL` 这种声明方式来指定使用怎样的合并策略，例如使用 `EXCLUDING ALL INCLUDING WATERMARKS`，那么代表只有源表的 WATERMARKS 属性才会被包含进新表。 + +示例如下： +{% highlight sql %} + +-- 存储在文件系统的源表 +CREATE TABLE Orders_in_file ( + user BIGINT, + product STRING, + order_time_string STRING, + order_time AS to_timestamp(order_time) + +) +PARTITIONED BY user +WITH ( + 'connector' = 'filesystem' + 'path' = '...' +); + +-- 对应存储在 kafka 的源表 +CREATE TABLE Orders_in_kafka ( + -- 添加 watermark 定义 + WATERMARK FOR order_time AS order_time - INTERVAL '5' SECOND +) WITH ( + 'connector': 'kafka' + ... +) +LIKE Orders_in_file ( + -- 排除需要生成 watermark 的计算列之外的所有内容。 + -- 去除不适用于 kafka 的所有分区和文件系统的相关属性。 + EXCLUDING ALL + INCLUDING GENERATED +); +{% endhighlight %} + +如果未提供 like 配置项（like options），默认将使用 `INCLUDING ALL OVERWRITING OPTIONS` 的合并策略。 + +**注意：** 您无法选择物理列的合并策略，当物理列进行合并时就如使用了 `INCLUDING` 策略。 + {% top %} ## CREATE CATALOG @@ -249,14 +377,37 @@ CREATE DATABASE [IF NOT EXISTS] [catalog_name.]db_name {% top %} +## CREATE VIEW +{% highlight sql %} +CREATE [TEMPORARY] VIEW [IF NOT EXISTS] [catalog_name.][db_name.]view_name + [{columnName [, columnName ]* }] [COMMENT view_comment] + AS query_expression +{% endhighlight %} + +根据给定的 query 语句创建一个视图。若数据库中已经存在同名视图会抛出异常. + +**TEMPORARY** + +创建一个有 catalog 和数据库命名空间的临时视图，并覆盖原有的视图。 + +**IF NOT EXISTS** + +若该视图已经存在，则不会进行任何操作。 + +{% top %} + ## CREATE FUNCTION {% highlight sql%} CREATE [TEMPORARY|TEMPORARY SYSTEM] FUNCTION [IF NOT EXISTS] [[catalog_name.]db_name.]function_name - AS identifier [LANGUAGE JAVA|SCALA] + AS identifier [LANGUAGE JAVA|SCALA|PYTHON] {% endhighlight %} -创建一个有 catalog 和数据库命名空间的 catalog function ，其需要指定 JAVA / SCALA 或其他 language tag 完整的 classpath。 若 catalog 中，已经有同名的函数注册了，则无法注册。 +创建一个有 catalog 和数据库命名空间的 catalog function ，需要指定一个 identifier ，可指定 language tag 。 若 catalog 中，已经有同名的函数注册了，则无法注册。 + +如果 language tag 是 JAVA 或者 SCALA ，则 identifier 是 UDF 实现类的全限定名。关于 JAVA/SCALA UDF 的实现，请参考 [自定义函数]({{ site.baseurl }}/zh/dev/table/functions/udfs.html)。 + +如果 language tag 是 PYTHON ，则 identifier 是 UDF 对象的全限定名，例如 `pyflink.table.tests.test_udf.add`。关于 PYTHON UDF 的实现，请参考 [Python UDFs]({{ site.baseurl }}/zh/dev/table/python/python_udfs.html)。 **TEMPORARY** @@ -270,7 +421,7 @@ CREATE [TEMPORARY|TEMPORARY SYSTEM] FUNCTION 若该函数已经存在，则不会进行任何操作。 -**LANGUAGE JAVA\|SCALA** +**LANGUAGE JAVA\|SCALA\|PYTHON** -Language tag 用于指定 Flink runtime 如何执行这个函数。目前，只支持 JAVA 和 SCALA，且函数的默认语言为 JAVA。 +Language tag 用于指定 Flink runtime 如何执行这个函数。目前，只支持 JAVA, SCALA 和 PYTHON，且函数的默认语言为 JAVA。 diff --git a/docs/dev/table/sql/describe.md b/docs/dev/table/sql/describe.md new file mode 100644 index 00000000000000..de188692ec9d49 --- /dev/null +++ b/docs/dev/table/sql/describe.md @@ -0,0 +1,202 @@ +--- +title: "DESCRIBE Statements" +nav-parent_id: sql +nav-pos: 7 +--- + + +* This will be replaced by the TOC +{:toc} + +DESCRIBE statements are used to describe the schema of a table or a view. + + +## Run a DESCRIBE statement + +DESCRIBE statements can be executed with the `executeSql()` method of the `TableEnvironment`, or executed in [SQL CLI]({{ site.baseurl }}/dev/table/sqlClient.html). The `executeSql()` method returns the schema of given table for a successful DESCRIBE operation, otherwise will throw an exception. + +The following examples show how to run a DESCRIBE statement in `TableEnvironment` and in SQL CLI. + + ++ +The result of the above example is: ++{% highlight java %} +EnvironmentSettings settings = EnvironmentSettings.newInstance()... +TableEnvironment tableEnv = TableEnvironment.create(settings); + +// register a table named "Orders" +tableEnv.executeSql( + "CREATE TABLE Orders (" + + " `user` BIGINT NOT NULl," + + " product VARCHAR(32)," + + " amount INT," + + " ts TIMESTAMP(3)," + + " ptime AS PROCTIME()," + + " PRIMARY KEY(`user`) NOT ENFORCED," + + " WATERMARK FOR ts AS ts - INTERVAL '1' SECONDS" + + ") with (...)"); + +// print the schema +tableEnv.executeSql("DESCRIBE Orders").print(); + +{% endhighlight %} ++ ++{% highlight scala %} +val settings = EnvironmentSettings.newInstance()... +val tableEnv = TableEnvironment.create(settings) + +// register a table named "Orders" + tableEnv.executeSql( + "CREATE TABLE Orders (" + + " `user` BIGINT NOT NULl," + + " product VARCHAR(32)," + + " amount INT," + + " ts TIMESTAMP(3)," + + " ptime AS PROCTIME()," + + " PRIMARY KEY(`user`) NOT ENFORCED," + + " WATERMARK FOR ts AS ts - INTERVAL '1' SECONDS" + + ") with (...)") + +// print the schema +tableEnv.executeSql("DESCRIBE Orders").print() + +{% endhighlight %} ++ ++{% highlight python %} +settings = EnvironmentSettings.new_instance()... +table_env = StreamTableEnvironment.create(env, settings) + +# register a table named "Orders" +table_env.execute_sql( \ + "CREATE TABLE Orders (" + " `user` BIGINT NOT NULl," + " product VARCHAR(32)," + " amount INT," + " ts TIMESTAMP(3)," + " ptime AS PROCTIME()," + " PRIMARY KEY(`user`) NOT ENFORCED," + " WATERMARK FOR ts AS ts - INTERVAL '1' SECONDS" + ") with (...)"); + +# print the schema +table_env.execute_sql("DESCRIBE Orders").print() + +{% endhighlight %} ++ ++{% highlight sql %} +Flink SQL> CREATE TABLE Orders ( +> `user` BIGINT NOT NULl, +> product VARCHAR(32), +> amount INT, +> ts TIMESTAMP(3), +> ptime AS PROCTIME(), +> PRIMARY KEY(`user`) NOT ENFORCED, +> WATERMARK FOR ts AS ts - INTERVAL '1' SECONDS +> ) with ( +> ... +> ); +[INFO] Table has been created. + +Flink SQL> DESCRIBE Orders; + +{% endhighlight %} ++++ + +{% top %} + +## Syntax + +{% highlight sql %} +DESCRIBE [catalog_name.][db_name.]table_name +{% endhighlight %} diff --git a/docs/dev/table/sql/describe.zh.md b/docs/dev/table/sql/describe.zh.md new file mode 100644 index 00000000000000..cb532ca07b1328 --- /dev/null +++ b/docs/dev/table/sql/describe.zh.md @@ -0,0 +1,201 @@ +--- +title: "DESCRIBE 语句" +nav-parent_id: sql +nav-pos: 7 +--- + + +* This will be replaced by the TOC +{:toc} + +DESCRIBE 语句用来描述一张表或者视图的 Schema。 + + +## 执行 DESCRIBE 语句 + +DESCRIBE 语句可以通过 `TableEnvironment` 的 `executeSql()` 执行，也可以在 [SQL CLI]({{ site.baseurl }}/dev/table/sqlClient.html) 中执行 DROP 语句。 若 DESCRIBE 操作执行成功，executeSql() 方法返回该表的 Schema，否则会抛出异常。 + +以下的例子展示了如何在 TableEnvironment 和 SQL CLI 中执行一个 DESCRIBE 语句。 + ++{% highlight text %} + ++---------+----------------------------------+-------+-----------+-----------------+----------------------------+ +| name | type | null | key | computed column | watermark | ++---------+----------------------------------+-------+-----------+-----------------+----------------------------+ +| user | BIGINT | false | PRI(user) | | | +| product | VARCHAR(32) | true | | | | +| amount | INT | true | | | | +| ts | TIMESTAMP(3) *ROWTIME* | true | | | `ts` - INTERVAL '1' SECOND | +| ptime | TIMESTAMP(3) NOT NULL *PROCTIME* | false | | PROCTIME() | | ++---------+----------------------------------+-------+-----------+-----------------+----------------------------+ +5 rows in set + +{% endhighlight %} +++{% highlight text %} + ++---------+----------------------------------+-------+-----------+-----------------+----------------------------+ +| name | type | null | key | computed column | watermark | ++---------+----------------------------------+-------+-----------+-----------------+----------------------------+ +| user | BIGINT | false | PRI(user) | | | +| product | VARCHAR(32) | true | | | | +| amount | INT | true | | | | +| ts | TIMESTAMP(3) *ROWTIME* | true | | | `ts` - INTERVAL '1' SECOND | +| ptime | TIMESTAMP(3) NOT NULL *PROCTIME* | false | | PROCTIME() | | ++---------+----------------------------------+-------+-----------+-----------------+----------------------------+ +5 rows in set + +{% endhighlight %} +++{% highlight text %} + ++---------+----------------------------------+-------+-----------+-----------------+----------------------------+ +| name | type | null | key | computed column | watermark | ++---------+----------------------------------+-------+-----------+-----------------+----------------------------+ +| user | BIGINT | false | PRI(user) | | | +| product | VARCHAR(32) | true | | | | +| amount | INT | true | | | | +| ts | TIMESTAMP(3) *ROWTIME* | true | | | `ts` - INTERVAL '1' SECOND | +| ptime | TIMESTAMP(3) NOT NULL *PROCTIME* | false | | PROCTIME() | | ++---------+----------------------------------+-------+-----------+-----------------+----------------------------+ +5 rows in set + +{% endhighlight %} +++{% highlight text %} + +root + |-- user: BIGINT NOT NULL + |-- product: VARCHAR(32) + |-- amount: INT + |-- ts: TIMESTAMP(3) *ROWTIME* + |-- ptime: TIMESTAMP(3) NOT NULL *PROCTIME* AS PROCTIME() + |-- WATERMARK FOR ts AS `ts` - INTERVAL '1' SECOND + |-- CONSTRAINT PK_3599338 PRIMARY KEY (user) + +{% endhighlight %} ++ +++ +上述例子执行的结果为: ++{% highlight java %} +EnvironmentSettings settings = EnvironmentSettings.newInstance()... +TableEnvironment tableEnv = TableEnvironment.create(settings); + +// register a table named "Orders" +tableEnv.executeSql( + "CREATE TABLE Orders (" + + " `user` BIGINT NOT NULl," + + " product VARCHAR(32)," + + " amount INT," + + " ts TIMESTAMP(3)," + + " ptime AS PROCTIME()," + + " PRIMARY KEY(`user`) NOT ENFORCED," + + " WATERMARK FOR ts AS ts - INTERVAL '1' SECONDS" + + ") with (...)"); + +// print the schema +tableEnv.executeSql("DESCRIBE Orders").print(); + +{% endhighlight %} ++ ++{% highlight scala %} +val settings = EnvironmentSettings.newInstance()... +val tableEnv = TableEnvironment.create(settings) + +// register a table named "Orders" + tableEnv.executeSql( + "CREATE TABLE Orders (" + + " `user` BIGINT NOT NULl," + + " product VARCHAR(32)," + + " amount INT," + + " ts TIMESTAMP(3)," + + " ptime AS PROCTIME()," + + " PRIMARY KEY(`user`) NOT ENFORCED," + + " WATERMARK FOR ts AS ts - INTERVAL '1' SECONDS" + + ") with (...)") + +// print the schema +tableEnv.executeSql("DESCRIBE Orders").print() + +{% endhighlight %} ++ ++{% highlight python %} +settings = EnvironmentSettings.new_instance()... +table_env = StreamTableEnvironment.create(env, settings) + +# register a table named "Orders" +table_env.execute_sql( \ + "CREATE TABLE Orders (" + " `user` BIGINT NOT NULl," + " product VARCHAR(32)," + " amount INT," + " ts TIMESTAMP(3)," + " ptime AS PROCTIME()," + " PRIMARY KEY(`user`) NOT ENFORCED," + " WATERMARK FOR ts AS ts - INTERVAL '1' SECONDS" + ") with (...)"); + +# print the schema +table_env.execute_sql("DESCRIBE Orders").print() + +{% endhighlight %} ++ ++{% highlight sql %} +Flink SQL> CREATE TABLE Orders ( +> `user` BIGINT NOT NULl, +> product VARCHAR(32), +> amount INT, +> ts TIMESTAMP(3), +> ptime AS PROCTIME(), +> PRIMARY KEY(`user`) NOT ENFORCED, +> WATERMARK FOR ts AS ts - INTERVAL '1' SECONDS +> ) with ( +> ... +> ); +[INFO] Table has been created. + +Flink SQL> DESCRIBE Orders; + +{% endhighlight %} ++++ +{% top %} + +## 语法 + +{% highlight sql %} +DESCRIBE [catalog_name.][db_name.]table_name +{% endhighlight %} diff --git a/docs/dev/table/sql/drop.md b/docs/dev/table/sql/drop.md index 6f340e8bd446af..c12105ff338bac 100644 --- a/docs/dev/table/sql/drop.md +++ b/docs/dev/table/sql/drop.md @@ -31,11 +31,12 @@ Flink SQL supports the following DROP statements for now: - DROP TABLE - DROP DATABASE +- DROP VIEW - DROP FUNCTION ## Run a DROP statement -DROP statements can be executed with the `sqlUpdate()` method of the `TableEnvironment`, or executed in [SQL CLI]({{ site.baseurl }}/dev/table/sqlClient.html). The `sqlUpdate()` method returns nothing for a successful DROP operation, otherwise will throw an exception. +DROP statements can be executed with the `executeSql()` method of the `TableEnvironment`, or executed in [SQL CLI]({{ site.baseurl }}/dev/table/sqlClient.html). The `executeSql()` method returns 'OK' for a successful DROP operation, otherwise will throw an exception. The following examples show how to run a DROP statement in `TableEnvironment` and in SQL CLI. @@ -46,16 +47,18 @@ EnvironmentSettings settings = EnvironmentSettings.newInstance()... TableEnvironment tableEnv = TableEnvironment.create(settings); // register a table named "Orders" -tableEnv.sqlUpdate("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)"); +tableEnv.executeSql("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)"); // a string array: ["Orders"] -String[] tables = tableEnv.listTable(); +String[] tables = tableEnv.listTables(); +// or tableEnv.executeSql("SHOW TABLES").print(); // drop "Orders" table from catalog -tableEnv.sqlUpdate("DROP TABLE Orders"); +tableEnv.executeSql("DROP TABLE Orders"); // an empty string array -String[] tables = tableEnv.listTable(); +String[] tables = tableEnv.listTables(); +// or tableEnv.executeSql("SHOW TABLES").print(); {% endhighlight %}+{% highlight text %} + ++---------+----------------------------------+-------+-----------+-----------------+----------------------------+ +| name | type | null | key | computed column | watermark | ++---------+----------------------------------+-------+-----------+-----------------+----------------------------+ +| user | BIGINT | false | PRI(user) | | | +| product | VARCHAR(32) | true | | | | +| amount | INT | true | | | | +| ts | TIMESTAMP(3) *ROWTIME* | true | | | `ts` - INTERVAL '1' SECOND | +| ptime | TIMESTAMP(3) NOT NULL *PROCTIME* | false | | PROCTIME() | | ++---------+----------------------------------+-------+-----------+-----------------+----------------------------+ +5 rows in set + +{% endhighlight %} +++{% highlight text %} + ++---------+----------------------------------+-------+-----------+-----------------+----------------------------+ +| name | type | null | key | computed column | watermark | ++---------+----------------------------------+-------+-----------+-----------------+----------------------------+ +| user | BIGINT | false | PRI(user) | | | +| product | VARCHAR(32) | true | | | | +| amount | INT | true | | | | +| ts | TIMESTAMP(3) *ROWTIME* | true | | | `ts` - INTERVAL '1' SECOND | +| ptime | TIMESTAMP(3) NOT NULL *PROCTIME* | false | | PROCTIME() | | ++---------+----------------------------------+-------+-----------+-----------------+----------------------------+ +5 rows in set + +{% endhighlight %} +++{% highlight text %} + ++---------+----------------------------------+-------+-----------+-----------------+----------------------------+ +| name | type | null | key | computed column | watermark | ++---------+----------------------------------+-------+-----------+-----------------+----------------------------+ +| user | BIGINT | false | PRI(user) | | | +| product | VARCHAR(32) | true | | | | +| amount | INT | true | | | | +| ts | TIMESTAMP(3) *ROWTIME* | true | | | `ts` - INTERVAL '1' SECOND | +| ptime | TIMESTAMP(3) NOT NULL *PROCTIME* | false | | PROCTIME() | | ++---------+----------------------------------+-------+-----------+-----------------+----------------------------+ +5 rows in set + +{% endhighlight %} +++{% highlight text %} + +root + |-- user: BIGINT NOT NULL + |-- product: VARCHAR(32) + |-- amount: INT + |-- ts: TIMESTAMP(3) *ROWTIME* + |-- ptime: TIMESTAMP(3) NOT NULL *PROCTIME* AS PROCTIME() + |-- WATERMARK FOR ts AS `ts` - INTERVAL '1' SECOND + |-- CONSTRAINT PK_3599338 PRIMARY KEY (user) + +{% endhighlight %} ++ +{% highlight python %} -settings = EnvironmentSettings.newInstance()... -table_env = TableEnvironment.create(settings) +settings = EnvironmentSettings.new_instance()... +table_env = StreamTableEnvironment.create(env, settings) # a string array: ["Orders"] -tables = tableEnv.listTable() +tables = table_env.list_tables() +# or table_env.execute_sql("SHOW TABLES").print() # drop "Orders" table from catalog -tableEnv.sqlUpdate("DROP TABLE Orders") +table_env.execute_sql("DROP TABLE Orders") # an empty string array -tables = tableEnv.listTable() +tables = table_env.list_tables() +# or table_env.execute_sql("SHOW TABLES").print() {% endhighlight %}@@ -143,6 +150,25 @@ Dropping a non-empty database triggers an exception. Enabled by default. Dropping a non-empty database also drops all associated tables and functions. +## DROP VIEW + +{% highlight sql %} +DROP [TEMPORARY] VIEW [IF EXISTS] [catalog_name.][db_name.]view_name +{% endhighlight %} + +Drop a view that has catalog and database namespaces. If the view to drop does not exist, an exception is thrown. + +**TEMPORARY** + +Drop temporary view that has catalog and database namespaces. + +**IF EXISTS** + +If the view does not exist, nothing happens. + +**MAINTAIN DEPENDENCIES** +Flink does not maintain dependencies of view by CASCADE/RESTRICT keywords, the current way is producing postpone error message when user tries to use the view under the scenarios like the underlying table of view has been dropped. + ## DROP FUNCTION {% highlight sql%} diff --git a/docs/dev/table/sql/drop.zh.md b/docs/dev/table/sql/drop.zh.md index 51ac456c7ad681..911ae97c20d7a9 100644 --- a/docs/dev/table/sql/drop.zh.md +++ b/docs/dev/table/sql/drop.zh.md @@ -31,11 +31,12 @@ Flink SQL 目前支持以下 DROP 语句： - DROP TABLE - DROP DATABASE +- DROP VIEW - DROP FUNCTION ## 执行 DROP 语句 -可以使用 `TableEnvironment` 中的 `sqlUpdate()` 方法执行 DROP 语句，也可以在 [SQL CLI]({{ site.baseurl }}/zh/dev/table/sqlClient.html) 中执行 DROP 语句。 若 DROP 操作执行成功，`sqlUpdate()` 方法不返回任何内容，否则会抛出异常。 +可以使用 `TableEnvironment` 中的 `executeSql()` 方法执行 DROP 语句，也可以在 [SQL CLI]({{ site.baseurl }}/zh/dev/table/sqlClient.html) 中执行 DROP 语句。 若 DROP 操作执行成功，`executeSql()` 方法返回 'OK'，否则会抛出异常。 以下的例子展示了如何在 `TableEnvironment` 和 SQL CLI 中执行一个 DROP 语句。 @@ -46,16 +47,18 @@ EnvironmentSettings settings = EnvironmentSettings.newInstance()... TableEnvironment tableEnv = TableEnvironment.create(settings); // 注册名为 “Orders” 的表 -tableEnv.sqlUpdate("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)"); +tableEnv.executeSql("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)"); // 字符串数组： ["Orders"] -String[] tables = tableEnv.listTable(); +String[] tables = tableEnv.listTables(); +// or tableEnv.executeSql("SHOW TABLES").print(); // 从 catalog 删除 “Orders” 表 -tableEnv.sqlUpdate("DROP TABLE Orders"); +tableEnv.executeSql("DROP TABLE Orders"); // 空字符串数组 -String[] tables = tableEnv.listTable(); +String[] tables = tableEnv.listTables(); +// or tableEnv.executeSql("SHOW TABLES").print(); {% endhighlight %}{% highlight python %} -settings = EnvironmentSettings.newInstance()... -table_env = TableEnvironment.create(settings) +settings = EnvironmentSettings.new_instance()... +table_env = StreamTableEnvironment.create(env, settings) # 字符串数组： ["Orders"] -tables = tableEnv.listTable() +tables = table_env.listTables() +# or table_env.executeSql("SHOW TABLES").print() # 从 catalog 删除 “Orders” 表 -tableEnv.sqlUpdate("DROP TABLE Orders") +table_env.execute_sql("DROP TABLE Orders") # 空字符串数组 -tables = tableEnv.listTable() +tables = table_env.list_tables() +# or table_env.execute_sql("SHOW TABLES").print() {% endhighlight %}@@ -143,6 +150,25 @@ DROP DATABASE [IF EXISTS] [catalog_name.]db_name [ (RESTRICT | CASCADE) ] 删除一个非空数据库时，把相关联的表与函数一并删除。 +## DROP VIEW + +{% highlight sql %} +DROP [TEMPORARY] VIEW [IF EXISTS] [catalog_name.][db_name.]view_name +{% endhighlight %} + +删除一个有 catalog 和数据库命名空间的视图。若需要删除的视图不存在，则会产生异常。 + +**TEMPORARY** + +删除一个有 catalog 和数据库命名空间的临时视图。 + +**IF EXISTS** + +若视图不存在，则不会进行任何操作。 + +**依赖管理** +Flink 没有使用 CASCADE / RESTRICT 关键字来维护视图的依赖关系，当前的方案是在用户使用视图时再提示错误信息，比如在视图的底层表已经被删除等场景。 + ## DROP FUNCTION {% highlight sql%} diff --git a/docs/dev/table/sql/explain.md b/docs/dev/table/sql/explain.md new file mode 100644 index 00000000000000..ae26b0652aa300 --- /dev/null +++ b/docs/dev/table/sql/explain.md @@ -0,0 +1,180 @@ +--- +title: "EXPLAIN Statements" +nav-parent_id: sql +nav-pos: 8 +--- + + +* This will be replaced by the TOC +{:toc} + +EXPLAIN statements are used to explain the logical and optimized query plans of a query or an INSERT statement. + + +## Run an EXPLAIN statement + +EXPLAIN statements can be executed with the `executeSql()` method of the `TableEnvironment`, or executed in [SQL CLI]({{ site.baseurl }}/dev/table/sqlClient.html). The `executeSql()` method returns explain result for a successful EXPLAIN operation, otherwise will throw an exception. + +The following examples show how to run an EXPLAIN statement in `TableEnvironment` and in SQL CLI. + +++ +The `EXPLAIN` result is: ++{% highlight java %} +StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); +StreamTableEnvironment tEnv = StreamTableEnvironment.create(env); + +// register a table named "Orders" +tEnv.executeSql("CREATE TABLE MyTable1 (count bigint, work VARCHAR(256) WITH (...)"); +tEnv.executeSql("CREATE TABLE MyTable2 (count bigint, work VARCHAR(256) WITH (...)"); + +// explain SELECT statement through TableEnvironment.explainSql() +String explanation = tEnv.explainSql( + "SELECT count, word FROM MyTable1 WHERE word LIKE 'F%' " + + "UNION ALL " + + "SELECT count, word FROM MyTable2"); +System.out.println(explanation); + +// explain SELECT statement through TableEnvironment.executeSql() +TableResult tableResult = tEnv.executeSql( + "EXPLAIN PLAN FOR " + + "SELECT count, word FROM MyTable1 WHERE word LIKE 'F%' " + + "UNION ALL " + + "SELECT count, word FROM MyTable2"); +tableResult.print(); + +{% endhighlight %} ++ ++{% highlight scala %} +val env = StreamExecutionEnvironment.getExecutionEnvironment() +val tEnv = StreamTableEnvironment.create(env) + +// register a table named "Orders" +tEnv.executeSql("CREATE TABLE MyTable1 (count bigint, work VARCHAR(256) WITH (...)") +tEnv.executeSql("CREATE TABLE MyTable2 (count bigint, work VARCHAR(256) WITH (...)") + +// explain SELECT statement through TableEnvironment.explainSql() +val explanation = tEnv.explainSql( + "SELECT count, word FROM MyTable1 WHERE word LIKE 'F%' " + + "UNION ALL " + + "SELECT count, word FROM MyTable2") +println(explanation) + +// explain SELECT statement through TableEnvironment.executeSql() +val tableResult = tEnv.executeSql( + "EXPLAIN PLAN FOR " + + "SELECT count, word FROM MyTable1 WHERE word LIKE 'F%' " + + "UNION ALL " + + "SELECT count, word FROM MyTable2") +tableResult.print() + +{% endhighlight %} ++ ++{% highlight python %} +settings = EnvironmentSettings.new_instance()... +table_env = StreamTableEnvironment.create(env, settings) + +t_env.execute_sql("CREATE TABLE MyTable1 (count bigint, work VARCHAR(256) WITH (...)") +t_env.execute_sql("CREATE TABLE MyTable2 (count bigint, work VARCHAR(256) WITH (...)") + +# explain SELECT statement through TableEnvironment.explain_sql() +explanation1 = t_env.explain_sql( + "SELECT count, word FROM MyTable1 WHERE word LIKE 'F%' " + "UNION ALL " + "SELECT count, word FROM MyTable2") +print(explanation1) + +# explain SELECT statement through TableEnvironment.execute_sql() +table_result = t_env.execute_sql( + "EXPLAIN PLAN FOR " + "SELECT count, word FROM MyTable1 WHERE word LIKE 'F%' " + "UNION ALL " + "SELECT count, word FROM MyTable2") +table_result.print() + +{% endhighlight %} ++ ++{% highlight sql %} +Flink SQL> CREATE TABLE MyTable1 (count bigint, work VARCHAR(256); +[INFO] Table has been created. + +Flink SQL> CREATE TABLE MyTable2 (count bigint, work VARCHAR(256); +[INFO] Table has been created. + +Flink SQL> EXPLAIN PLAN FOR SELECT count, word FROM MyTable1 WHERE word LIKE 'F%' +> UNION ALL +> SELECT count, word FROM MyTable2; + +{% endhighlight %} +++{% highlight text %} +== Abstract Syntax Tree == +LogicalUnion(all=[true]) + LogicalFilter(condition=[LIKE($1, _UTF-16LE'F%')]) + FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, MyTable1]], fields=[count, word]) + FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, MyTable2]], fields=[count, word]) + + +== Optimized Logical Plan == +DataStreamUnion(all=[true], union all=[count, word]) + DataStreamCalc(select=[count, word], where=[LIKE(word, _UTF-16LE'F%')]) + TableSourceScan(table=[[default_catalog, default_database, MyTable1]], fields=[count, word]) + TableSourceScan(table=[[default_catalog, default_database, MyTable2]], fields=[count, word]) + +== Physical Execution Plan == +Stage 1 : Data Source + content : collect elements with CollectionInputFormat + +Stage 2 : Data Source + content : collect elements with CollectionInputFormat + + Stage 3 : Operator + content : from: (count, word) + ship_strategy : REBALANCE + + Stage 4 : Operator + content : where: (LIKE(word, _UTF-16LE'F%')), select: (count, word) + ship_strategy : FORWARD + + Stage 5 : Operator + content : from: (count, word) + ship_strategy : REBALANCE +{% endhighlight %} ++ +{% top %} + +## Syntax + +{% highlight sql %} +EXPLAIN PLAN FOR+{% endhighlight %} + +For query syntax, please refer to [Queries]({{ site.baseurl }}/dev/table/sql/queries.html#supported-syntax) page. +For INSERT, please refer to [INSERT]({{ site.baseurl }}/dev/table/sql/insert.html) page. diff --git a/docs/dev/table/sql/explain.zh.md b/docs/dev/table/sql/explain.zh.md new file mode 100644 index 00000000000000..e2f2c1e9d370ef --- /dev/null +++ b/docs/dev/table/sql/explain.zh.md @@ -0,0 +1,182 @@ +--- +title: "EXPLAIN 语句" +nav-parent_id: sql +nav-pos: 8 +--- + + +* This will be replaced by the TOC +{:toc} + +EXPLAIN 语句用来解释一条 query 语句或者 INSERT 语句的逻辑计划和优化后的计划。 + + +## 运行一条 EXPLAIN 语句 + +EXPLAIN 语句可以通过 `TableEnvironment` 的 `executeSql()` 执行，也可以在 [SQL CLI]({{ site.baseurl }}/zh/dev/table/sqlClient.html) 中执行 EXPLAIN 语句。 若 EXPLAIN 操作执行成功，`executeSql()` 方法返回解释的结果，否则会抛出异常。 + +以下的例子展示了如何在 TableEnvironment 和 SQL CLI 中执行一条 EXPLAIN 语句。 + + ++ +执行 `EXPLAIN` 语句后的结果为： ++{% highlight java %} +StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); +StreamTableEnvironment tEnv = StreamTableEnvironment.create(env); + +// register a table named "Orders" +tEnv.executeSql("CREATE TABLE MyTable1 (count bigint, work VARCHAR(256) WITH (...)"); +tEnv.executeSql("CREATE TABLE MyTable2 (count bigint, work VARCHAR(256) WITH (...)"); + +// explain SELECT statement through TableEnvironment.explainSql() +String explanation = tEnv.explainSql( + "SELECT count, word FROM MyTable1 WHERE word LIKE 'F%' " + + "UNION ALL " + + "SELECT count, word FROM MyTable2"); +System.out.println(explanation); + +// explain SELECT statement through TableEnvironment.executeSql() +TableResult tableResult = tEnv.executeSql( + "EXPLAIN PLAN FOR " + + "SELECT count, word FROM MyTable1 WHERE word LIKE 'F%' " + + "UNION ALL " + + "SELECT count, word FROM MyTable2"); +tableResult.print(); + +{% endhighlight %} ++ ++{% highlight scala %} +val env = StreamExecutionEnvironment.getExecutionEnvironment() +val tEnv = StreamTableEnvironment.create(env) + +// register a table named "Orders" +tEnv.executeSql("CREATE TABLE MyTable1 (count bigint, work VARCHAR(256) WITH (...)") +tEnv.executeSql("CREATE TABLE MyTable2 (count bigint, work VARCHAR(256) WITH (...)") + +// explain SELECT statement through TableEnvironment.explainSql() +val explanation = tEnv.explainSql( + "SELECT count, word FROM MyTable1 WHERE word LIKE 'F%' " + + "UNION ALL " + + "SELECT count, word FROM MyTable2") +println(explanation) + +// explain SELECT statement through TableEnvironment.executeSql() +val tableResult = tEnv.executeSql( + "EXPLAIN PLAN FOR " + + "SELECT count, word FROM MyTable1 WHERE word LIKE 'F%' " + + "UNION ALL " + + "SELECT count, word FROM MyTable2") +tableResult.print() + +{% endhighlight %} ++ ++{% highlight python %} +settings = EnvironmentSettings.new_instance()... +table_env = StreamTableEnvironment.create(env, settings) + +t_env.execute_sql("CREATE TABLE MyTable1 (count bigint, work VARCHAR(256) WITH (...)") +t_env.execute_sql("CREATE TABLE MyTable2 (count bigint, work VARCHAR(256) WITH (...)") + +# explain SELECT statement through TableEnvironment.explain_sql() +explanation1 = t_env.explain_sql( + "SELECT count, word FROM MyTable1 WHERE word LIKE 'F%' " + "UNION ALL " + "SELECT count, word FROM MyTable2") +print(explanation1) + +# explain SELECT statement through TableEnvironment.execute_sql() +table_result = t_env.execute_sql( + "EXPLAIN PLAN FOR " + "SELECT count, word FROM MyTable1 WHERE word LIKE 'F%' " + "UNION ALL " + "SELECT count, word FROM MyTable2") +table_result.print() + +{% endhighlight %} ++ ++{% highlight sql %} +Flink SQL> CREATE TABLE MyTable1 (count bigint, work VARCHAR(256); +[INFO] Table has been created. + +Flink SQL> CREATE TABLE MyTable2 (count bigint, work VARCHAR(256); +[INFO] Table has been created. + +Flink SQL> EXPLAIN PLAN FOR SELECT count, word FROM MyTable1 WHERE word LIKE 'F%' +> UNION ALL +> SELECT count, word FROM MyTable2; + +{% endhighlight %} +++{% highlight text %} +== Abstract Syntax Tree == +LogicalUnion(all=[true]) + LogicalFilter(condition=[LIKE($1, _UTF-16LE'F%')]) + FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, MyTable1]], fields=[count, word]) + FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, MyTable2]], fields=[count, word]) + + +== Optimized Logical Plan == +DataStreamUnion(all=[true], union all=[count, word]) + DataStreamCalc(select=[count, word], where=[LIKE(word, _UTF-16LE'F%')]) + TableSourceScan(table=[[default_catalog, default_database, MyTable1]], fields=[count, word]) + TableSourceScan(table=[[default_catalog, default_database, MyTable2]], fields=[count, word]) + +== Physical Execution Plan == +Stage 1 : Data Source + content : collect elements with CollectionInputFormat + +Stage 2 : Data Source + content : collect elements with CollectionInputFormat + + Stage 3 : Operator + content : from: (count, word) + ship_strategy : REBALANCE + + Stage 4 : Operator + content : where: (LIKE(word, _UTF-16LE'F%')), select: (count, word) + ship_strategy : FORWARD + + Stage 5 : Operator + content : from: (count, word) + ship_strategy : REBALANCE +{% endhighlight %} ++ +{% top %} + +## 语法 + +{% highlight sql %} +EXPLAIN PLAN FOR+{% endhighlight %} + +请参阅 [Queries]({{ site.baseurl }}/zh/dev/table/sql/queries.html#supported-syntax) 页面获得 query 的语法。 +请参阅 [INSERT]({{ site.baseurl }}/zh/dev/table/sql/insert.html) 页面获得 INSERT 的语法。 + +{% top %} diff --git a/docs/dev/table/sql/hints.md b/docs/dev/table/sql/hints.md new file mode 100644 index 00000000000000..d8c4c900316e2c --- /dev/null +++ b/docs/dev/table/sql/hints.md @@ -0,0 +1,88 @@ +--- +title: "SQL Hints" +nav-parent_id: sql +nav-pos: 6 +--- + + +* This will be replaced by the TOC +{:toc} + +SQL hints can be used with SQL statements to alter execution plans. This chapter explains how to use hints to force various approaches. + +Generally a hint can be used to: + +- Enforce planner: there's no perfect planner, so it makes sense to implement hints to +allow user better control the execution; +- Append meta data(or statistics): some statistics like “table index for scan” and +“skew info of some shuffle keys” are somewhat dynamic for the query, it would be very +convenient to config them with hints because our planning metadata from the planner is very +often not that accurate; +- Operator resource constraints: for many cases, we would give a default resource +configuration for the execution operators, i.e. min parallelism or +managed memory (resource consuming UDF) or special resource requirement (GPU or SSD disk) +and so on, it would be very flexible to profile the resource with hints per query(instead of the Job). + +## Dynamic Table Options +Dynamic table options allows to specify or override table options dynamically, different with static table options defined with SQL DDL or connect API, +these options can be specified flexibly in per-table scope within each query. + +Thus it is very suitable to use for the ad-hoc queries in interactive terminal, for example, in the SQL-CLI, +you can specify to ignore the parse error for a CSV source just by adding a dynamic option `/*+ OPTIONS('csv.ignore-parse-errors'='true') */`. + +Note: Dynamic table options default is forbidden to use because it may change the semantics of the query. +You need to set the config option `table.dynamic-table-options.enabled` to be `true` explicitly (default is false), +See the Configuration for details on how to set up the config options. + +### Syntax +In order to not break the SQL compatibility, we use the Oracle style SQL hint syntax: +{% highlight sql %} +table_path /*+ OPTIONS(key=val [, key=val]*) */ + +key: + stringLiteral +val: + stringLiteral + +{% endhighlight %} + +### Examples + +{% highlight sql %} + +CREATE TABLE kafka_table1 (id BIGINT, name STRING, age INT) WITH (...); +CREATE TABLE kafka_table2 (id BIGINT, name STRING, age INT) WITH (...); + +-- override table options in query source +select id, name from kafka_table1 /*+ OPTIONS('scan.startup.mode'='earliest-offset') */; + +-- override table options in join +select * from + kafka_table1 /*+ OPTIONS('scan.startup.mode'='earliest-offset') */ t1 + join + kafka_table2 /*+ OPTIONS('scan.startup.mode'='earliest-offset') */ t2 + on t1.id = t2.id; + +-- override table options for INSERT target table +insert into kafka_table1 /*+ OPTIONS('sink.partitioner'='round-robin') */ select * from kafka_table2; + +{% endhighlight %} + +{% top %} \ No newline at end of file diff --git a/docs/dev/table/sql/hints.zh.md b/docs/dev/table/sql/hints.zh.md new file mode 100644 index 00000000000000..d8c4c900316e2c --- /dev/null +++ b/docs/dev/table/sql/hints.zh.md @@ -0,0 +1,88 @@ +--- +title: "SQL Hints" +nav-parent_id: sql +nav-pos: 6 +--- + + +* This will be replaced by the TOC +{:toc} + +SQL hints can be used with SQL statements to alter execution plans. This chapter explains how to use hints to force various approaches. + +Generally a hint can be used to: + +- Enforce planner: there's no perfect planner, so it makes sense to implement hints to +allow user better control the execution; +- Append meta data(or statistics): some statistics like “table index for scan” and +“skew info of some shuffle keys” are somewhat dynamic for the query, it would be very +convenient to config them with hints because our planning metadata from the planner is very +often not that accurate; +- Operator resource constraints: for many cases, we would give a default resource +configuration for the execution operators, i.e. min parallelism or +managed memory (resource consuming UDF) or special resource requirement (GPU or SSD disk) +and so on, it would be very flexible to profile the resource with hints per query(instead of the Job). + +## Dynamic Table Options +Dynamic table options allows to specify or override table options dynamically, different with static table options defined with SQL DDL or connect API, +these options can be specified flexibly in per-table scope within each query. + +Thus it is very suitable to use for the ad-hoc queries in interactive terminal, for example, in the SQL-CLI, +you can specify to ignore the parse error for a CSV source just by adding a dynamic option `/*+ OPTIONS('csv.ignore-parse-errors'='true') */`. + +Note: Dynamic table options default is forbidden to use because it may change the semantics of the query. +You need to set the config option `table.dynamic-table-options.enabled` to be `true` explicitly (default is false), +See the Configuration for details on how to set up the config options. + +### Syntax +In order to not break the SQL compatibility, we use the Oracle style SQL hint syntax: +{% highlight sql %} +table_path /*+ OPTIONS(key=val [, key=val]*) */ + +key: + stringLiteral +val: + stringLiteral + +{% endhighlight %} + +### Examples + +{% highlight sql %} + +CREATE TABLE kafka_table1 (id BIGINT, name STRING, age INT) WITH (...); +CREATE TABLE kafka_table2 (id BIGINT, name STRING, age INT) WITH (...); + +-- override table options in query source +select id, name from kafka_table1 /*+ OPTIONS('scan.startup.mode'='earliest-offset') */; + +-- override table options in join +select * from + kafka_table1 /*+ OPTIONS('scan.startup.mode'='earliest-offset') */ t1 + join + kafka_table2 /*+ OPTIONS('scan.startup.mode'='earliest-offset') */ t2 + on t1.id = t2.id; + +-- override table options for INSERT target table +insert into kafka_table1 /*+ OPTIONS('sink.partitioner'='round-robin') */ select * from kafka_table2; + +{% endhighlight %} + +{% top %} \ No newline at end of file diff --git a/docs/dev/table/sql/index.md b/docs/dev/table/sql/index.md index 1affb421cee517..63d549c7547b84 100644 --- a/docs/dev/table/sql/index.md +++ b/docs/dev/table/sql/index.md @@ -29,10 +29,15 @@ This page describes the SQL language supported in Flink, including Data Definiti This page lists all the supported statements supported in Flink SQL for now: - [SELECT (Queries)](queries.html) -- [CREATE TABLE, DATABASE, FUNCTION](create.html) -- [DROP TABLE, DATABASE, FUNCTION](drop.html) +- [CREATE TABLE, DATABASE, VIEW, FUNCTION](create.html) +- [DROP TABLE, DATABASE, VIEW, FUNCTION](drop.html) - [ALTER TABLE, DATABASE, FUNCTION](alter.html) - [INSERT](insert.html) +- [SQL HINTS](hints.html) +- [DESCRIBE](describe.html) +- [EXPLAIN](explain.html) +- [USE](use.html) +- [SHOW](show.html) ## Data Types diff --git a/docs/dev/table/sql/index.zh.md b/docs/dev/table/sql/index.zh.md index 5cdc8d0ced07dc..0def8c2f79c88c 100644 --- a/docs/dev/table/sql/index.zh.md +++ b/docs/dev/table/sql/index.zh.md @@ -28,11 +28,16 @@ under the License. 本页面列出了目前 Flink SQL 所支持的所有语句： -- [SELECT (查询)](queries.html) -- [CREATE TABLE, DATABASE, FUNCTION](create.html) -- [DROP TABLE, DATABASE, FUNCTION](drop.html) +- [SELECT (Queries)](queries.html) +- [CREATE TABLE, DATABASE, VIEW, FUNCTION](create.html) +- [DROP TABLE, DATABASE, VIEW, FUNCTION](drop.html) - [ALTER TABLE, DATABASE, FUNCTION](alter.html) - [INSERT](insert.html) +- [SQL HINTS](hints.html) +- [DESCRIBE](describe.html) +- [EXPLAIN](explain.html) +- [USE](use.html) +- [SHOW](show.html) ## 数据类型 diff --git a/docs/dev/table/sql/insert.md b/docs/dev/table/sql/insert.md index 96052adff5945f..01fa413e05ddbf 100644 --- a/docs/dev/table/sql/insert.md +++ b/docs/dev/table/sql/insert.md @@ -29,9 +29,10 @@ INSERT statements are used to add rows to a table. ## Run an INSERT statement -INSERT statements are specified with the `sqlUpdate()` method of the `TableEnvironment` or executed in [SQL CLI]({{ site.baseurl }}/dev/table/sqlClient.html). The method `sqlUpdate()` for INSERT statements is a lazy execution, they will be executed only when `TableEnvironment.execute(jobName)` is invoked. +Single INSERT statement can be executed through the `executeSql()` method of the `TableEnvironment`, or executed in [SQL CLI]({{ site.baseurl }}/dev/table/sqlClient.html). The `executeSql()` method for INSERT statement will submit a Flink job immediately, and return a `TableResult` instance which associates the submitted job. +Multiple INSERT statements can be executed through the `addInsertSql()` method of the `StatementSet` which can be created by the `TableEnvironment.createStatementSet()` method. The `addInsertSql()` method is a lazy execution, they will be executed only when `StatementSet.execute()` is invoked. -The following examples show how to run an INSERT statement in `TableEnvironment` and in SQL CLI. +The following examples show how to run a single INSERT statement in `TableEnvironment` and in SQL CLI, run multiple INSERT statements in `StatementSet`. @@ -40,12 +41,31 @@ EnvironmentSettings settings = EnvironmentSettings.newInstance()... TableEnvironment tEnv = TableEnvironment.create(settings); // register a source table named "Orders" and a sink table named "RubberOrders" -tEnv.sqlUpdate("CREATE TABLE Orders (`user` BIGINT, product VARCHAR, amount INT) WITH (...)"); -tEnv.sqlUpdate("CREATE TABLE RubberOrders(product VARCHAR, amount INT) WITH (...)"); +tEnv.executeSql("CREATE TABLE Orders (`user` BIGINT, product VARCHAR, amount INT) WITH (...)"); +tEnv.executeSql("CREATE TABLE RubberOrders(product VARCHAR, amount INT) WITH (...)"); -// run a SQL update query on the registered source table and emit the result to registered sink table -tEnv.sqlUpdate( +// run a single INSERT query on the registered source table and emit the result to registered sink table +TableResult tableResult1 = tEnv.executeSql( "INSERT INTO RubberOrders SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'"); +// get job status through TableResult +System.out.println(tableResult1.getJobClient().get().getJobStatus()); + +//---------------------------------------------------------------------------- +// register another sink table named "GlassOrders" for multiple INSERT queries +tEnv.executeSql("CREATE TABLE GlassOrders(product VARCHAR, amount INT) WITH (...)"); + +// run multiple INSERT queries on the registered source table and emit the result to registered sink tables +StatementSet stmtSet = tEnv.createStatementSet(); +// only single INSERT query can be accepted by `addInsertSql` method +stmtSet.addInsertSql( + "INSERT INTO RubberOrders SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'"); +stmtSet.addInsertSql( + "INSERT INTO GlassOrders SELECT product, amount FROM Orders WHERE product LIKE '%Glass%'"); +// execute all statements together +TableResult tableResult2 = stmtSet.execute(); +// get job status through TableResult +System.out.println(tableResult2.getJobClient().get().getJobStatus()); + {% endhighlight %}@@ -55,27 +75,65 @@ val settings = EnvironmentSettings.newInstance()... val tEnv = TableEnvironment.create(settings) // register a source table named "Orders" and a sink table named "RubberOrders" -tEnv.sqlUpdate("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)") -tEnv.sqlUpdate("CREATE TABLE RubberOrders(product STRING, amount INT) WITH (...)") +tEnv.executeSql("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)") +tEnv.executeSql("CREATE TABLE RubberOrders(product STRING, amount INT) WITH (...)") -// run a SQL update query on the registered source table and emit the result to registered sink table -tEnv.sqlUpdate( +// run a single INSERT query on the registered source table and emit the result to registered sink table +val tableResult1 = tEnv.executeSql( "INSERT INTO RubberOrders SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'") +// get job status through TableResult +println(tableResult1.getJobClient().get().getJobStatus()) + +//---------------------------------------------------------------------------- +// register another sink table named "GlassOrders" for multiple INSERT queries +tEnv.executeSql("CREATE TABLE GlassOrders(product VARCHAR, amount INT) WITH (...)") + +// run multiple INSERT queries on the registered source table and emit the result to registered sink tables +val stmtSet = tEnv.createStatementSet() +// only single INSERT query can be accepted by `addInsertSql` method +stmtSet.addInsertSql( + "INSERT INTO RubberOrders SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'") +stmtSet.addInsertSql( + "INSERT INTO GlassOrders SELECT product, amount FROM Orders WHERE product LIKE '%Glass%'") +// execute all statements together +val tableResult2 = stmtSet.execute() +// get job status through TableResult +println(tableResult2.getJobClient().get().getJobStatus()) + {% endhighlight %}{% highlight python %} -settings = EnvironmentSettings.newInstance()... -table_env = TableEnvironment.create(settings) +settings = EnvironmentSettings.new_instance()... +table_env = StreamTableEnvironment.create(env, settings) # register a source table named "Orders" and a sink table named "RubberOrders" -table_env.sqlUpdate("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)") -table_env.sqlUpdate("CREATE TABLE RubberOrders(product STRING, amount INT) WITH (...)") +table_env.execute_sql("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)") +table_env.execute_sql("CREATE TABLE RubberOrders(product STRING, amount INT) WITH (...)") + +# run a single INSERT query on the registered source table and emit the result to registered sink table +table_result1 = table_env \ + .execute_sql("INSERT INTO RubberOrders SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'") +# get job status through TableResult +print(table_result1get_job_client().get_job_status()) + +#---------------------------------------------------------------------------- +# register another sink table named "GlassOrders" for multiple INSERT queries +table_env.execute_sql("CREATE TABLE GlassOrders(product VARCHAR, amount INT) WITH (...)") + +# run multiple INSERT queries on the registered source table and emit the result to registered sink tables +stmt_set = table_env.create_statement_set() +# only single INSERT query can be accepted by `add_insert_sql` method +stmt_set \ + .add_insert_sql("INSERT INTO RubberOrders SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'") +stmt_set \ + .add_insert_sql("INSERT INTO GlassOrders SELECT product, amount FROM Orders WHERE product LIKE '%Glass%'") +# execute all statements together +table_result2 = stmt_set.execute() +# get job status through TableResult +print(table_result2.get_job_client().get_job_status()) -# run a SQL update query on the registered source table and emit the result to registered sink table -table_env \ - .sqlUpdate("INSERT INTO RubberOrders SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'") {% endhighlight %}diff --git a/docs/dev/table/sql/insert.zh.md b/docs/dev/table/sql/insert.zh.md index 14ad9a4536cdb2..863447c193a31d 100644 --- a/docs/dev/table/sql/insert.zh.md +++ b/docs/dev/table/sql/insert.zh.md @@ -29,9 +29,10 @@ INSERT 语句用来向表中添加行。 ## 执行 INSERT 语句 -可以使用 `TableEnvironment` 中的 `sqlUpdate()` 方法执行 INSERT 语句，也可以在 [SQL CLI]({{ site.baseurl }}/zh/dev/table/sqlClient.html) 中执行 INSERT 语句。`sqlUpdate()` 方法执行 INSERT 语句时时懒执行的，只有当`TableEnvironment.execute(jobName)`被调用时才会被执行。 +单条 INSERT 语句，可以使用 `TableEnvironment` 中的 `executeSql()` 方法执行，也可以在 [SQL CLI]({{ site.baseurl }}/zh/dev/table/sqlClient.html) 中执行 INSERT 语句。`executeSql()` 方法执行 INSERT 语句时会立即提交一个 Flink 作业，并且返回一个 TableResult 对象，通过该对象可以获取 JobClient 方便的操作提交的作业。 +多条 INSERT 语句，使用 `TableEnvironment` 中的 `createStatementSet` 创建一个 `StatementSet` 对象，然后使用 `StatementSet` 中的 `addInsertSql()` 方法添加多条 INSERT 语句，最后通过 `StatementSet` 中的 `execute()` 方法来执行。 -以下的例子展示了如何在 `TableEnvironment` 和 SQL CLI 中执行一个 INSERT 语句。 +以下的例子展示了如何在 `TableEnvironment` 和 SQL CLI 中执行一条 INSERT 语句，或者通过 `StatementSet` 执行多条 INSERT 语句。@@ -40,12 +41,31 @@ EnvironmentSettings settings = EnvironmentSettings.newInstance()... TableEnvironment tEnv = TableEnvironment.create(settings); // 注册一个 "Orders" 源表，和 "RubberOrders" 结果表 -tEnv.sqlUpdate("CREATE TABLE Orders (`user` BIGINT, product VARCHAR, amount INT) WITH (...)"); -tEnv.sqlUpdate("CREATE TABLE RubberOrders(product VARCHAR, amount INT) WITH (...)"); +tEnv.executeSql("CREATE TABLE Orders (`user` BIGINT, product VARCHAR, amount INT) WITH (...)"); +tEnv.executeSql("CREATE TABLE RubberOrders(product VARCHAR, amount INT) WITH (...)"); -// 运行一个 INSERT 语句，将源表的数据输出到结果表中 -tEnv.sqlUpdate( +// 运行一条 INSERT 语句，将源表的数据输出到结果表中 +TableResult tableResult1 = tEnv.executeSql( "INSERT INTO RubberOrders SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'"); +// 通过 TableResult 来获取作业状态 +System.out.println(tableResult1.getJobClient().get().getJobStatus()); + +//---------------------------------------------------------------------------- +// 注册一个 "GlassOrders" 结果表用于运行多 INSERT 语句 +tEnv.executeSql("CREATE TABLE GlassOrders(product VARCHAR, amount INT) WITH (...)"); + +// 运行多条 INSERT 语句，将原表数据输出到多个结果表中 +StatementSet stmtSet = tEnv.createStatementSet(); +// `addInsertSql` 方法每次只接收单条 INSERT 语句 +stmtSet.addInsertSql( + "INSERT INTO RubberOrders SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'"); +stmtSet.addInsertSql( + "INSERT INTO GlassOrders SELECT product, amount FROM Orders WHERE product LIKE '%Glass%'"); +// 执行刚刚添加的所有 INSERT 语句 +TableResult tableResult2 = stmtSet.execute(); +// 通过 TableResult 来获取作业状态 +System.out.println(tableResult1.getJobClient().get().getJobStatus()); + {% endhighlight %}@@ -55,27 +75,66 @@ val settings = EnvironmentSettings.newInstance()... val tEnv = TableEnvironment.create(settings) // 注册一个 "Orders" 源表，和 "RubberOrders" 结果表 -tEnv.sqlUpdate("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)") -tEnv.sqlUpdate("CREATE TABLE RubberOrders(product STRING, amount INT) WITH (...)") +tEnv.executeSql("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)") +tEnv.executeSql("CREATE TABLE RubberOrders(product STRING, amount INT) WITH (...)") // 运行一个 INSERT 语句，将源表的数据输出到结果表中 -tEnv.sqlUpdate( +val tableResult1 = tEnv.executeSql( "INSERT INTO RubberOrders SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'") +// 通过 TableResult 来获取作业状态 +println(tableResult1.getJobClient().get().getJobStatus()) + +//---------------------------------------------------------------------------- +// 注册一个 "GlassOrders" 结果表用于运行多 INSERT 语句 +tEnv.executeSql("CREATE TABLE GlassOrders(product VARCHAR, amount INT) WITH (...)"); + +// 运行多个 INSERT 语句，将原表数据输出到多个结果表中 +val stmtSet = tEnv.createStatementSet() +// `addInsertSql` 方法每次只接收单条 INSERT 语句 +stmtSet.addInsertSql( + "INSERT INTO RubberOrders SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'") +stmtSet.addInsertSql( + "INSERT INTO GlassOrders SELECT product, amount FROM Orders WHERE product LIKE '%Glass%'") +// 执行刚刚添加的所有 INSERT 语句 +val tableResult2 = stmtSet.execute() +// 通过 TableResult 来获取作业状态 +println(tableResult1.getJobClient().get().getJobStatus()) + {% endhighlight %}{% highlight python %} -settings = EnvironmentSettings.newInstance()... -table_env = TableEnvironment.create(settings) +settings = EnvironmentSettings.new_instance()... +table_env = StreamTableEnvironment.create(env, settings) # 注册一个 "Orders" 源表，和 "RubberOrders" 结果表 -table_env.sqlUpdate("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)") -table_env.sqlUpdate("CREATE TABLE RubberOrders(product STRING, amount INT) WITH (...)") +table_env.executeSql("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)") +table_env.executeSql("CREATE TABLE RubberOrders(product STRING, amount INT) WITH (...)") + +# 运行一条 INSERT 语句，将源表的数据输出到结果表中 +table_result1 = table_env \ + .executeSql("INSERT INTO RubberOrders SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'") +# 通过 TableResult 来获取作业状态 +print(table_result1.get_job_client().get_job_status()) + +#---------------------------------------------------------------------------- +# 注册一个 "GlassOrders" 结果表用于运行多 INSERT 语句 +table_env.execute_sql("CREATE TABLE GlassOrders(product VARCHAR, amount INT) WITH (...)") + +# 运行多条 INSERT 语句，将原表数据输出到多个结果表中 +stmt_set = table_env.create_statement_set() +# `add_insert_sql` 方法每次只接收单条 INSERT 语句 +stmt_set \ + .add_insert_sql("INSERT INTO RubberOrders SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'") +stmt_set \ + .add_insert_sql("INSERT INTO GlassOrders SELECT product, amount FROM Orders WHERE product LIKE '%Glass%'") +# 执行刚刚添加的所有 INSERT 语句 +table_result2 = stmt_set.execute() +# 通过 TableResult 来获取作业状态 +print(table_result2.get_job_client().get_job_status()) + -# 运行一个 INSERT 语句，将源表的数据输出到结果表中 -table_env \ - .sqlUpdate("INSERT INTO RubberOrders SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'") {% endhighlight %}diff --git a/docs/dev/table/sql/queries.md b/docs/dev/table/sql/queries.md index f860d5a10bb659..df78622cd5a5da 100644 --- a/docs/dev/table/sql/queries.md +++ b/docs/dev/table/sql/queries.md @@ -25,7 +25,7 @@ under the License. * This will be replaced by the TOC {:toc} -SELECT queries are specified with the `sqlQuery()` method of the `TableEnvironment`. The method returns the result of the SELECT query as a `Table`. A `Table` can be used in [subsequent SQL and Table API queries]({{ site.baseurl }}/dev/table/common.html#mixing-table-api-and-sql), be [converted into a DataSet or DataStream]({{ site.baseurl }}/dev/table/common.html#integration-with-datastream-and-dataset-api), or [written to a TableSink]({{ site.baseurl }}/dev/table/common.html#emit-a-table). SQL and Table API queries can be seamlessly mixed and are holistically optimized and translated into a single program. +SELECT statements and VALUES statements are specified with the `sqlQuery()` method of the `TableEnvironment`. The method returns the result of the SELECT statement (or the VALUES statements) as a `Table`. A `Table` can be used in [subsequent SQL and Table API queries]({{ site.baseurl }}/dev/table/common.html#mixing-table-api-and-sql), be [converted into a DataSet or DataStream]({{ site.baseurl }}/dev/table/common.html#integration-with-datastream-and-dataset-api), or [written to a TableSink]({{ site.baseurl }}/dev/table/common.html#emit-a-table). SQL and Table API queries can be seamlessly mixed and are holistically optimized and translated into a single program. In order to access a table in a SQL query, it must be [registered in the TableEnvironment]({{ site.baseurl }}/dev/table/common.html#register-tables-in-the-catalog). A table can be registered from a [TableSource]({{ site.baseurl }}/dev/table/common.html#register-a-tablesource), [Table]({{ site.baseurl }}/dev/table/common.html#register-a-table), [CREATE TABLE statement](#create-table), [DataStream, or DataSet]({{ site.baseurl }}/dev/table/common.html#register-a-datastream-or-dataset-as-table). Alternatively, users can also [register catalogs in a TableEnvironment]({{ site.baseurl }}/dev/table/catalogs.html) to specify the location of the data sources. @@ -58,7 +58,6 @@ tableEnv.createTemporaryView("Orders", ds, $("user"), $("product"), $("amount")) Table result2 = tableEnv.sqlQuery( "SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'"); -// SQL update with a registered table // create and register a TableSink final Schema schema = new Schema() .field("product", DataTypes.STRING()) @@ -69,8 +68,8 @@ tableEnv.connect(new FileSystem("/path/to/file")) .withSchema(schema) .createTemporaryTable("RubberOrders"); -// run a SQL update query on the Table and emit the result to the TableSink -tableEnv.sqlUpdate( +// run an INSERT SQL on the Table and emit the result to the TableSink +tableEnv.executeSql( "INSERT INTO RubberOrders SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'"); {% endhighlight %}++ +{% top %} + + +## Syntax Flink parses SQL using [Apache Calcite](https://calcite.apache.org/docs/reference.html), which supports standard ANSI SQL. @@ -198,10 +286,19 @@ tableReference: [ [ AS ] alias [ '(' columnAlias [, columnAlias ]* ')' ] ] tablePrimary: - [ TABLE ] [ [ catalogName . ] schemaName . ] tableName + [ TABLE ] [ [ catalogName . ] schemaName . ] tableName [ dynamicTableOptions ] | LATERAL TABLE '(' functionName '(' expression [, expression ]* ')' ')' | UNNEST '(' expression ')' +dynamicTableOptions: + /*+ OPTIONS(key=val [, key=val]*) */ + +key: + stringLiteral + +val: + stringLiteral + values: VALUES expression [, expression ]* @@ -285,57 +382,6 @@ String literals must be enclosed in single quotes (e.g., `SELECT 'Hello World'`) ## Operations -### Show and Use - -+{% highlight java %} +StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); +StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env, settings); +// enable checkpointing +tableEnv.getConfig().getConfiguration().set( + ExecutionCheckpointingOptions.CHECKPOINTING_MODE, CheckpointingMode.EXACTLY_ONCE); +tableEnv.getConfig().getConfiguration().set( + ExecutionCheckpointingOptions.CHECKPOINTING_INTERVAL, Duration.ofSeconds(10)); + +tableEnv.executeSql("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)"); + +// execute SELECT statement +TableResult tableResult1 = tableEnv.executeSql("SELECT * FROM Orders"); +// use try-with-resources statement to make sure the iterator will be closed automatically +try (CloseableIterator+it = tableResult1.collect()) { + while(it.hasNext()) { + Row row = it.next(); + // handle row + } +} + +// execute Table +TableResult tableResult2 = tableEnv.sqlQuery("SELECT * FROM Orders").execute(); +tableResult2.print(); + +{% endhighlight %} +
+{% highlight scala %} +val env = StreamExecutionEnvironment.getExecutionEnvironment() +val tableEnv = StreamTableEnvironment.create(env, settings) +// enable checkpointing +tableEnv.getConfig.getConfiguration.set( + ExecutionCheckpointingOptions.CHECKPOINTING_MODE, CheckpointingMode.EXACTLY_ONCE) +tableEnv.getConfig.getConfiguration.set( + ExecutionCheckpointingOptions.CHECKPOINTING_INTERVAL, Duration.ofSeconds(10)) + +tableEnv.executeSql("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)") + +// execute SELECT statement +val tableResult1 = tableEnv.executeSql("SELECT * FROM Orders") +val it = tableResult1.collect() +try while (it.hasNext) { + val row = it.next + // handle row +} +finally it.close() // close the iterator to avoid resource leak + +// execute Table +val tableResult2 = tableEnv.sqlQuery("SELECT * FROM Orders").execute() +tableResult2.print() + +{% endhighlight %} +++{% highlight python %} +env = StreamExecutionEnvironment.get_execution_environment() +table_env = StreamTableEnvironment.create(env, settings) +# enable checkpointing +table_env.get_config().get_configuration().set_string("execution.checkpointing.mode", "EXACTLY_ONCE") +table_env.get_config().get_configuration().set_string("execution.checkpointing.interval", "10s") + +table_env.execute_sql("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)") + +# execute SELECT statement +table_result1 = table_env.execute_sql("SELECT * FROM Orders") +table_result1.print() + +# execute Table +table_result2 = table_env.sql_query("SELECT * FROM Orders").execute() +table_result2.print() + +{% endhighlight %} ++-- ### Scan, Projection, and Filter- -
-- - - -Operation -Description -- -- Show -
- Batch Streaming -- -Show all catalogs
-{% highlight sql %} -SHOW CATALOGS; -{% endhighlight %} -Show all databases in the current catalog
-{% highlight sql %} -SHOW DATABASES; -{% endhighlight %} -Show all tables in the current database in the current catalog
-{% highlight sql %} -SHOW TABLES; -{% endhighlight %} -- - -- Use -
- Batch Streaming -- -Set current catalog for the session
-{% highlight sql %} -USE CATALOG mycatalog; -{% endhighlight %} -Set current database of the current catalog for the session
-{% highlight sql %} -USE mydatabase; -{% endhighlight %} -@@ -566,15 +612,15 @@ FROM Orders FULL OUTER JOIN Product ON Orders.productId = Product.id- Time-windowed Join
+Interval Join
Batch Streaming- Note: Time-windowed joins are a subset of regular joins that can be processed in a streaming fashion.
+Note: Interval joins are a subset of regular joins that can be processed in a streaming fashion.
-A time-windowed join requires at least one equi-join predicate and a join condition that bounds the time on both sides. Such a condition can be defined by two appropriate range predicates (
-<, <=, >=, >), aBETWEENpredicate, or a single equality predicate that compares time attributes of the same type (i.e., processing time or event time) of both input tables.For example, the following predicates are valid window join conditions:
+A interval join requires at least one equi-join predicate and a join condition that bounds the time on both sides. Such a condition can be defined by two appropriate range predicates (
+<, <=, >=, >), aBETWEENpredicate, or a single equality predicate that compares time attributes of the same type (i.e., processing time or event time) of both input tables.For example, the following predicates are valid interval join conditions:
ltime = rtime
@@ -1127,7 +1173,7 @@ The start and end timestamps of group windows as well as time attributes can be
SESSION_END(time_attr, interval)
Returns the timestamp of the exclusive upper bound of the corresponding tumbling, hopping, or session window.
-Note: The exclusive upper bound timestamp cannot be used as a rowtime attribute in subsequent time-based operations, such as time-windowed joins and group window or over window aggregations.
Note: The exclusive upper bound timestamp cannot be used as a rowtime attribute in subsequent time-based operations, such as interval joins and group window or over window aggregations.
@@ -1136,7 +1182,7 @@ The start and end timestamps of group windows as well as time attributes can be SESSION_ROWTIME(time_attr, interval)
Returns the timestamp of the inclusive upper bound of the corresponding tumbling, hopping, or session window.
-The resulting attribute is a rowtime attribute that can be used in subsequent time-based operations such as time-windowed joins and group window or over window aggregations.
The resulting attribute is a rowtime attribute that can be used in subsequent time-based operations such as interval joins and group window or over window aggregations.
@@ -1144,7 +1190,7 @@ The start and end timestamps of group windows as well as time attributes can be -HOP_PROCTIME(time_attr, interval, interval)
SESSION_PROCTIME(time_attr, interval)
Returns a proctime attribute that can be used in subsequent time-based operations such as time-windowed joins and group window or over window aggregations.
Returns a proctime attribute that can be used in subsequent time-based operations such as interval joins and group window or over window aggregations.
+{% top %} -## 支持的语法 +## 语法 Flink 通过支持标准 ANSI SQL的 [Apache Calcite](https://calcite.apache.org/docs/reference.html) 解析 SQL。 @@ -198,10 +284,19 @@ tableReference: [ [ AS ] alias [ '(' columnAlias [, columnAlias ]* ')' ] ] tablePrimary: - [ TABLE ] [ [ catalogName . ] schemaName . ] tableName + [ TABLE ] [ [ catalogName . ] schemaName . ] tableName [ dynamicTableOptions ] | LATERAL TABLE '(' functionName '(' expression [, expression ]* ')' ')' | UNNEST '(' expression ')' +dynamicTableOptions: + /*+ OPTIONS(key=val [, key=val]*) */ + +key: + stringLiteral + +val: + stringLiteral + values: VALUES expression [, expression ]* @@ -285,57 +380,6 @@ Flink SQL 对于标识符（表、属性、函数名）有类似于 Java 的词 ## 操作符 -### Show 与 Use - -+{% highlight java %} +StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); +StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env, settings); +// enable checkpointing +tableEnv.getConfig().getConfiguration().set( + ExecutionCheckpointingOptions.CHECKPOINTING_MODE, CheckpointingMode.EXACTLY_ONCE); +tableEnv.getConfig().getConfiguration().set( + ExecutionCheckpointingOptions.CHECKPOINTING_INTERVAL, Duration.ofSeconds(10)); + +tableEnv.executeSql("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)"); + +// execute SELECT statement +TableResult tableResult1 = tableEnv.executeSql("SELECT * FROM Orders"); +// use try-with-resources statement to make sure the iterator will be closed automatically +try (CloseableIterator+it = tableResult1.collect()) { + while(it.hasNext()) { + Row row = it.next(); + // handle row + } +} + +// execute Table +TableResult tableResult2 = tableEnv.sqlQuery("SELECT * FROM Orders").execute(); +tableResult2.print(); + +{% endhighlight %} +
+{% highlight scala %} +val env = StreamExecutionEnvironment.getExecutionEnvironment() +val tableEnv = StreamTableEnvironment.create(env, settings) +// enable checkpointing +tableEnv.getConfig.getConfiguration.set( + ExecutionCheckpointingOptions.CHECKPOINTING_MODE, CheckpointingMode.EXACTLY_ONCE) +tableEnv.getConfig.getConfiguration.set( + ExecutionCheckpointingOptions.CHECKPOINTING_INTERVAL, Duration.ofSeconds(10)) + +tableEnv.executeSql("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)") + +// execute SELECT statement +val tableResult1 = tableEnv.executeSql("SELECT * FROM Orders") +val it = tableResult1.collect() +try while (it.hasNext) { + val row = it.next + // handle row +} +finally it.close() // close the iterator to avoid resource leak + +// execute Table +val tableResult2 = tableEnv.sqlQuery("SELECT * FROM Orders").execute() +tableResult2.print() + +{% endhighlight %} +++{% highlight python %} +env = StreamExecutionEnvironment.get_execution_environment() +table_env = StreamTableEnvironment.create(env, settings) +# enable checkpointing +table_env.get_config().get_configuration().set_string("execution.checkpointing.mode", "EXACTLY_ONCE") +table_env.get_config().get_configuration().set_string("execution.checkpointing.interval", "10s") + +table_env.execute_sql("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)") + +# execute SELECT statement +table_result1 = table_env.execute_sql("SELECT * FROM Orders") +table_result1.print() + +# execute Table +table_result2 = table_env.sql_query("SELECT * FROM Orders").execute() +table_result2.print() + {% endhighlight %}-- ### Scan、Projection 与 Filter- -
-- - - -操作符 -描述 -- -- Show -
- 批处理 流处理 -- -显示所有 catalog
-{% highlight sql %} -SHOW CATALOGS; -{% endhighlight %} -显示当前 catalog 中所有的数据库
-{% highlight sql %} -SHOW DATABASES; -{% endhighlight %} -显示当前数据库、Catalog中的所有表
-{% highlight sql %} -SHOW TABLES; -{% endhighlight %} -- - -- Use -
- 批处理 流处理 -- -为本次会话设置 catalog
-{% highlight sql %} -USE CATALOG mycatalog; -{% endhighlight %} -为会话设置一个属于当前 catalog 的数据库
-{% highlight sql %} -USE mydatabase; -{% endhighlight %} -@@ -566,15 +610,15 @@ FROM Orders FULL OUTER JOIN Product ON Orders.productId = Product.id- Time-windowed Join
+Interval Join
批处理 流处理- 注意： 时间窗口 join 是常规 join 的子集，可以使用流的方式进行处理。
+注意：Interval join （时间区间关联）是常规 join 的子集，可以使用流的方式进行处理。
-时间窗口join需要至少一个 equi-join 谓词和一个限制了双方时间的 join 条件。例如使用两个适当的范围谓词（
-<, <=, >=, >），一个BETWEEN谓词或一个比较两个输入表中相同类型的 时间属性 （即处理时间和事件时间）的相等谓词比如，以下谓词是合法的窗口 join 条件：
+Interval join需要至少一个 equi-join 谓词和一个限制了双方时间的 join 条件。例如使用两个适当的范围谓词（
+<, <=, >=, >），一个BETWEEN谓词或一个比较两个输入表中相同类型的 时间属性 （即处理时间和事件时间）的相等谓词比如，以下谓词是合法的 interval join 条件：
ltime = rtime
@@ -1122,7 +1166,7 @@ SQL 查询的分组窗口是通过 `GROUP BY` 子句定义的。类似于使用
SESSION_END(time_attr, interval)
返回相对应的滚动、滑动和会话窗口范围以外的上界时间戳。
-注意： 范围以外的上界时间戳不可以 在随后基于时间的操作中，作为 行时间属性 使用，比如 基于时间窗口的 join 以及 分组窗口或分组窗口上的聚合。
注意： 范围以外的上界时间戳不可以 在随后基于时间的操作中，作为 行时间属性 使用，比如 interval join 以及 分组窗口或分组窗口上的聚合。
@@ -1131,7 +1175,7 @@ SQL 查询的分组窗口是通过 `GROUP BY` 子句定义的。类似于使用 SESSION_ROWTIME(time_attr, interval)
返回相对应的滚动、滑动和会话窗口范围以内的上界时间戳。
-返回的是一个可用于后续需要基于时间的操作的时间属性（rowtime attribute），比如基于时间窗口的 join 以及 分组窗口或分组窗口上的聚合。
返回的是一个可用于后续需要基于时间的操作的时间属性（rowtime attribute），比如interval join 以及 分组窗口或分组窗口上的聚合。
@@ -1139,7 +1183,7 @@ SQL 查询的分组窗口是通过 `GROUP BY` 子句定义的。类似于使用 -HOP_PROCTIME(time_attr, interval, interval)
SESSION_PROCTIME(time_attr, interval)
返回一个可用于后续需要基于时间的操作的 处理时间参数，比如基于时间窗口的 join 以及 分组窗口或分组窗口上的聚合.
返回一个可用于后续需要基于时间的操作的 处理时间参数，比如interval join 以及 分组窗口或分组窗口上的聚合.
++ +{% top %} + +## SHOW CATALOGS + +{% highlight sql %} +SHOW CATALOGS +{% endhighlight %} + +Show all catalogs. + +## SHOW DATABASES + +{% highlight sql %} +SHOW DATABASES +{% endhighlight %} + +Show all databases in the current catalog. + +## SHOW TABLES + +{% highlight sql %} +SHOW TABLES +{% endhighlight %} + +Show all tables in the current catalog and the current database. + +## SHOW VIEWS + +{% highlight sql %} +SHOW VIEWS +{% endhighlight %} + +Show all views in the current catalog and the current database. + +## SHOW FUNCTIONS + +{% highlight sql %} +SHOW FUNCTIONS +{% endhighlight %} + +Show all functions including temp system functions, system functions, temp catalog functions and catalog functions in the current catalog and current database. \ No newline at end of file diff --git a/docs/dev/table/sql/show.zh.md b/docs/dev/table/sql/show.zh.md new file mode 100644 index 00000000000000..9f6c648330fd5e --- /dev/null +++ b/docs/dev/table/sql/show.zh.md @@ -0,0 +1,277 @@ +--- +title: "SHOW 语句" +nav-parent_id: sql +nav-pos: 10 +--- + + +* This will be replaced by the TOC +{:toc} + +SHOW 语句用于列出所有的 catalog，或者列出当前 catalog 中所有的 database，或者列出当前 catalog 和当前 database 的所有表或视图，或者列出所有的 function，包括：临时系统 function，系统 function，临时 catalog function，当前 catalog 和 database 中的 catalog function。 + +目前 Flink SQL 支持下列 SHOW 语句： +- SHOW CATALOGS +- SHOW DATABASES +- SHOW TABLES +- SHOW VIEWS +- SHOW FUNCTIONS + + +## 执行 SHOW 语句 + +可以使用 `TableEnvironment` 中的 `executeSql()` 方法执行 SHOW 语句，也可以在 [SQL CLI]({{ site.baseurl }}/zh/dev/table/sqlClient.html) 中执行 SHOW 语句。 若 SHOW 操作执行成功，`executeSql()` 方法返回所有对象，否则会抛出异常。 + +以下的例子展示了如何在 `TableEnvironment` 和 SQL CLI 中执行一个 SHOW 语句。 + ++{% highlight java %} +StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); +StreamTableEnvironment tEnv = StreamTableEnvironment.create(env); + +// show catalogs +tEnv.executeSql("SHOW CATALOGS").print(); +// +-----------------+ +// | catalog name | +// +-----------------+ +// | default_catalog | +// +-----------------+ + +// show databases +tEnv.executeSql("SHOW DATABASES").print(); +// +------------------+ +// | database name | +// +------------------+ +// | default_database | +// +------------------+ + +// create a table +tEnv.executeSql("CREATE TABLE my_table (...) WITH (...)"); +// show tables +tEnv.executeSql("SHOW TABLES").print(); +// +------------+ +// | table name | +// +------------+ +// | my_table | +// +------------+ + +// create a view +tEnv.executeSql("CREATE VIEW my_view AS ..."); +// show views +tEnv.executeSql("SHOW VIEWS").print(); +// +-----------+ +// | view name | +// +-----------+ +// | my_view | +// +-----------+ + +// show functions +tEnv.executeSql("SHOW FUNCTIONS").print(); +// +---------------+ +// | function name | +// +---------------+ +// | mod | +// | sha256 | +// | ... | +// +---------------+ + +{% endhighlight %} ++ ++{% highlight scala %} +val env = StreamExecutionEnvironment.getExecutionEnvironment() +val tEnv = StreamTableEnvironment.create(env) + +// show catalogs +tEnv.executeSql("SHOW CATALOGS").print() +// +-----------------+ +// | catalog name | +// +-----------------+ +// | default_catalog | +// +-----------------+ + +// show databases +tEnv.executeSql("SHOW DATABASES").print() +// +------------------+ +// | database name | +// +------------------+ +// | default_database | +// +------------------+ + +// create a table +tEnv.executeSql("CREATE TABLE my_table (...) WITH (...)") +// show tables +tEnv.executeSql("SHOW TABLES").print() +// +------------+ +// | table name | +// +------------+ +// | my_table | +// +------------+ + +// create a view +tEnv.executeSql("CREATE VIEW my_view AS ...") +// show views +tEnv.executeSql("SHOW VIEWS").print() +// +-----------+ +// | view name | +// +-----------+ +// | my_view | +// +-----------+ + +// show functions +tEnv.executeSql("SHOW FUNCTIONS").print() +// +---------------+ +// | function name | +// +---------------+ +// | mod | +// | sha256 | +// | ... | +// +---------------+ + +{% endhighlight %} ++ ++{% highlight python %} +settings = EnvironmentSettings.new_instance()... +table_env = StreamTableEnvironment.create(env, settings) + +# show catalogs +table_env.execute_sql("SHOW CATALOGS").print() +# +-----------------+ +# | catalog name | +# +-----------------+ +# | default_catalog | +# +-----------------+ + +# show databases +table_env.execute_sql("SHOW DATABASES").print() +# +------------------+ +# | database name | +# +------------------+ +# | default_database | +# +------------------+ + +# create a table +table_env.execute_sql("CREATE TABLE my_table (...) WITH (...)") +# show tables +table_env.execute_sql("SHOW TABLES").print() +# +------------+ +# | table name | +# +------------+ +# | my_table | +# +------------+ + +# create a view +table_env.execute_sql("CREATE VIEW my_view AS ...") +# show views +table_env.execute_sql("SHOW VIEWS").print() +# +-----------+ +# | view name | +# +-----------+ +# | my_view | +# +-----------+ + +# show functions +table_env.execute_sql("SHOW FUNCTIONS").print() +# +---------------+ +# | function name | +# +---------------+ +# | mod | +# | sha256 | +# | ... | +# +---------------+ + +{% endhighlight %} ++ ++{% highlight sql %} + +Flink SQL> SHOW CATALOGS; +default_catalog + +Flink SQL> SHOW DATABASES; +default_database + +Flink SQL> CREATE TABLE my_table (...) WITH (...); +[INFO] Table has been created. + +Flink SQL> SHOW TABLES; +my_table + +Flink SQL> CREATE VIEW my_view AS ...; +[INFO] View has been created. + +Flink SQL> SHOW VIEWS; +my_view + +Flink SQL> SHOW FUNCTIONS; +mod +sha256 +... + +{% endhighlight %} ++++ +{% top %} + +## SHOW CATALOGS + +{% highlight sql %} +SHOW CATALOGS +{% endhighlight %} + +展示所有的 catalog。 + +## SHOW DATABASES + +{% highlight sql %} +SHOW DATABASES +{% endhighlight %} + +展示当前 catalog 中所有的 database。 + +## SHOW TABLES + +{% highlight sql %} +SHOW TABLES +{% endhighlight %} + +展示当前 catalog 和当前 database 中所有的表。 + +## SHOW VIEWS + +{% highlight sql %} +SHOW VIEWS +{% endhighlight %} + +展示当前 catalog 和当前 database 中所有的视图。 + +## SHOW FUNCTIONS + +{% highlight sql %} +SHOW FUNCTIONS +{% endhighlight %} + +展示所有的 function，包括：临时系统 function, 系统 function, 临时 catalog function，当前 catalog 和 database 中的 catalog function。 \ No newline at end of file diff --git a/docs/dev/table/sql/use.md b/docs/dev/table/sql/use.md new file mode 100644 index 00000000000000..983d2d4b310fc7 --- /dev/null +++ b/docs/dev/table/sql/use.md @@ -0,0 +1,200 @@ +--- +title: "USE Statements" +nav-parent_id: sql +nav-pos: 9 +--- + + +* This will be replaced by the TOC +{:toc} + +USE statements are used to set the current database or catalog. + + +## Run a USE statement + +USE statements can be executed with the `executeSql()` method of the `TableEnvironment`, or executed in [SQL CLI]({{ site.baseurl }}/dev/table/sqlClient.html). The `executeSql()` method returns 'OK' for a successful USE operation, otherwise will throw an exception. + +The following examples show how to run a USE statement in `TableEnvironment` and in SQL CLI. + ++{% highlight java %} +StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); +StreamTableEnvironment tEnv = StreamTableEnvironment.create(env); + +// show catalogs +tEnv.executeSql("SHOW CATALOGS").print(); +// +-----------------+ +// | catalog name | +// +-----------------+ +// | default_catalog | +// +-----------------+ + +// show databases +tEnv.executeSql("SHOW DATABASES").print(); +// +------------------+ +// | database name | +// +------------------+ +// | default_database | +// +------------------+ + +// create a table +tEnv.executeSql("CREATE TABLE my_table (...) WITH (...)"); +// show tables +tEnv.executeSql("SHOW TABLES").print(); +// +------------+ +// | table name | +// +------------+ +// | my_table | +// +------------+ + +// create a view +tEnv.executeSql("CREATE VIEW my_view AS ..."); +// show views +tEnv.executeSql("SHOW VIEWS").print(); +// +-----------+ +// | view name | +// +-----------+ +// | my_view | +// +-----------+ + +// show functions +tEnv.executeSql("SHOW FUNCTIONS").print(); +// +---------------+ +// | function name | +// +---------------+ +// | mod | +// | sha256 | +// | ... | +// +---------------+ + +{% endhighlight %} ++ ++{% highlight scala %} +val env = StreamExecutionEnvironment.getExecutionEnvironment() +val tEnv = StreamTableEnvironment.create(env) + +// show catalogs +tEnv.executeSql("SHOW CATALOGS").print() +// +-----------------+ +// | catalog name | +// +-----------------+ +// | default_catalog | +// +-----------------+ + +// show databases +tEnv.executeSql("SHOW DATABASES").print() +// +------------------+ +// | database name | +// +------------------+ +// | default_database | +// +------------------+ + +// create a table +tEnv.executeSql("CREATE TABLE my_table (...) WITH (...)") +// show tables +tEnv.executeSql("SHOW TABLES").print() +// +------------+ +// | table name | +// +------------+ +// | my_table | +// +------------+ + +// create a view +tEnv.executeSql("CREATE VIEW my_view AS ...") +// show views +tEnv.executeSql("SHOW VIEWS").print() +// +-----------+ +// | view name | +// +-----------+ +// | my_view | +// +-----------+ + +// show functions +tEnv.executeSql("SHOW FUNCTIONS").print() +// +---------------+ +// | function name | +// +---------------+ +// | mod | +// | sha256 | +// | ... | +// +---------------+ + +{% endhighlight %} ++ ++{% highlight python %} +settings = EnvironmentSettings.new_instance()... +table_env = StreamTableEnvironment.create(env, settings) + +# show catalogs +table_env.execute_sql("SHOW CATALOGS").print() +# +-----------------+ +# | catalog name | +# +-----------------+ +# | default_catalog | +# +-----------------+ + +# show databases +table_env.execute_sql("SHOW DATABASES").print() +# +------------------+ +# | database name | +# +------------------+ +# | default_database | +# +------------------+ + +# create a table +table_env.execute_sql("CREATE TABLE my_table (...) WITH (...)") +# show tables +table_env.execute_sql("SHOW TABLES").print() +# +------------+ +# | table name | +# +------------+ +# | my_table | +# +------------+ + +# create a view +table_env.execute_sql("CREATE VIEW my_view AS ...") +# show views +table_env.execute_sql("SHOW VIEWS").print() +# +-----------+ +# | view name | +# +-----------+ +# | my_view | +# +-----------+ + +# show functions +table_env.execute_sql("SHOW FUNCTIONS").print() +# +---------------+ +# | function name | +# +---------------+ +# | mod | +# | sha256 | +# | ... | +# +---------------+ + +{% endhighlight %} ++ ++{% highlight sql %} + +Flink SQL> SHOW CATALOGS; +default_catalog + +Flink SQL> SHOW DATABASES; +default_database + +Flink SQL> CREATE TABLE my_table (...) WITH (...); +[INFO] Table has been created. + +Flink SQL> SHOW TABLES; +my_table + +Flink SQL> CREATE VIEW my_view AS ...; +[INFO] View has been created. + +Flink SQL> SHOW VIEWS; +my_view + +Flink SQL> SHOW FUNCTIONS; +mod +sha256 +... + +{% endhighlight %} ++++ +{% top %} + +## USE CATLOAG + +{% highlight sql %} +USE CATALOG catalog_name +{% endhighlight %} + +Set the current catalog. All subsequent commands that do not explicitly specify a catalog will use this one. If the provided catalog does not exist, an exception is thrown. The default current catalog is `default_catalog`. + + +## USE + +{% highlight sql %} +USE [catalog_name.]database_name +{% endhighlight %} + +Set the current database. All subsequent commands that do not explicitly specify a database will use this one. If the provided database does not exist, an exception is thrown. The default current database is `default_database`. diff --git a/docs/dev/table/sql/use.zh.md b/docs/dev/table/sql/use.zh.md new file mode 100644 index 00000000000000..128faf521c9201 --- /dev/null +++ b/docs/dev/table/sql/use.zh.md @@ -0,0 +1,199 @@ +--- +title: "USE 语句" +nav-parent_id: sql +nav-pos: 9 +--- + + +* This will be replaced by the TOC +{:toc} + +USE 语句用来设置当前的 catalog 或者 database。 + +## 运行一个 USE 语句 + +可以使用 TableEnvironment 中的 executeSql() 方法执行 USE 语句，也可以在 SQL CLI 中执行 USE 语句。 若 USE 操作执行成功，executeSql() 方法返回 'OK'，否则会抛出异常。 + +以下的例子展示了如何在 TableEnvironment 和 SQL CLI 中执行一个 USE 语句。 + ++{% highlight java %} +StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); +StreamTableEnvironment tEnv = StreamTableEnvironment.create(env); + +// create a catalog +tEnv.executeSql("CREATE CATALOG cat1 WITH (...)"); +tEnv.executeSql("SHOW CATALOGS").print(); +// +-----------------+ +// | catalog name | +// +-----------------+ +// | default_catalog | +// | cat1 | +// +-----------------+ + +// change default catalog +tEnv.executeSql("USE CATALOG cat1"); + +tEnv.executeSql("SHOW DATABASES").print(); +// databases are empty +// +---------------+ +// | database name | +// +---------------+ +// +---------------+ + +// create a database +tEnv.executeSql("CREATE DATABASE db1 WITH (...)"); +tEnv.executeSql("SHOW DATABASES").print(); +// +---------------+ +// | database name | +// +---------------+ +// | db1 | +// +---------------+ + +// change default database +tEnv.executeSql("USE db1"); + +{% endhighlight %} ++ ++{% highlight scala %} +val env = StreamExecutionEnvironment.getExecutionEnvironment() +val tEnv = StreamTableEnvironment.create(env) + +// create a catalog +tEnv.executeSql("CREATE CATALOG cat1 WITH (...)") +tEnv.executeSql("SHOW CATALOGS").print() +// +-----------------+ +// | catalog name | +// +-----------------+ +// | default_catalog | +// | cat1 | +// +-----------------+ + +// change default catalog +tEnv.executeSql("USE CATALOG cat1") + +tEnv.executeSql("SHOW DATABASES").print() +// databases are empty +// +---------------+ +// | database name | +// +---------------+ +// +---------------+ + +// create a database +tEnv.executeSql("CREATE DATABASE db1 WITH (...)") +tEnv.executeSql("SHOW DATABASES").print() +// +---------------+ +// | database name | +// +---------------+ +// | db1 | +// +---------------+ + +// change default database +tEnv.executeSql("USE db1") + +{% endhighlight %} ++ ++{% highlight python %} +settings = EnvironmentSettings.new_instance()... +table_env = StreamTableEnvironment.create(env, settings) + +# create a catalog +table_env.execute_sql("CREATE CATALOG cat1 WITH (...)") +table_env.execute_sql("SHOW CATALOGS").print() +# +-----------------+ +# | catalog name | +# +-----------------+ +# | default_catalog | +# | cat1 | +# +-----------------+ + +# change default catalog +table_env.execute_sql("USE CATALOG cat1") + +table_env.execute_sql("SHOW DATABASES").print() +# databases are empty +# +---------------+ +# | database name | +# +---------------+ +# +---------------+ + +# create a database +table_env.execute_sql("CREATE DATABASE db1 WITH (...)") +table_env.execute_sql("SHOW DATABASES").print() +# +---------------+ +# | database name | +# +---------------+ +# | db1 | +# +---------------+ + +# change default database +table_env.execute_sql("USE db1") + +{% endhighlight %} ++ ++{% highlight sql %} +Flink SQL> CREATE CATALOG cat1 WITH (...); +[INFO] Catalog has been created. + +Flink SQL> SHOW CATALOGS; +default_catalog +cat1 + +Flink SQL> USE CATALOG cat1; + +Flink SQL> SHOW DATABASES; + +Flink SQL> CREATE DATABASE db1 WITH (...); +[INFO] Database has been created. + +Flink SQL> SHOW DATABASES; +db1 + +Flink SQL> USE db1; + +{% endhighlight %} ++++ +{% top %} + +## USE CATLOAG + +{% highlight sql %} +USE CATALOG catalog_name +{% endhighlight %} + +设置当前的 catalog。所有后续命令未显式指定 catalog 的将使用此 catalog。如果指定的的 catalog 不存在，则抛出异常。默认的当前 catalog 是 `default_catalog`。 + + +## USE + +{% highlight sql %} +USE [catalog_name.]database_name +{% endhighlight %} + +设置当前的 database。所有后续命令未显式指定 database 的将使用此 database。如果指定的的 database 不存在，则抛出异常。默认的当前 database 是 `default_database`。 diff --git a/docs/dev/table/sqlClient.md b/docs/dev/table/sqlClient.md index 8331efef472bc2..ad1030855b576a 100644 --- a/docs/dev/table/sqlClient.md +++ b/docs/dev/table/sqlClient.md @@ -66,7 +66,7 @@ SELECT 'Hello World'; This query requires no table source and produces a single row result. The CLI will retrieve results from the cluster and visualize them. You can close the result view by pressing the `Q` key. -The CLI supports **two modes** for maintaining and visualizing results. +The CLI supports **three modes** for maintaining and visualizing results. The **table mode** materializes results in memory and visualizes them in a regular, paginated table representation. It can be enabled by executing the following command in the CLI: @@ -80,7 +80,18 @@ The **changelog mode** does not materialize results and visualizes the result st SET execution.result-mode=changelog; {% endhighlight %} -You can use the following query to see both result modes in action: +The **tableau mode** is more like a traditional way which will display the results in the screen directly with a tableau format. +The displaying content will be influenced by the query execution type(`execution.type`). + +{% highlight text %} +SET execution.result-mode=tableau; +{% endhighlight %} + +Note that when you use this mode with streaming query, the result will be continuously printed on the console. If the input data of +this query is bounded, the job will terminate after Flink processed all input data, and the printing will also be stopped automatically. +Otherwise, if you want to terminate a running query, just type `CTRL-C` in this case, the job and the printing will be stopped. + +You can use the following query to see all the result modes in action: {% highlight sql %} SELECT name, COUNT(*) AS cnt FROM (VALUES ('Bob'), ('Alice'), ('Greg'), ('Bob')) AS NameTable(name) GROUP BY name; @@ -106,9 +117,35 @@ Alice, 1 Greg, 1 {% endhighlight %} -Both result modes can be useful during the prototyping of SQL queries. In both modes, results are stored in the Java heap memory of the SQL Client. In order to keep the CLI interface responsive, the changelog mode only shows the latest 1000 changes. The table mode allows for navigating through bigger results that are only limited by the available main memory and the configured [maximum number of rows](sqlClient.html#configuration) (`max-table-result-rows`). +In *tableau mode*, if you ran the query in streaming mode, the displayed result would be: +{% highlight text %} ++-----+----------------------+----------------------+ +| +/- | name | cnt | ++-----+----------------------+----------------------+ +| + | Bob | 1 | +| + | Alice | 1 | +| + | Greg | 1 | +| - | Bob | 1 | +| + | Bob | 2 | ++-----+----------------------+----------------------+ +Received a total of 5 rows +{% endhighlight %} + +And if you ran the query in batch mode, the displayed result would be: +{% highlight text %} ++-------+-----+ +| name | cnt | ++-------+-----+ +| Alice | 1 | +| Bob | 2 | +| Greg | 1 | ++-------+-----+ +3 rows in set +{% endhighlight %} + +All these result modes can be useful during the prototyping of SQL queries. In all these modes, results are stored in the Java heap memory of the SQL Client. In order to keep the CLI interface responsive, the changelog mode only shows the latest 1000 changes. The table mode allows for navigating through bigger results that are only limited by the available main memory and the configured [maximum number of rows](sqlClient.html#configuration) (`max-table-result-rows`). -Attention Queries that are executed in a batch environment, can only be retrieved using the `table` result mode. +Attention Queries that are executed in a batch environment, can only be retrieved using the `table` or `tableau` result mode. After a query is defined, it can be submitted to the cluster as a long-running, detached Flink job. For this, a target system that stores the results needs to be specified using the [INSERT INTO statement](sqlClient.html#detached-sql-queries). The [configuration section](sqlClient.html#configuration) explains how to declare table sources for reading data, how to declare table sinks for writing data, and how to configure other table program properties. @@ -254,16 +291,16 @@ tables: type: csv fields: - name: MyField1 - type: INT + data-type: INT - name: MyField2 - type: VARCHAR + data-type: VARCHAR line-delimiter: "\n" comment-prefix: "#" schema: - name: MyField1 - type: INT + data-type: INT - name: MyField2 - type: VARCHAR + data-type: VARCHAR - name: MyCustomView type: view query: "SELECT MyField2 FROM MyTableSource" diff --git a/docs/dev/table/sqlClient.zh.md b/docs/dev/table/sqlClient.zh.md index c234390df014f3..7210f405a3eeec 100644 --- a/docs/dev/table/sqlClient.zh.md +++ b/docs/dev/table/sqlClient.zh.md @@ -65,7 +65,7 @@ SELECT 'Hello World'; 该查询不需要 table source，并且只产生一行结果。CLI 将从集群中检索结果并将其可视化。按 `Q` 键退出结果视图。 -CLI 为维护和可视化结果提供**两种模式**。 +CLI 为维护和可视化结果提供**三种模式**。 **表格模式**（table mode）在内存中实体化结果，并将结果用规则的分页表格可视化展示出来。执行如下命令启用： @@ -79,7 +79,18 @@ SET execution.result-mode=table; SET execution.result-mode=changelog; {% endhighlight %} -你可以用如下查询来查看两种结果模式的运行情况： +**Tableau模式**（tableau mode）更接近传统的数据库，会将执行的结果以制表的形式直接打在屏幕之上。具体显示的内容会取决于作业 +执行模式的不同(`execution.type`)： + +{% highlight text %} +SET execution.result-mode=tableau; +{% endhighlight %} + +注意当你使用这个模式运行一个流式查询的时候，Flink 会将结果持续的打印在当前的屏幕之上。如果这个流式查询的输入是有限的数据集， +那么Flink在处理完所有的数据之后，会自动的停止作业，同时屏幕上的打印也会相应的停止。如果你想提前结束这个查询，那么可以直接使用 +`CTRL-C` 按键，这个会停掉作业同时停止屏幕上的打印。 + +你可以用如下查询来查看三种结果模式的运行情况： {% highlight sql %} SELECT name, COUNT(*) AS cnt FROM (VALUES ('Bob'), ('Alice'), ('Greg'), ('Bob')) AS NameTable(name) GROUP BY name; @@ -105,9 +116,35 @@ Alice, 1 Greg, 1 {% endhighlight %} -这两种结果模式在 SQL 查询的原型设计过程中都非常有用。这两种模式结果都存储在 SQL 客户端 的 Java 堆内存中。为了保持 CLI 界面及时响应，变更日志模式仅显示最近的 1000 个更改。表格模式支持浏览更大的结果，这些结果仅受可用主内存和配置的[最大行数](sqlClient.html#configuration)（`max-table-result-rows`）的限制。 +*Tableau模式* 下，如果这个查询以流的方式执行，那么将显示以下内容： +{% highlight text %} ++-----+----------------------+----------------------+ +| +/- | name | cnt | ++-----+----------------------+----------------------+ +| + | Bob | 1 | +| + | Alice | 1 | +| + | Greg | 1 | +| - | Bob | 1 | +| + | Bob | 2 | ++-----+----------------------+----------------------+ +Received a total of 5 rows +{% endhighlight %} + +如果这个查询以批的方式执行，显示的内容如下： +{% highlight text %} ++-------+-----+ +| name | cnt | ++-------+-----+ +| Alice | 1 | +| Bob | 2 | +| Greg | 1 | ++-------+-----+ +3 rows in set +{% endhighlight %} + +这几种结果模式在 SQL 查询的原型设计过程中都非常有用。这些模式的结果都存储在 SQL 客户端 的 Java 堆内存中。为了保持 CLI 界面及时响应，变更日志模式仅显示最近的 1000 个更改。表格模式支持浏览更大的结果，这些结果仅受可用主内存和配置的[最大行数](sqlClient.html#configuration)（`max-table-result-rows`）的限制。 -注意 在批处理环境下执行的查询只能用表格模式进行检索。 +注意 在批处理环境下执行的查询只能用表格模式或者Tableau模式进行检索。 定义查询语句后，可以将其作为长时间运行的独立 Flink 作业提交给集群。为此，其目标系统需要使用 [INSERT INTO 语句](sqlClient.html#detached-sql-queries)指定存储结果。[配置部分](sqlClient.html#configuration)解释如何声明读取数据的 table source，写入数据的 sink 以及配置其他表程序属性的方法。 @@ -257,16 +294,16 @@ tables: type: csv fields: - name: MyField1 - type: INT + data-type: INT - name: MyField2 - type: VARCHAR + data-type: VARCHAR line-delimiter: "\n" comment-prefix: "#" schema: - name: MyField1 - type: INT + data-type: INT - name: MyField2 - type: VARCHAR + data-type: VARCHAR - name: MyCustomView type: view query: "SELECT MyField2 FROM MyTableSource" @@ -347,7 +384,7 @@ CLI commands > session environment file > defaults environment file #### 重启策略（Restart Strategies） -重启策略控制 Flink 作业失败时的重启方式。与 Flink 集群的[全局重启策略]({{ site.baseurl }}/zh/dev/restart_strategies.html)相似，更细精度的重启配置可以在环境配置文件中声明。 +重启策略控制 Flink 作业失败时的重启方式。与 Flink 集群的[全局重启策略]({{ site.baseurl }}/zh/dev/task_failure_recovery.html)相似，更细精度的重启配置可以在环境配置文件中声明。 Flink 支持以下策略： @@ -600,7 +637,7 @@ Job ID: 6f922fe5cba87406ff23ae4a7bb79044 Web interface: http://localhost:8081 {% endhighlight %} -注意 提交后，SQL 客户端不追踪正在运行的 Flink 作业状态。提交后可以关闭 CLI 进程，并且不会影响分离的查询。Flink 的[重启策略]({{ site.baseurl }}/zh/dev/restart_strategies.html)负责容错。取消查询可以用 Flink 的 web 接口、命令行或 REST API 。 +注意 提交后，SQL 客户端不追踪正在运行的 Flink 作业状态。提交后可以关闭 CLI 进程，并且不会影响分离的查询。Flink 的[重启策略]({{ site.baseurl }}/zh/dev/task_failure_recovery.html)负责容错。取消查询可以用 Flink 的 web 接口、命令行或 REST API 。 {% top %} diff --git a/docs/dev/table/streaming/joins.md b/docs/dev/table/streaming/joins.md index e621a2057dec05..98c96e6d5b3e46 100644 --- a/docs/dev/table/streaming/joins.md +++ b/docs/dev/table/streaming/joins.md @@ -48,10 +48,10 @@ These semantics allow for any kind of updating (insert, update, delete) input ta However, this operation has an important implication: it requires to keep both sides of the join input in Flink's state forever. Thus, the resource usage will grow indefinitely as well, if one or both input tables are continuously growing. -Time-windowed Joins +Interval Joins ------------------- -A time-windowed join is defined by a join predicate, that checks if the [time attributes](time_attributes.html) of the input +A interval join is defined by a join predicate, that checks if the [time attributes](time_attributes.html) of the input records are within certain time constraints, i.e., a time window. {% highlight sql %} @@ -148,7 +148,7 @@ In our example, each record from `Orders` will be joined with the version of `Ra In contrast to [regular joins](#regular-joins), this means that if there is a new record on the build side, it will not affect the previous results of the join. This again allows Flink to limit the number of elements that must be kept in the state. -Compared to [time-windowed joins](#time-windowed-joins), temporal table joins do not define a time window within which bounds the records will be joined. +Compared to [interval joins](#interval-joins), temporal table joins do not define a time window within which bounds the records will be joined. Records from the probe side are always joined with the build side's version at the time specified by the time attribute. Thus, records on the build side might be arbitrarily old. As time passes, the previous and no longer needed versions of the record (for the given primary key) will be removed from the state. @@ -304,7 +304,7 @@ Each record from the probe side will be joined with the current version of the b In contrast to [regular joins](#regular-joins), the previous results of the temporal table join will not be affected despite the changes on the build side. Also, the temporal table join operator is very lightweight and does not keep any state. -Compared to [time-windowed joins](#time-windowed-joins), temporal table joins do not define a time window within which the records will be joined. +Compared to [interval joins](#interval-joins), temporal table joins do not define a time window within which the records will be joined. Records from the probe side are always joined with the build side's latest version at processing time. Thus, records on the build side might be arbitrarily old. Both [temporal table function join](#join-with-a-temporal-table-function) and temporal table join come from the same motivation but have different SQL syntax and runtime implementations: diff --git a/docs/dev/table/streaming/joins.zh.md b/docs/dev/table/streaming/joins.zh.md index e621a2057dec05..98c96e6d5b3e46 100644 --- a/docs/dev/table/streaming/joins.zh.md +++ b/docs/dev/table/streaming/joins.zh.md @@ -48,10 +48,10 @@ These semantics allow for any kind of updating (insert, update, delete) input ta However, this operation has an important implication: it requires to keep both sides of the join input in Flink's state forever. Thus, the resource usage will grow indefinitely as well, if one or both input tables are continuously growing. -Time-windowed Joins +Interval Joins ------------------- -A time-windowed join is defined by a join predicate, that checks if the [time attributes](time_attributes.html) of the input +A interval join is defined by a join predicate, that checks if the [time attributes](time_attributes.html) of the input records are within certain time constraints, i.e., a time window. {% highlight sql %} @@ -148,7 +148,7 @@ In our example, each record from `Orders` will be joined with the version of `Ra In contrast to [regular joins](#regular-joins), this means that if there is a new record on the build side, it will not affect the previous results of the join. This again allows Flink to limit the number of elements that must be kept in the state. -Compared to [time-windowed joins](#time-windowed-joins), temporal table joins do not define a time window within which bounds the records will be joined. +Compared to [interval joins](#interval-joins), temporal table joins do not define a time window within which bounds the records will be joined. Records from the probe side are always joined with the build side's version at the time specified by the time attribute. Thus, records on the build side might be arbitrarily old. As time passes, the previous and no longer needed versions of the record (for the given primary key) will be removed from the state. @@ -304,7 +304,7 @@ Each record from the probe side will be joined with the current version of the b In contrast to [regular joins](#regular-joins), the previous results of the temporal table join will not be affected despite the changes on the build side. Also, the temporal table join operator is very lightweight and does not keep any state. -Compared to [time-windowed joins](#time-windowed-joins), temporal table joins do not define a time window within which the records will be joined. +Compared to [interval joins](#interval-joins), temporal table joins do not define a time window within which the records will be joined. Records from the probe side are always joined with the build side's latest version at processing time. Thus, records on the build side might be arbitrarily old. Both [temporal table function join](#join-with-a-temporal-table-function) and temporal table join come from the same motivation but have different SQL syntax and runtime implementations: diff --git a/docs/dev/table/streaming/match_recognize.md b/docs/dev/table/streaming/match_recognize.md index 8fb12fa5494c90..3fd20ecc777805 100644 --- a/docs/dev/table/streaming/match_recognize.md +++ b/docs/dev/table/streaming/match_recognize.md @@ -3,7 +3,6 @@ title: 'Detecting Patterns in Tables' nav-parent_id: streaming_tableapi nav-title: 'Detecting Patterns' nav-pos: 5 -is_beta: true --- It is a common use case to search for a set of event patterns, especially in case of data streams. -Flink comes with a [complex event processing (CEP) library]({{ site.baseurl }}/dev/libs/cep.html) +Flink comes with a [complex event processing (CEP) library]({% link dev/libs/cep.md %}) which allows for pattern detection in event streams. Furthermore, Flink's SQL API provides a relational way of expressing queries with a large set of built-in functions and rule-based optimizations that can be used out of the box. @@ -67,9 +66,9 @@ FROM MyTable This page will explain each keyword in more detail and will illustrate more complex examples. -Attention Flink's implementation of the `MATCH_RECOGNIZE` +{% info Notice %} Flink's implementation of the `MATCH_RECOGNIZE` clause is a subset of the full standard. Only those features documented in the following sections -are supported. Since the development is still in an early phase, please also take a look at the +are supported. Additional features may be supported based on community feedback, please also take a look at the [known limitations](#known-limitations). * This will be replaced by the TOC @@ -93,10 +92,10 @@ project. {% endhighlight %} Alternatively, you can also add the dependency to the cluster classpath (see the -[dependency section]({{ site.baseurl}}/dev/projectsetup/dependencies.html) for more information). +[dependency section]({% link dev/project-configuration.md %}) for more information). If you want to use the `MATCH_RECOGNIZE` clause in the -[SQL Client]({{ site.baseurl}}/dev/table/sqlClient.html), you don't have to do anything as all the +[SQL Client]({% link dev/table/sqlClient.md %}), you don't have to do anything as all the dependencies are included by default. ### SQL Semantics @@ -263,8 +262,8 @@ look at the [event stream navigation](#pattern-navigation) section. ### Aggregations Aggregations can be used in `DEFINE` and `MEASURES` clauses. Both -[built-in]({{ site.baseurl }}/dev/table/functions/systemFunctions.html) and custom -[user defined]({{ site.baseurl }}/dev/table/functions/udfs.html) functions are supported. +[built-in]({% link dev/table/functions/systemFunctions.md %}) and custom +[user defined]({% link dev/table/functions/udfs.md %}) functions are supported. Aggregate functions are applied to each subset of rows mapped to a match. In order to understand how those subsets are evaluated have a look at the [event stream navigation](#pattern-navigation) @@ -1038,7 +1037,7 @@ use [time attributes](time_attributes.html). To select those there are available+{% highlight java %} +StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); +StreamTableEnvironment tEnv = StreamTableEnvironment.create(env); + +// create a catalog +tEnv.executeSql("CREATE CATALOG cat1 WITH (...)"); +tEnv.executeSql("SHOW CATALOGS").print(); +// +-----------------+ +// | catalog name | +// +-----------------+ +// | default_catalog | +// | cat1 | +// +-----------------+ + +// change default catalog +tEnv.executeSql("USE CATALOG cat1"); + +tEnv.executeSql("SHOW DATABASES").print(); +// databases are empty +// +---------------+ +// | database name | +// +---------------+ +// +---------------+ + +// create a database +tEnv.executeSql("CREATE DATABASE db1 WITH (...)"); +tEnv.executeSql("SHOW DATABASES").print(); +// +---------------+ +// | database name | +// +---------------+ +// | db1 | +// +---------------+ + +// change default database +tEnv.executeSql("USE db1"); + +{% endhighlight %} ++ ++{% highlight scala %} +val env = StreamExecutionEnvironment.getExecutionEnvironment() +val tEnv = StreamTableEnvironment.create(env) + +// create a catalog +tEnv.executeSql("CREATE CATALOG cat1 WITH (...)") +tEnv.executeSql("SHOW CATALOGS").print() +// +-----------------+ +// | catalog name | +// +-----------------+ +// | default_catalog | +// | cat1 | +// +-----------------+ + +// change default catalog +tEnv.executeSql("USE CATALOG cat1") + +tEnv.executeSql("SHOW DATABASES").print() +// databases are empty +// +---------------+ +// | database name | +// +---------------+ +// +---------------+ + +// create a database +tEnv.executeSql("CREATE DATABASE db1 WITH (...)") +tEnv.executeSql("SHOW DATABASES").print() +// +---------------+ +// | database name | +// +---------------+ +// | db1 | +// +---------------+ + +// change default database +tEnv.executeSql("USE db1") + +{% endhighlight %} ++ ++{% highlight python %} +settings = EnvironmentSettings.new_instance()... +table_env = StreamTableEnvironment.create(env, settings) + +# create a catalog +table_env.execute_sql("CREATE CATALOG cat1 WITH (...)") +table_env.execute_sql("SHOW CATALOGS").print() +# +-----------------+ +# | catalog name | +# +-----------------+ +# | default_catalog | +# | cat1 | +# +-----------------+ + +# change default catalog +table_env.execute_sql("USE CATALOG cat1") + +table_env.execute_sql("SHOW DATABASES").print() +# databases are empty +# +---------------+ +# | database name | +# +---------------+ +# +---------------+ + +# create a database +table_env.execute_sql("CREATE DATABASE db1 WITH (...)") +table_env.execute_sql("SHOW DATABASES").print() +# +---------------+ +# | database name | +# +---------------+ +# | db1 | +# +---------------+ + +# change default database +table_env.execute_sql("USE db1") + +{% endhighlight %} ++ ++{% highlight sql %} +Flink SQL> CREATE CATALOG cat1 WITH (...); +[INFO] Catalog has been created. + +Flink SQL> SHOW CATALOGS; +default_catalog +cat1 + +Flink SQL> USE CATALOG cat1; + +Flink SQL> SHOW DATABASES; + +Flink SQL> CREATE DATABASE db1 WITH (...); +[INFO] Database has been created. + +Flink SQL> SHOW DATABASES; +db1 + +Flink SQL> USE db1; + +{% endhighlight %} ++Returns the timestamp of the last row that was mapped to the given pattern.
The resulting attribute is a rowtime attribute that can be used in subsequent time-based operations such as - time-windowed joins and group window or over + interval joins and group window or over window aggregations.
@@ -1047,7 +1046,7 @@ use [time attributes](time_attributes.html). To select those there are available diff --git a/docs/dev/table/streaming/match_recognize.zh.md b/docs/dev/table/streaming/match_recognize.zh.md index 8fb12fa5494c90..54091d415dcd31 100644 --- a/docs/dev/table/streaming/match_recognize.zh.md +++ b/docs/dev/table/streaming/match_recognize.zh.md @@ -3,7 +3,6 @@ title: 'Detecting Patterns in Tables' nav-parent_id: streaming_tableapi nav-title: 'Detecting Patterns' nav-pos: 5 -is_beta: true --- -Temporal Tables represent a concept of a (parameterized) view on a changing table that returns the content of a table at a specific point in time. +时态表（Temporal Table）代表基于表的（参数化）视图概念，该表记录变更历史，该视图返回表在某个特定时间点的内容。 -The changing table can either be a changing history table which tracks the changes (e.g. database changelogs) or a changing dimension table which materializes the changes (e.g. database tables). +变更表可以是跟踪变化的历史记录表（例如数据库变更日志），也可以是有具体更改的维表（例如数据库表）。 -For the changing history table, Flink can keep track of the changes and allows for accessing the content of the table at a certain point in time within a query. In Flink, this kind of table is represented by a *Temporal Table Function*. +对于记录变更历史的表，Flink 可以追踪这些变化，并且允许查询这张表在某个特定时间点的内容。在 Flink 中，这类表由*时态表函数（Temporal Table Function）*表示。 -For the changing dimension table, Flink allows for accessing the content of the table at processing time within a query. In Flink, this kind of table is represented by a *Temporal Table*. +对于变化的维表，Flink 允许查询这张表在处理时的内容，在 Flink 中，此类表由*时态表（Temporal Table）*表示。 * This will be replaced by the TOC {:toc} -Motivation +设计初衷 ---------- -### Correlate with a changing history table +### 与记录变更历史的表相关 -Let's assume that we have the following table `RatesHistory`. +假设我们有表 `RatesHistory` 如下所示。 {% highlight sql %} SELECT * FROM RatesHistory; @@ -53,10 +53,9 @@ rowtime currency rate 11:49 Pounds 108 {% endhighlight %} -`RatesHistory` represents an ever growing append-only table of currency exchange rates with respect to `Yen` (which has a rate of `1`). -For example, the exchange rate for the period from `09:00` to `10:45` of `Euro` to `Yen` was `114`. From `10:45` to `11:15` it was `116`. +`RatesHistory` 代表一个兑换日元货币汇率表（日元汇率为1），该表是不断增长的 append-only 表。例如，`欧元`兑`日元`从 `09:00` 到 `10:45` 的汇率为 `114`。从 `10:45` 到 `11:15`，汇率为 `116`。 -Given that we would like to output all current rates at the time `10:58`, we would need the following SQL query to compute a result table: +假设我们要输出 `10:58` 的所有当前汇率，则需要以下 SQL 查询来计算结果表： {% highlight sql %} SELECT * @@ -68,9 +67,9 @@ WHERE r.rowtime = ( AND r2.rowtime <= TIME '10:58'); {% endhighlight %} -The correlated subquery determines the maximum time for the corresponding currency that is lower or equal than the desired time. The outer query lists the rates that have a maximum timestamp. - -The following table shows the result of such a computation. In our example, the update to `Euro` at `10:45` is taken into account, however, the update to `Euro` at `11:15` and the new entry of `Pounds` are not considered in the table's version at time `10:58`. +子查询确定对应货币的最大时间小于或等于所需时间。外部查询列出具有最大时间戳的汇率。 + +下表显示了这种计算的结果。我们的示例中，在 `10:58` 时表的内容，考虑了 `10:45` 时`欧元`的更新，但未考虑 `11:15` 时的`欧元`更新和`英镑`的新值。 {% highlight text %} rowtime currency rate @@ -80,17 +79,17 @@ rowtime currency rate 10:45 Euro 116 {% endhighlight %} -The concept of *Temporal Tables* aims to simplify such queries, speed up their execution, and reduce Flink's state usage. A *Temporal Table* is a parameterized view on an append-only table that interprets the rows of the append-only table as the changelog of a table and provides the version of that table at a specific point in time. Interpreting the append-only table as a changelog requires the specification of a primary key attribute and a timestamp attribute. The primary key determines which rows are overwritten and the timestamp determines the time during which a row is valid. + *时态表*的概念旨在简化此类查询，加快其执行速度，并减少 Flink 的状态使用。*时态表*是 append-only 表上的参数化视图，该视图将 append-only 表的行解释为表的变更日志，并在特定时间点提供该表的版本。将 append-only 表解释为变更日志需要指定主键属性和时间戳属性。主键确定哪些行将被覆盖，时间戳确定行有效的时间。 -In the above example `currency` would be a primary key for `RatesHistory` table and `rowtime` would be the timestamp attribute. +在上面的示例中，`currency` 是 `RatesHistory` 表的主键，而 `rowtime` 是时间戳属性。 -In Flink, this is represented by a [*Temporal Table Function*](#temporal-table-function). +在 Flink 中，这由[*时态表函数*](#temporal-table-function)表示。 -### Correlate with a changing dimension table +### 与维表变化相关 -On the other hand, some use cases require to join a changing dimension table which is an external database table. +另一方面，某些用例需要连接变化的维表，该表是外部数据库表。 -Let's assume that `LatestRates` is a table (e.g. stored in) which is materialized with the latest rate. The `LatestRates` is the materialized history `RatesHistory`. Then the content of `LatestRates` table at time `10:58` will be: +假设 `LatestRates` 是一个被物化的最新汇率表。`LatestRates` 是物化的 `RatesHistory` 历史。那么 `LatestRates` 表在 `10:58` 的内容将是： {% highlight text %} 10:58> SELECT * FROM LatestRates; @@ -101,7 +100,7 @@ Yen 1 Euro 116 {% endhighlight %} -The content of `LatestRates` table at time `12:00` will be: +`12:00` 时 `LatestRates` 表的内容将是： {% highlight text %} 12:00> SELECT * FROM LatestRates; @@ -113,18 +112,20 @@ Euro 119 Pounds 108 {% endhighlight %} -In Flink, this is represented by a [*Temporal Table*](#temporal-table). +在 Flink 中，这由[*时态表*](#temporal-table)表示。 + + -Temporal Table Function +时态表函数 ------------------------ -In order to access the data in a temporal table, one must pass a [time attribute](time_attributes.html) that determines the version of the table that will be returned. -Flink uses the SQL syntax of [table functions]({{ site.baseurl }}/dev/table/functions/udfs.html#table-functions) to provide a way to express it. +为了访问时态表中的数据，必须传递一个[时间属性](time_attributes.html)，该属性确定将要返回的表的版本。 +Flink 使用[表函数]({{ site.baseurl }}/zh/dev/table/functions/udfs.html#table-functions)的 SQL 语法提供一种表达它的方法。 -Once defined, a *Temporal Table Function* takes a single time argument `timeAttribute` and returns a set of rows. -This set contains the latest versions of the rows for all of the existing primary keys with respect to the given time attribute. +定义后，*时态表函数*将使用单个时间参数 timeAttribute 并返回一个行集合。 +该集合包含相对于给定时间属性的所有现有主键的行的最新版本。 -Assuming that we defined a temporal table function `Rates(timeAttribute)` based on `RatesHistory` table, we could query such a function in the following way: +假设我们基于 `RatesHistory` 表定义了一个时态表函数，我们可以通过以下方式查询该函数 `Rates(timeAttribute)`： {% highlight sql %} SELECT * FROM Rates('10:15'); @@ -144,16 +145,15 @@ rowtime currency rate 09:00 Yen 1 {% endhighlight %} -Each query to `Rates(timeAttribute)` would return the state of the `Rates` for the given `timeAttribute`. +对 `Rates(timeAttribute)` 的每个查询都将返回给定 `timeAttribute` 的 `Rates` 状态。 -**Note**: Currently, Flink doesn't support directly querying the temporal table functions with a constant time attribute parameter. At the moment, temporal table functions can only be used in joins. -The example above was used to provide an intuition about what the function `Rates(timeAttribute)` returns. +**注意**：当前 Flink 不支持使用常量时间属性参数直接查询时态表函数。目前，时态表函数只能在 join 中使用。上面的示例用于为函数 `Rates(timeAttribute)` 返回内容提供直观信息。 -See also the page about [joins for continuous queries](joins.html) for more information about how to join with a temporal table. +另请参阅有关[用于持续查询的 join ](joins.html)页面，以获取有关如何与时态表 join 的更多信息。 -### Defining Temporal Table Function +### 定义时态表函数 -The following code snippet illustrates how to create a temporal table function from an append-only table. +以下代码段说明了如何从 append-only 表中创建时态表函数。Returns a proctime attribute that can be used in subsequent time-based operations such as - time-windowed joins and group window or over + interval joins and group window or over window aggregations.
-Line `(1)` creates a `rates` [temporal table function](#temporal-table-functions), -which allows us to use the function `rates` in the [Table API](../tableApi.html#joins). +行`(1)`创建了一个 `rates` [时态表函数](#temporal-table-function)， +这使我们可以在[ Table API ](../tableApi.html#joins)中使用 `rates` 函数。 -Line `(2)` registers this function under the name `Rates` in our table environment, -which allows us to use the `Rates` function in [SQL]({{ site.baseurl }}/dev/table/sql/queries.html#joins). +行`(2)`在表环境中注册名称为 `Rates` 的函数，这使我们可以在[ SQL ]({{ site.baseurl }}/zh/dev/table/sql/queries.html#joins)中使用 `Rates` 函数。 -## Temporal Table + -Attention This is only supported in Blink planner. +## 时态表 -In order to access data in temporal table, currently one must define a `TableSource` with `LookupableTableSource`. Flink uses the SQL syntax of `FOR SYSTEM_TIME AS OF` to query temporal table, which is proposed in SQL:2011. +注意 仅 Blink planner 支持此功能。 -Assuming that we defined a temporal table called `LatestRates`, we can query such a table in the following way: +为了访问时态表中的数据，当前必须使用 `LookupableTableSource` 定义一个 `TableSource`。Flink 使用 SQL:2011 中提出的 `FOR SYSTEM_TIME AS OF` 的 SQL 语法查询时态表。 + +假设我们定义了一个时态表 `LatestRates`，我们可以通过以下方式查询此表： {% highlight sql %} SELECT * FROM LatestRates FOR SYSTEM_TIME AS OF TIME '10:15'; @@ -248,48 +249,60 @@ Euro 116 Yen 1 {% endhighlight %} -**Note**: Currently, Flink doesn't support directly querying the temporal table with a constant time. At the moment, temporal table can only be used in joins. The example above is used to provide an intuition about what the temporal table `LatestRates` returns. +**注意**：当前，Flink 不支持以固定时间直接查询时态表。目前，时态表只能在 join 中使用。上面的示例用于为时态表 `LatestRates` 返回内容提供直观信息。 -See also the page about [joins for continuous queries](joins.html) for more information about how to join with a temporal table. +另请参阅有关[用于持续查询的 join ](joins.html)页面，以获取有关如何与时态表 join 的更多信息。 -### Defining Temporal Table +### 定义时态表@@ -161,11 +161,11 @@ The following code snippet illustrates how to create a temporal table function f import org.apache.flink.table.functions.TemporalTableFunction; (...) -// Get the stream and table environments. +// 获取 stream 和 table 环境 StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); StreamTableEnvironment tEnv = StreamTableEnvironment.create(env); -// Provide a static data set of the rates history table. +// 提供一个汇率历史记录表静态数据集 List{% highlight scala %} -// Get the stream and table environments. +// 获取 stream 和 table 环境 val env = StreamExecutionEnvironment.getExecutionEnvironment val tEnv = StreamTableEnvironment.create(env) -// Provide a static data set of the rates history table. +// 提供一个汇率历史记录表静态数据集 val ratesHistoryData = new mutable.MutableList[(String, Long)] ratesHistoryData.+=(("US Dollar", 102L)) ratesHistoryData.+=(("Euro", 114L)) @@ -200,35 +200,36 @@ ratesHistoryData.+=(("Yen", 1L)) ratesHistoryData.+=(("Euro", 116L)) ratesHistoryData.+=(("Euro", 119L)) -// Create and register an example table using above data set. -// In the real setup, you should replace this with your own table. +// 用上面的数据集创建并注册一个示例表 +// 在实际设置中，应使用自己的表替换它 val ratesHistory = env .fromCollection(ratesHistoryData) .toTable(tEnv, 'r_currency, 'r_rate, 'r_proctime.proctime) tEnv.createTemporaryView("RatesHistory", ratesHistory) -// Create and register TemporalTableFunction. -// Define "r_proctime" as the time attribute and "r_currency" as the primary key. +// 创建和注册时态表函数 +// 指定 "r_proctime" 为时间属性，指定 "r_currency" 为主键 val rates = ratesHistory.createTemporalTableFunction($"r_proctime", $"r_currency") // <==== (1) tEnv.registerFunction("Rates", rates) // <==== (2) {% endhighlight %}-See also the page about [how to define LookupableTableSource](../sourceSinks.html#defining-a-tablesource-for-lookups). +另请参阅有关[如何定义 LookupableTableSource ](../sourceSinks.html#defining-a-tablesource-for-lookups)的页面。 {% top %} diff --git a/docs/dev/table/streaming/time_attributes.md b/docs/dev/table/streaming/time_attributes.md index a94d0bdfb6a38b..6896007ea58255 100644 --- a/docs/dev/table/streaming/time_attributes.md +++ b/docs/dev/table/streaming/time_attributes.md @@ -120,10 +120,10 @@ The processing time attribute is defined with the `.proctime` property during sc DataStream{% highlight java %} -// Get the stream and table environments. +// 获取 stream 和 table 环境 StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); -StreamTableEnvironment tEnv = TableEnvironment.getTableEnvironment(env); - -// Create an HBaseTableSource as a temporal table which implements LookableTableSource -// In the real setup, you should replace this with your own table. -HBaseTableSource rates = new HBaseTableSource(conf, "Rates"); -rates.setRowKey("currency", String.class); // currency as the primary key -rates.addColumn("fam1", "rate", Double.class); - -// register the temporal table into environment, then we can query it in sql -tEnv.registerTableSource("Rates", rates); +EnvironmentSettings settings = EnvironmentSettings.newInstance().build(); +StreamTableEnvironment tEnv = StreamTableEnvironment.create(env, settings); +// or TableEnvironment tEnv = TableEnvironment.create(settings); + +// 用 DDL 定义一张 HBase 表，然后我们可以在 SQL 中将其当作一张时态表使用 +// 'currency' 列是 HBase 表中的 rowKey +tEnv.executeSql( + "CREATE TABLE LatestRates (" + + " currency STRING," + + " fam1 ROW" + + ") WITH (" + + " 'connector' = 'hbase-1.4'," + + " 'table-name' = 'Rates'," + + " 'zookeeper.quorum' = 'localhost:2181'" + + ")"); {% endhighlight %} {% highlight scala %} -// Get the stream and table environments. +// 获取 stream 和 table 环境 val env = StreamExecutionEnvironment.getExecutionEnvironment -val tEnv = TableEnvironment.getTableEnvironment(env) - -// Create an HBaseTableSource as a temporal table which implements LookableTableSource -// In the real setup, you should replace this with your own table. -val rates = new HBaseTableSource(conf, "Rates") -rates.setRowKey("currency", String.class) // currency as the primary key -rates.addColumn("fam1", "rate", Double.class) - -// register the temporal table into environment, then we can query it in sql -tEnv.registerTableSource("Rates", rates) +val settings = EnvironmentSettings.newInstance().build() +val tEnv = StreamTableEnvironment.create(env, settings) +// or val tEnv = TableEnvironment.create(settings) + +// 用 DDL 定义一张 HBase 表，然后我们可以在 SQL 中将其当作一张时态表使用 +// 'currency' 列是 HBase 表中的 rowKey +tEnv.executeSql( + s""" + |CREATE TABLE LatestRates ( + | currency STRING, + | fam1 ROW+ |) WITH ( + | 'connector' = 'hbase-1.4', + | 'table-name' = 'Rates', + | 'zookeeper.quorum' = 'localhost:2181' + |) + |""".stripMargin) {% endhighlight %} diff --git a/docs/dev/table/streaming/time_attributes.zh.md b/docs/dev/table/streaming/time_attributes.zh.md index 86050d5b48cb70..d44a1faa8b073a 100644 --- a/docs/dev/table/streaming/time_attributes.zh.md +++ b/docs/dev/table/streaming/time_attributes.zh.md @@ -120,10 +120,10 @@ GROUP BY TUMBLE(user_action_time, INTERVAL '10' MINUTE); DataStreamdiff --git a/docs/dev/table/tableApi.md b/docs/dev/table/tableApi.md index 71977bddc1415e..1fc323df02bf2e 100644 --- a/docs/dev/table/tableApi.md +++ b/docs/dev/table/tableApi.md @@ -46,7 +46,6 @@ For the Expression DSL it is also necessary to import static `org.apache.flink.t {% highlight java %} import org.apache.flink.table.api.* -import org.apache.flink.table.api.java.* import static org.apache.flink.table.api.Expressions.* @@ -73,14 +72,14 @@ result.print();-The Scala Table API is enabled by importing `org.apache.flink.api.scala._` and `org.apache.flink.table.api.scala._`. +The Scala Table API is enabled by importing `org.apache.flink.table.api._`, `org.apache.flink.api.scala._`, and `org.apache.flink.table.api.bridge.scala._` (for bridging to/from DataStream). The following example shows how a Scala Table API program is constructed. Table fields are referenced using Scala's String interpolation using a dollar character (`$`). {% highlight scala %} import org.apache.flink.api.scala._ import org.apache.flink.table.api._ -import org.apache.flink.table.api.scala._ +import org.apache.flink.table.api.bridge.scala._ // environment configuration val env = ExecutionEnvironment.getExecutionEnvironment @@ -149,7 +148,7 @@ Table result = orders $("c").isNotNull() )) .select($("a").lowerCase().as("a"), $("b"), $("rowtime")) - .window(Tumble.over(interval(Duration.ofHours(1))).on($("rowtime")).as("hourlyWindow")) + .window(Tumble.over(lit(1).hours()).on($("rowtime")).as("hourlyWindow")) .groupBy($("hourlyWindow"), $("a")) .select($("a"), $("hourlyWindow").end().as("hour"), $("b").avg().as("avgBillingAmount")); {% endhighlight %} @@ -228,6 +227,50 @@ Table orders = tableEnv.from("Orders"); {% endhighlight %} ++ + Values +
+ Batch Streaming ++ +Similar to the VALUES clause in a SQL query. Produces an inline table out of the provided rows.
+You can use a `row(...)` expression to create composite rows:
+{% highlight java %} +Table table = tEnv.fromValues( + row(1, "ABC"), + row(2L, "ABCDE") +); +{% endhighlight %} +will produce a Table with a schema as follows:
+{% highlight text %} +root +|-- f0: BIGINT NOT NULL // original types INT and BIGINT are generalized to BIGINT +|-- f1: VARCHAR(5) NOT NULL // original types CHAR(3) and CHAR(5) are generalized + // to VARCHAR(5). VARCHAR is used instead of CHAR so that + // no padding is applied +{% endhighlight %} +The method will derive the types automatically from the input expressions. If types + at a certain position differ, the method will try to find a common super type for all types. If a common + super type does not exist, an exception will be thrown.
+You can also specify the requested type explicitly. It might be helpful for assigning more generic types like e.g. DECIMAL or naming the columns.
+{% highlight java %} +Table table = tEnv.fromValues( + DataTypes.ROW( + DataTypes.FIELD("id", DataTypes.DECIMAL(10, 2)), + DataTypes.FIELD("name", DataTypes.STRING()) + ), + row(1, "ABC"), + row(2L, "ABCDE") +); +{% endhighlight %} +will produce a Table with a schema as follows:
+{% highlight text %} +root +|-- id: DECIMAL(10, 2) +|-- name: STRING +{% endhighlight %} +Select
@@ -304,6 +347,50 @@ val orders: Table = tableEnv.from("Orders")+ ++ Values +
+ Batch Streaming ++ +Similar to the VALUES clause in a SQL query. Produces an inline table out of the provided rows.
+You can use a `row(...)` expression to create composite rows:
+{% highlight scala %} +val table = tEnv.fromValues( + row(1, "ABC"), + row(2L, "ABCDE") +) +{% endhighlight %} +will produce a Table with a schema as follows:
+{% highlight text %} +root +|-- f0: BIGINT NOT NULL // original types INT and BIGINT are generalized to BIGINT +|-- f1: VARCHAR(5) NOT NULL // original types CHAR(3) and CHAR(5) are generalized + // to VARCHAR(5). VARCHAR is used instead of CHAR so that + // no padding is applied +{% endhighlight %} +The method will derive the types automatically from the input expressions. If types + at a certain position differ, the method will try to find a common super type for all types. If a common + super type does not exist, an exception will be thrown.
+You can also specify the requested type explicitly. It might be helpful for assigning more generic types like e.g. DECIMAL or naming the columns.
+{% highlight scala %} +val table = tEnv.fromValues( + DataTypes.ROW( + DataTypes.FIELD("id", DataTypes.DECIMAL(10, 2)), + DataTypes.FIELD("name", DataTypes.STRING()) + ), + row(1, "ABC"), + row(2L, "ABCDE") +) +{% endhighlight %} +will produce a Table with a schema as follows:
+{% highlight text %} +root +|-- id: DECIMAL(10, 2) +|-- name: STRING +{% endhighlight %} +Select
Batch Streaming @@ -677,7 +764,7 @@ Table result = orders.groupBy($("a")).select($("a"), $("b").sum().as("d")); {% highlight java %} Table orders = tableEnv.from("Orders"); Table result = orders - .window(Tumble.over(interval(Duration.ofMinutes(5))).on($("rowtime")).as("w")) // define window + .window(Tumble.over(lit(5).minutes())).on($("rowtime")).as("w")) // define window .groupBy($("a"), $("w")) // group by key and window // access window properties and aggregate .select( @@ -736,7 +823,7 @@ Table groupByDistinctResult = orders // Distinct aggregation on time window group by Table groupByWindowDistinctResult = orders .window(Tumble - .over(interval(Duration.ofMinutes(5))) + .over(lit(5).minutes())) .on($("rowtime")) .as("w") ) @@ -1075,15 +1162,15 @@ Table fullOuterResult = left.fullOuterJoin(right, $("a").isEqual($("d")))- Time-windowed Join
+Interval Join
Batch Streaming- Note: Time-windowed joins are a subset of regular joins that can be processed in a streaming fashion.
+Note: Interval joins are a subset of regular joins that can be processed in a streaming fashion.
-A time-windowed join requires at least one equi-join predicate and a join condition that bounds the time on both sides. Such a condition can be defined by two appropriate range predicates (
-<, <=, >=, >) or a single equality predicate that compares time attributes of the same type (i.e., processing time or event time) of both input tables.For example, the following predicates are valid window join conditions:
+An interval join requires at least one equi-join predicate and a join condition that bounds the time on both sides. Such a condition can be defined by two appropriate range predicates (
+<, <=, >=, >) or a single equality predicate that compares time attributes of the same type (i.e., processing time or event time) of both input tables.For example, the following predicates are valid interval join conditions:
ltime === rtime
@@ -1098,8 +1185,8 @@ Table result = left.join(right)
.where(
and(
$("a").isEqual($("d")),
- $("ltime").isGreaterEqual($("rtime").minus(interval(Duration.ofMinutes(5)))),
- $("ltime").isLess($("rtime").plus(interval(Duration.ofMinutes(10))))
+ $("ltime").isGreaterEqual($("rtime").minus(lit(5).minutes())),
+ $("ltime").isLess($("rtime").plus(lit(10).minutes()))
))
.select($("a"), $("b"), $("e"), $("ltime"));
{% endhighlight %}
@@ -1228,15 +1315,15 @@ val fullOuterResult = left.fullOuterJoin(right, $"a" === $"d").select($"a", $"b"
- Time-windowed Join
+Interval Join
Batch Streaming- Note: Time-windowed joins are a subset of regular joins that can be processed in a streaming fashion.
+Note: Interval joins are a subset of regular joins that can be processed in a streaming fashion.
-A time-windowed join requires at least one equi-join predicate and a join condition that bounds the time on both sides. Such a condition can be defined by two appropriate range predicates (
-<, <=, >=, >) or a single equality predicate that compares time attributes of the same type (i.e., processing time or event time) of both input tables.For example, the following predicates are valid window join conditions:
+An interval join requires at least one equi-join predicate and a join condition that bounds the time on both sides. Such a condition can be defined by two appropriate range predicates (
+<, <=, >=, >) or a single equality predicate that compares time attributes of the same type (i.e., processing time or event time) of both input tables.For example, the following predicates are valid interval join conditions:
$"ltime" === $"rtime"
@@ -1368,13 +1455,31 @@ full_outer_result = left.full_outer_join(right, "a = d").select("a, b, e")
- Time-windowed Join
+Interval Join -
Batch Streaming- Currently not supported in python API.
+ ++ +Note: Interval joins are a subset of regular joins that can be processed in a streaming fashion.
+ +An interval join requires at least one equi-join predicate and a join condition that bounds the time on both sides. Such a condition can be defined by two appropriate range predicates (
+<, <=, >=, >) or a single equality predicate that compares time attributes of the same type (i.e., processing time or event time) of both input tables.For example, the following predicates are valid interval join conditions:
+ +-
+
ltime = rtime
+ ltime >= rtime && ltime < rtime + 2.second
+
@@ -1973,13 +2078,13 @@ result3 = in.order_by("a.asc").offset(10).fetch(5) Batch Streaming - Similar to the INSERT INTO clause in a SQL query. Performs a insertion into a registered output table.
+Similar to the `INSERT INTO` clause in a SQL query, the method performs an insertion into a registered output table. The `executeInsert()` method will immediately submit a Flink job which execute the insert operation.
Output tables must be registered in the TableEnvironment (see Connector tables). Moreover, the schema of the registered table must match the schema of the query.
{% highlight java %} Table orders = tableEnv.from("Orders"); -orders.insertInto("OutOrders"); +orders.executeInsert("OutOrders"); {% endhighlight %}- @@ -2033,13 +2138,13 @@ orders.insertInto("OutOrders") Batch StreamingSimilar to the INSERT INTO clause in a SQL query. Performs a insertion into a registered output table.
+Similar to the `INSERT INTO` clause in a SQL query, the method performs an insertion into a registered output table. The `executeInsert()` method will immediately submit a Flink job which execute the insert operation.
Output tables must be registered in the TableEnvironment (see Connector tables). Moreover, the schema of the registered table must match the schema of the query.
{% highlight scala %} val orders: Table = tableEnv.from("Orders") -orders.insertInto("OutOrders") +orders.executeInsert("OutOrders") {% endhighlight %}- @@ -2186,10 +2291,10 @@ Tumbling windows are defined by using the `Tumble` class as follows:Similar to the INSERT INTO clause in a SQL query. Performs a insertion into a registered output table.
+Similar to the INSERT INTO clause in a SQL query. Performs a insertion into a registered output table. The executeInsert method will immediately submit a flink job which execute the insert operation.
Output tables must be registered in the TableEnvironment (see Register a TableSink). Moreover, the schema of the registered table must match the schema of the query.
{% highlight python %} -orders = table_env.from_path("Orders"); -orders.insert_into("OutOrders"); +orders = table_env.from_path("Orders") +orders.execute_insert("OutOrders") {% endhighlight %}{% highlight java %} // Tumbling Event-time Window -.window(Tumble.over(interval(Duration.ofMinutes(10))).on($("rowtime")).as("w")); +.window(Tumble.over(lit(10).minutes()).on($("rowtime")).as("w")); // Tumbling Processing-time Window (assuming a processing-time attribute "proctime") -.window(Tumble.over(interval(Duration.ofMinutes(10))).on($("proctime")).as("w")); +.window(Tumble.over(lit(10).minutes()).on($("proctime")).as("w")); // Tumbling Row-count Window (assuming a processing-time attribute "proctime") .window(Tumble.over(rowInterval(10)).on($("proctime")).as("w")); @@ -2261,14 +2366,14 @@ Sliding windows are defined by using the `Slide` class as follows:@@ -76,7 +154,7 @@ Apache Flink ships with first class support for a number of common deployment ta{% highlight java %} // Sliding Event-time Window -.window(Slide.over(interval(Duration.ofMinutes(10))) - .every(interval(Duration.ofMinutes(5))) +.window(Slide.over(lit(10).minutes()) + .every(lit(5).minutes()) .on($("rowtime")) .as("w")); // Sliding Processing-time window (assuming a processing-time attribute "proctime") -.window(Slide.over(interval(Duration.ofMinutes(10))) - .every(interval(Duration.ofMinutes(5))) +.window(Slide.over(lit(10).minutes()) + .every(lit(5).minutes()) .on($("proctime")) .as("w")); @@ -2338,10 +2443,10 @@ A session window is defined by using the `Session` class as follows:{% highlight java %} // Session Event-time Window -.window(Session.withGap(interval(Duration.ofMinutes(10))).on($("rowtime")).as("w")); +.window(Session.withGap(lit(10).minutes()).on($("rowtime")).as("w")); // Session Processing-time Window (assuming a processing-time attribute "proctime") -.window(Session.withGap(interval(Duration.ofMinutes(10))).on($("proctime")).as("w")); +.window(Session.withGap(lit(10).minutes()).on($("proctime")).as("w")); {% endhighlight %}@@ -2527,10 +2632,10 @@ The `OverWindow` defines a range of rows over which aggregates are computed. `Ov{% highlight java %} // Bounded Event-time over window (assuming an event-time attribute "rowtime") -.window(Over.partitionBy($("a")).orderBy($("rowtime")).preceding(interval(Duration.ofMinutes(1))).as("w")) +.window(Over.partitionBy($("a")).orderBy($("rowtime")).preceding(lit(1).minutes()).as("w")) // Bounded Processing-time over window (assuming a processing-time attribute "proctime") -.window(Over.partitionBy($("a")).orderBy($("proctime")).preceding(interval(Duration.ofMinutes(1))).as("w")) +.window(Over.partitionBy($("a")).orderBy($("proctime")).preceding(lit(1).minutes()).as("w")) // Bounded Event-time Row-count over window (assuming an event-time attribute "rowtime") .window(Over.partitionBy($("a")).orderBy($("rowtime")).preceding(rowInterval(10)).as("w")) @@ -2718,7 +2823,7 @@ AggregateFunction myAggFunc = new MyMinMax(); tableEnv.registerFunction("myAggFunc", myAggFunc); Table table = input - .window(Tumble.over(interval(Duration.ofMinutes(5))) + .window(Tumble.over(lit(5).minutes()) .on($("rowtime")) .as("w")) // define window .groupBy($("key"), $("w")) // group by key and window @@ -2810,7 +2915,7 @@ Table result = orders tableEnv.registerFunction("top2", new Top2()); Table orders = tableEnv.from("Orders"); Table result = orders - .window(Tumble.over(interval(Duration.ofMinutes(5))) + .window(Tumble.over(lit(5).minutes()) .on($("rowtime")) .as("w")) // define window .groupBy($("a"), $("w")) // group by key and window @@ -3158,6 +3263,6 @@ timeIndicator = fieldReference , "." , ( "proctime" | "rowtime" ) ; **Temporal intervals:** Temporal intervals can be represented as number of months (`Types.INTERVAL_MONTHS`) or number of milliseconds (`Types.INTERVAL_MILLIS`). Intervals of same type can be added or subtracted (e.g. `1.hour + 10.minutes`). Intervals of milliseconds can be added to time points (e.g. `"2016-08-10".toDate + 5.days`). -**Scala expressions:** Scala expressions use implicit conversions. Therefore, make sure to add the wildcard import `org.apache.flink.table.api.scala._` to your programs. In case a literal is not treated as an expression, use `.toExpr` such as `3.toExpr` to force a literal to be converted. +**Scala expressions:** Scala expressions use implicit conversions. Therefore, make sure to add the wildcard import `org.apache.flink.table.api._` to your programs. In case a literal is not treated as an expression, use `.toExpr` such as `3.toExpr` to force a literal to be converted. {% top %} diff --git a/docs/dev/table/tableApi.zh.md b/docs/dev/table/tableApi.zh.md index 25e2904ae414ec..50fc7287566c71 100644 --- a/docs/dev/table/tableApi.zh.md +++ b/docs/dev/table/tableApi.zh.md @@ -46,7 +46,6 @@ For the Expression DSL it is also necessary to import static `org.apache.flink.t {% highlight java %} import org.apache.flink.table.api.* -import org.apache.flink.table.api.java.* import static org.apache.flink.table.api.Expressions.* @@ -73,14 +72,14 @@ result.print();-The Scala Table API is enabled by importing `org.apache.flink.api.scala._` and `org.apache.flink.table.api.scala._`. +The Scala Table API is enabled by importing `org.apache.flink.table.api._`, `org.apache.flink.api.scala._`, and `org.apache.flink.table.api.bridge.scala._` (for bridging to/from DataStream). The following example shows how a Scala Table API program is constructed. Table fields are referenced using Scala's String interpolation using a dollar character (`$`). {% highlight scala %} import org.apache.flink.api.scala._ import org.apache.flink.table.api._ -import org.apache.flink.table.api.scala._ +import org.apache.flink.table.api.bridge.scala._ // environment configuration val env = ExecutionEnvironment.getExecutionEnvironment @@ -149,7 +148,7 @@ Table result = orders $("c").isNotNull() )) .select($("a").lowerCase().as("a"), $("b"), $("rowtime")) - .window(Tumble.over(interval(Duration.ofHours(1))).on($("rowtime")).as("hourlyWindow")) + .window(Tumble.over(lit(1).hours()).on($("rowtime")).as("hourlyWindow")) .groupBy($("hourlyWindow"), $("a")) .select($("a"), $("hourlyWindow").end().as("hour"), $("b").avg().as("avgBillingAmount")); {% endhighlight %} @@ -228,6 +227,50 @@ Table orders = tableEnv.from("Orders"); {% endhighlight %} +@@ -62,8 +140,8 @@ Apache Flink ships with first class support for a number of common deployment ta Yarn+ + Values +
+ Batch Streaming ++ +Similar to the VALUES clause in a SQL query. Produces an inline table out of the provided rows.
+You can use a `row(...)` expression to create composite rows:
+{% highlight java %} +Table table = tEnv.fromValues( + row(1, "ABC"), + row(2L, "ABCDE") +); +{% endhighlight %} +will produce a Table with a schema as follows:
+{% highlight text %} +root +|-- f0: BIGINT NOT NULL // original types INT and BIGINT are generalized to BIGINT +|-- f1: VARCHAR(5) NOT NULL // original types CHAR(3) and CHAR(5) are generalized + // to VARCHAR(5). VARCHAR is used instead of CHAR so that + // no padding is applied +{% endhighlight %} +The method will derive the types automatically from the input expressions. If types + at a certain position differ, the method will try to find a common super type for all types. If a common + super type does not exist, an exception will be thrown.
+You can also specify the requested type explicitly. It might be helpful for assigning more generic types like e.g. DECIMAL or naming the columns.
+{% highlight java %} +Table table = tEnv.fromValues( + DataTypes.ROW( + DataTypes.FIELD("id", DataTypes.DECIMAL(10, 2)), + DataTypes.FIELD("name", DataTypes.STRING()) + ), + row(1, "ABC"), + row(2L, "ABCDE") +); +{% endhighlight %} +will produce a Table with a schema as follows:
+{% highlight text %} +root +|-- id: DECIMAL(10, 2) +|-- name: STRING +{% endhighlight %} +Select
@@ -304,6 +347,50 @@ val orders: Table = tableEnv.from("Orders")+ ++ Values +
+ Batch Streaming ++ +Similar to the VALUES clause in a SQL query. Produces an inline table out of the provided rows.
+You can use a `row(...)` expression to create composite rows:
+{% highlight scala %} +val table = tEnv.fromValues( + row(1, "ABC"), + row(2L, "ABCDE") +) +{% endhighlight %} +will produce a Table with a schema as follows:
+{% highlight text %} +root +|-- f0: BIGINT NOT NULL // original types INT and BIGINT are generalized to BIGINT +|-- f1: VARCHAR(5) NOT NULL // original types CHAR(3) and CHAR(5) are generalized + // to VARCHAR(5). VARCHAR is used instead of CHAR so that + // no padding is applied +{% endhighlight %} +The method will derive the types automatically from the input expressions. If types + at a certain position differ, the method will try to find a common super type for all types. If a common + super type does not exist, an exception will be thrown.
+You can also specify the requested type explicitly. It might be helpful for assigning more generic types like e.g. DECIMAL or naming the columns.
+{% highlight scala %} +val table = tEnv.fromValues( + DataTypes.ROW( + DataTypes.FIELD("id", DataTypes.DECIMAL(10, 2)), + DataTypes.FIELD("name", DataTypes.STRING()) + ), + row(1, "ABC"), + row(2L, "ABCDE") +) +{% endhighlight %} +will produce a Table with a schema as follows:
+{% highlight text %} +root +|-- id: DECIMAL(10, 2) +|-- name: STRING +{% endhighlight %} +Select
Batch Streaming @@ -677,7 +764,7 @@ Table result = orders.groupBy($("a")).select($("a"), $("b").sum().as("d")); {% highlight java %} Table orders = tableEnv.from("Orders"); Table result = orders - .window(Tumble.over(interval(Duration.ofMinutes(5))).on($("rowtime")).as("w")) // define window + .window(Tumble.over(lit(5).minutes()).on($("rowtime")).as("w")) // define window .groupBy($("a"), $("w")) // group by key and window // access window properties and aggregate .select( @@ -736,7 +823,7 @@ Table groupByDistinctResult = orders // Distinct aggregation on time window group by Table groupByWindowDistinctResult = orders .window(Tumble - .over(interval(Duration.ofMinutes(5))) + .over(lit(5).minutes()) .on($("rowtime")) .as("w") ) @@ -1074,15 +1161,15 @@ Table fullOuterResult = left.fullOuterJoin(right, $("a").isEqual($("d")))- Time-windowed Join
+Interval Join
Batch Streaming- Note: Time-windowed joins are a subset of regular joins that can be processed in a streaming fashion.
+Note: Interval joins are a subset of regular joins that can be processed in a streaming fashion.
-A time-windowed join requires at least one equi-join predicate and a join condition that bounds the time on both sides. Such a condition can be defined by two appropriate range predicates (
-<, <=, >=, >) or a single equality predicate that compares time attributes of the same type (i.e., processing time or event time) of both input tables.For example, the following predicates are valid window join conditions:
+A interval join requires at least one equi-join predicate and a join condition that bounds the time on both sides. Such a condition can be defined by two appropriate range predicates (
+<, <=, >=, >) or a single equality predicate that compares time attributes of the same type (i.e., processing time or event time) of both input tables.For example, the following predicates are valid interval join conditions:
ltime === rtime
@@ -1097,8 +1184,8 @@ Table result = left.join(right)
.where(
and(
$("a").isEqual($("d")),
- $("ltime").isGreaterEqual($("rtime").minus(interval(Duration.ofMinutes(5)))),
- $("ltime").isLess($("rtime").plus(interval(Duration.ofMinutes(10))))
+ $("ltime").isGreaterEqual($("rtime").minus(lit(5).minutes())),
+ $("ltime").isLess($("rtime").plus(lit(10).minutes()))
))
.select($("a"), $("b"), $("e"), $("ltime"));
{% endhighlight %}
@@ -1227,15 +1314,15 @@ val fullOuterResult = left.fullOuterJoin(right, $"a" === $"d").select($"a", $"b"
- Time-windowed Join
+Interval Join
Batch Streaming- Note: Time-windowed joins are a subset of regular joins that can be processed in a streaming fashion.
+Note: Interval joins are a subset of regular joins that can be processed in a streaming fashion.
-A time-windowed join requires at least one equi-join predicate and a join condition that bounds the time on both sides. Such a condition can be defined by two appropriate range predicates (
-<, <=, >=, >) or a single equality predicate that compares time attributes of the same type (i.e., processing time or event time) of both input tables.For example, the following predicates are valid window join conditions:
+A interval join requires at least one equi-join predicate and a join condition that bounds the time on both sides. Such a condition can be defined by two appropriate range predicates (
+<, <=, >=, >) or a single equality predicate that compares time attributes of the same type (i.e., processing time or event time) of both input tables.For example, the following predicates are valid interval join conditions:
$"ltime" === $"rtime"
@@ -1367,13 +1454,28 @@ full_outer_result = left.full_outer_join(right, "a = d").select("a, b, e")
- Time-windowed Join
+Interval Join -
批处理 流处理- +Python API暂不支持。
-+ 注意： Interval Join 是所有常规流式数据处理join操作中的其中一种场景。
+Interval Join 要求至少有一个等值连接条件以及一个用于划定时间间隔的条件。对两条数据流时间的划定可以通过两个范围比较确定一个合适时间区间(
+<, <=, >=, >)， 也可以简单的通过对相同时间属性（当前处理时间或事件时间）的时间值进行等值比较确定。如下示例是合法的Interval Join条件：
+ +-
+
ltime = rtime
+ ltime >= rtime && ltime < rtime + 2.second
+
@@ -1972,13 +2074,13 @@ result3 = in.order_by("a.asc").offset(10).fetch(5) Batch Streaming - Similar to the INSERT INTO clause in a SQL query. Performs a insertion into a registered output table.
+Similar to the `INSERT INTO` clause in a SQL query, the method performs an insertion into a registered output table. The `executeInsert()` method will immediately submit a Flink job which execute the insert operation.
Output tables must be registered in the TableEnvironment (see Register a TableSink). Moreover, the schema of the registered table must match the schema of the query.
{% highlight java %} Table orders = tableEnv.from("Orders"); -orders.insertInto("OutOrders"); +orders.executeInsert("OutOrders"); {% endhighlight %}- @@ -2032,13 +2134,13 @@ orders.insertInto("OutOrders") 批处理 流处理Similar to the INSERT INTO clause in a SQL query. Performs a insertion into a registered output table.
+Similar to the `INSERT INTO` clause in a SQL query, the method performs an insertion into a registered output table. The `executeInsert()` method will immediately submit a Flink job which execute the insert operation.
Output tables must be registered in the TableEnvironment (see Connector tables). Moreover, the schema of the registered table must match the schema of the query.
{% highlight scala %} val orders: Table = tableEnv.from("Orders") -orders.insertInto("OutOrders") +orders.executeInsert("OutOrders") {% endhighlight %}- @@ -2185,10 +2287,10 @@ Tumbling windows are defined by using the `Tumble` class as follows:类似于SQL请求中的INSERT INTO子句。将数据输出到一个已注册的输出表中。
+类似于SQL请求中的INSERT INTO子句。将数据输出到一个已注册的输出表中。`execute_insert` 方法会立即提交一个 Flink 作业，触发插入操作。
输出表必须先在TableEnvironment中注册（详见注册一个TableSink）。此外，注册的表的模式（schema）必须和请求的结果的模式（schema）相匹配。
{% highlight python %} -orders = table_env.from_path("Orders"); -orders.insert_into("OutOrders"); +orders = table_env.from_path("Orders") +orders.execute_insert("OutOrders") {% endhighlight %}{% highlight java %} // Tumbling Event-time Window -.window(Tumble.over(interval(Duration.ofMinutes(10))).on($("rowtime")).as("w")); +.window(Tumble.over(lit(10).minutes()).on($("rowtime")).as("w")); // Tumbling Processing-time Window (assuming a processing-time attribute "proctime") -.window(Tumble.over(interval(Duration.ofMinutes(10))).on($("proctime")).as("w")); +.window(Tumble.over(lit(10).minutes()).on($("proctime")).as("w")); // Tumbling Row-count Window (assuming a processing-time attribute "proctime") .window(Tumble.over(rowInterval(10)).on($("proctime")).as("w")); @@ -2260,14 +2362,14 @@ Sliding windows are defined by using the `Slide` class as follows:{% highlight java %} // Sliding Event-time Window -.window(Slide.over(interval(Duration.ofMinutes(10))) - .every(interval(Duration.ofMinutes(5))) +.window(Slide.over(lit(10).minutes()) + .every(lit(5).minutes()) .on($("rowtime")) .as("w")); // Sliding Processing-time window (assuming a processing-time attribute "proctime") -.window(Slide.over(interval(Duration.ofMinutes(10))) - .every(interval(Duration.ofMinutes(5))) +.window(Slide.over(lit(10).minutes()) + .every(lit(5).minutes()) .on($("proctime")) .as("w")); @@ -2337,10 +2439,10 @@ A session window is defined by using the `Session` class as follows:{% highlight java %} // Session Event-time Window -.window(Session.withGap(interval(Duration.ofMinutes(10))).on($("rowtime")).as("w")); +.window(Session.withGap(lit(10).minutes()).on($("rowtime")).as("w")); // Session Processing-time Window (assuming a processing-time attribute "proctime") -.window(Session.withGap(interval(Duration.ofMinutes(10))).on($("proctime")).as("w")); +.window(Session.withGap(lit(10).minutes()).on($("proctime")).as("w")); {% endhighlight %}@@ -2526,10 +2628,10 @@ The `OverWindow` defines a range of rows over which aggregates are computed. `Ov{% highlight java %} // Bounded Event-time over window (assuming an event-time attribute "rowtime") -.window(Over.partitionBy($("a")).orderBy($("rowtime")).preceding(interval(Duration.ofMinutes(1))).as("w")) +.window(Over.partitionBy($("a")).orderBy($("rowtime")).preceding(lit(1).minutes()).as("w")) // Bounded Processing-time over window (assuming a processing-time attribute "proctime") -.window(Over.partitionBy($("a")).orderBy($("proctime")).preceding(interval(Duration.ofMinutes(1))).as("w")) +.window(Over.partitionBy($("a")).orderBy($("proctime")).preceding(lit(1).minutes()).as("w")) // Bounded Event-time Row-count over window (assuming an event-time attribute "rowtime") .window(Over.partitionBy($("a")).orderBy($("rowtime")).preceding(rowInterval(10)).as("w")) @@ -2717,7 +2819,7 @@ AggregateFunction myAggFunc = new MyMinMax(); tableEnv.registerFunction("myAggFunc", myAggFunc); Table table = input - .window(Tumble.over(interval(Duration.ofMinutes(5))) + .window(Tumble.over(lit(5).minutes()) .on($("rowtime")) .as("w")) // define window .groupBy($("key"), $("w")) // group by key and window @@ -2809,7 +2911,7 @@ Table result = orders tableEnv.registerFunction("top2", new Top2()); Table orders = tableEnv.from("Orders"); Table result = orders - .window(Tumble.over(interval(Duration.ofMinutes(5))) + .window(Tumble.over(lit(5).minutes()) .on($("rowtime")) .as("w")) // define window .groupBy($("a"), $("w")) // group by key and window @@ -3157,6 +3259,6 @@ timeIndicator = fieldReference , "." , ( "proctime" | "rowtime" ) ; **Temporal intervals:** Temporal intervals can be represented as number of months (`Types.INTERVAL_MONTHS`) or number of milliseconds (`Types.INTERVAL_MILLIS`). Intervals of same type can be added or subtracted (e.g. `1.hour + 10.minutes`). Intervals of milliseconds can be added to time points (e.g. `"2016-08-10".toDate + 5.days`). -**Scala expressions:** Scala expressions use implicit conversions. Therefore, make sure to add the wildcard import `org.apache.flink.table.api.scala._` to your programs. In case a literal is not treated as an expression, use `.toExpr` such as `3.toExpr` to force a literal to be converted. +**Scala expressions:** Scala expressions use implicit conversions. Therefore, make sure to add the wildcard import `org.apache.flink.table.api._` to your programs. In case a literal is not treated as an expression, use `.toExpr` such as `3.toExpr` to force a literal to be converted. {% top %} diff --git a/docs/dev/table/types.md b/docs/dev/table/types.md index 4dc42af4270b90..45b6a9456416e3 100644 --- a/docs/dev/table/types.md +++ b/docs/dev/table/types.md @@ -36,7 +36,7 @@ Starting with Flink 1.9, the Table & SQL API will receive a new type system that solution for API stability and standard compliance. Reworking the type system is a major effort that touches almost all user-facing interfaces. Therefore, its -introduction spans multiple releases, and the community aims to finish this effort by Flink 1.10. +introduction spans multiple releases, and the community aims to finish this effort by Flink 1.12. Due to the simultaneous addition of a new planner for table programs (see [FLINK-11439](https://issues.apache.org/jira/browse/FLINK-11439)), not every combination of planner and data type is supported. Furthermore, planners might not support every @@ -211,14 +211,15 @@ The following data types are supported: | `DOUBLE` | | | `DATE` | | | `TIME` | Supports only a precision of `0`. | -| `TIMESTAMP` | Supports only a precision of `3`. | -| `TIMESTAMP WITH LOCAL TIME ZONE` | Supports only a precision of `3`. | +| `TIMESTAMP` | | +| `TIMESTAMP WITH LOCAL TIME ZONE` | | | `INTERVAL` | Supports only interval of `MONTH` and `SECOND(3)`. | | `ARRAY` | | | `MULTISET` | | | `MAP` | | | `ROW` | | | `RAW` | | +| stuctured types | Only exposed in user-defined functions yet. | Limitations ----------- @@ -227,10 +228,7 @@ Limitations have not been updated to the new type system yet. Use the string representations declared in the [old planner section](#old-planner). -**Connector Descriptors and SQL Client**: Descriptor string representations have not been updated to the new -type system yet. Use the string representation declared in the [Connect to External Systems section](./connect.html#type-strings) - -**User-defined Functions**: User-defined functions cannot declare a data type yet. +**User-defined Functions**: User-defined aggregate functions cannot declare a data type yet. Scalar and table functions fully support data types. List of Data Types ------------------ @@ -267,10 +265,11 @@ and `2,147,483,647` (both inclusive). If no length is specified, `n` is equal to **Bridging to JVM Types** -| Java Type | Input | Output | Remarks | -|:-------------------|:-----:|:------:|:------------------------| -|`java.lang.String` | X | X | *Default* | -|`byte[]` | X | X | Assumes UTF-8 encoding. | +| Java Type | Input | Output | Remarks | +|:----------------------------------------|:-----:|:------:|:-------------------------| +|`java.lang.String` | X | X | *Default* | +|`byte[]` | X | X | Assumes UTF-8 encoding. | +|`org.apache.flink.table.data.StringData` | X | X | Internal data structure. | #### `VARCHAR` / `STRING` @@ -306,10 +305,11 @@ between `1` and `2,147,483,647` (both inclusive). If no length is specified, `n` **Bridging to JVM Types** -| Java Type | Input | Output | Remarks | -|:-------------------|:-----:|:------:|:------------------------| -|`java.lang.String` | X | X | *Default* | -|`byte[]` | X | X | Assumes UTF-8 encoding. | +| Java Type | Input | Output | Remarks | +|:----------------------------------------|:-----:|:------:|:-------------------------| +|`java.lang.String` | X | X | *Default* | +|`byte[]` | X | X | Assumes UTF-8 encoding. | +|`org.apache.flink.table.data.StringData` | X | X | Internal data structure. | ### Binary Strings @@ -428,9 +428,10 @@ The default value for `s` is `0`. **Bridging to JVM Types** -| Java Type | Input | Output | Remarks | -|:----------------------|:-----:|:------:|:------------------------| -|`java.math.BigDecimal` | X | X | *Default* | +| Java Type | Input | Output | Remarks | +|:-----------------------------------------|:-----:|:------:|:-------------------------| +|`java.math.BigDecimal` | X | X | *Default* | +|`org.apache.flink.table.data.DecimalData` | X | X | Internal data structure. | #### `TINYINT` @@ -459,7 +460,7 @@ DataTypes.TINYINT() | Java Type | Input | Output | Remarks | |:-------------------|:-----:|:------:|:---------------------------------------------| |`java.lang.Byte` | X | X | *Default* | -|`byte` | X | (X) | Output only if type is not nullable. | +|`byte` | X | (X) | Output only if type is not nullable. | #### `SMALLINT` @@ -488,7 +489,7 @@ DataTypes.SMALLINT() | Java Type | Input | Output | Remarks | |:-------------------|:-----:|:------:|:---------------------------------------------| |`java.lang.Short` | X | X | *Default* | -|`short` | X | (X) | Output only if type is not nullable. | +|`short` | X | (X) | Output only if type is not nullable. | #### `INT` @@ -521,7 +522,7 @@ DataTypes.INT() | Java Type | Input | Output | Remarks | |:-------------------|:-----:|:------:|:---------------------------------------------| |`java.lang.Integer` | X | X | *Default* | -|`int` | X | (X) | Output only if type is not nullable. | +|`int` | X | (X) | Output only if type is not nullable. | #### `BIGINT` @@ -551,7 +552,7 @@ DataTypes.BIGINT() | Java Type | Input | Output | Remarks | |:-------------------|:-----:|:------:|:---------------------------------------------| |`java.lang.Long` | X | X | *Default* | -|`long` | X | (X) | Output only if type is not nullable. | +|`long` | X | (X) | Output only if type is not nullable. | ### Approximate Numerics @@ -584,7 +585,7 @@ DataTypes.FLOAT() | Java Type | Input | Output | Remarks | |:-------------------|:-----:|:------:|:---------------------------------------------| |`java.lang.Float` | X | X | *Default* | -|`float` | X | (X) | Output only if type is not nullable. | +|`float` | X | (X) | Output only if type is not nullable. | #### `DOUBLE` @@ -617,7 +618,7 @@ DataTypes.DOUBLE() | Java Type | Input | Output | Remarks | |:-------------------|:-----:|:------:|:---------------------------------------------| |`java.lang.Double` | X | X | *Default* | -|`double` | X | (X) | Output only if type is not nullable. | +|`double` | X | (X) | Output only if type is not nullable. | ### Date and Time @@ -741,10 +742,11 @@ is specified, `p` is equal to `6`. **Bridging to JVM Types** -| Java Type | Input | Output | Remarks | -|:-------------------------|:-----:|:------:|:----------------------------------------------------| -|`java.time.LocalDateTime` | X | X | *Default* | -|`java.sql.Timestamp` | X | X | | +| Java Type | Input | Output | Remarks | +|:-------------------------------------------|:-----:|:------:|:-------------------------| +|`java.time.LocalDateTime` | X | X | *Default* | +|`java.sql.Timestamp` | X | X | | +|`org.apache.flink.table.data.TimestampData` | X | X | Internal data structure. | #### `TIMESTAMP WITH TIME ZONE` @@ -837,6 +839,7 @@ of digits of fractional seconds (*precision*). `p` must have a value between `0` |`int` | X | (X) | Describes the number of seconds since epoch.@@ -52,7 +130,7 @@ Apache Flink ships with first class support for a number of common deployment ta
Output only if type is not nullable. | |`java.lang.Long` | X | X | Describes the number of milliseconds since epoch. | |`long` | X | (X) | Describes the number of milliseconds since epoch.
Output only if type is not nullable. | +|`org.apache.flink.table.data.TimestampData` | X | X | Internal data structure. | #### `INTERVAL YEAR TO MONTH` @@ -1001,9 +1004,10 @@ equivalent to `ARRAY`. **Bridging to JVM Types** -| Java Type | Input | Output | Remarks | -|:----------|:-----:|:------:|:----------------------------------| -|*t*`[]` | (X) | (X) | Depends on the subtype. *Default* | +| Java Type | Input | Output | Remarks | +|:---------------------------------------|:-----:|:------:|:----------------------------------| +|*t*`[]` | (X) | (X) | Depends on the subtype. *Default* | +|`org.apache.flink.table.data.ArrayData` | X | X | Internal data structure. | #### `MAP` @@ -1037,10 +1041,11 @@ and `vt` is the data type of the value elements. **Bridging to JVM Types** -| Java Type | Input | Output | Remarks | -|:--------------------------------------|:-----:|:------:|:----------| -| `java.util.Map `. **Bridging to JVM Types** -| Java Type | Input | Output | Remarks | -|:-------------------------------------|:-----:|:------:|:---------------------------------------------------------| -|`java.util.Map `. **Bridging to JVM Types** -| Java Type | Input | Output | Remarks | -|:----------------------------|:-----:|:------:|:------------------------| -|`org.apache.flink.types.Row` | X | X | *Default* | +| Java Type | Input | Output | Remarks | +|:-------------------------------------|:-----:|:------:|:-------------------------| +|`org.apache.flink.types.Row` | X | X | *Default* | +|`org.apache.flink.table.data.RowData` | X | X | Internal data structure. | + +### User-Defined Data Types + +Attention User-defined data types are not fully supported yet. They are +currently (as of Flink 1.11) only exposed as unregistered structured types in parameters and return types of functions. + +A structured type is similar to an object in an object-oriented programming language. It contains +zero, one or more attributes. Each attribute consists of a name and a type. + +There are two kinds of structured types: + +- Types that are stored in a catalog and are identified by a _catalog identifer_ (like `cat.db.MyType`). Those +are equal to the SQL standard definition of structured types. + +- Anonymously defined, unregistered types (usually reflectively extracted) that are identified by +an _implementation class_ (like `com.myorg.model.MyType`). Those are useful when programmatically +defining a table program. They enable reusing existing JVM classes without manually defining the +schema of a data type again. + +#### Registered Structured Types + +Currently, registered structured types are not supported. Thus, they cannot be stored in a catalog +or referenced in a `CREATE TABLE` DDL. + +#### Unregistered Structured Types + +Unregistered structured types can be created from regular POJOs (Plain Old Java Objects) using automatic reflective extraction. + +The implementation class of a structured type must meet the following requirements: +- The class must be globally accessible which means it must be declared `public`, `static`, and not `abstract`. +- The class must offer a default constructor with zero arguments or a full constructor that assigns all +fields. +- All fields of the class must be readable by either `public` declaration or a getter that follows common +coding style such as `getField()`, `isField()`, `field()`. +- All fields of the class must be writable by either `public` declaration, fully assigning constructor, +or a setter that follows common coding style such as `setField(...)`, `field(...)`. +- All fields must be mapped to a data type either implicitly via reflective extraction or explicitly +using the `@DataTypeHint` [annotations](#data-type-annotations). +- Fields that are declared `static` or `transient` are ignored. + +The reflective extraction supports arbitrary nesting of fields as long as a field type does not +(transitively) refer to itself. + +The declared field class (e.g. `public int age;`) must be contained in the list of supported JVM +bridging classes defined for every data type in this document (e.g. `java.lang.Integer` or `int` for `INT`). + +For some classes an annotation is required in order to map the class to a data type (e.g. `@DataTypeHint("DECIMAL(10, 2)")` +to assign a fixed precision and scale for `java.math.BigDecimal`). + +**Declaration** + + + ++ +**Bridging to JVM Types** + +| Java Type | Input | Output | Remarks | +|:-------------------------------------|:-----:|:------:|:----------------------------------------| +|*class* | X | X | Originating class or subclasses (for input) or+{% highlight java %} +class User { + + // extract fields automatically + public int age; + public String name; + + // enrich the extraction with precision information + public @DataTypeHint("DECIMAL(10, 2)") BigDecimal totalBalance; + + // enrich the extraction with forcing using RAW types + public @DataTypeHint("RAW") Class modelClass; +} + +DataTypes.of(User.class); +{% endhighlight %} ++ ++{% highlight scala %} +case class User( + + // extract fields automatically + age: Int, + name: String, + + // enrich the extraction with precision information + @DataTypeHint("DECIMAL(10, 2)") totalBalance: java.math.BigDecimal, + + // enrich the extraction with forcing using a RAW type + @DataTypeHint("RAW") modelClass: Class[_] +) + +DataTypes.of(classOf[User]) +{% endhighlight %} ++ +
superclasses (for output). *Default* | +|`org.apache.flink.types.Row` | X | X | Represent the structured type as a row. | +|`org.apache.flink.table.data.RowData` | X | X | Internal data structure. | ### Other Data Types @@ -1160,6 +1267,48 @@ DataTypes.BOOLEAN() |`java.lang.Boolean` | X | X | *Default* | |`boolean` | X | (X) | Output only if type is not nullable. | +#### `RAW` + +Data type of an arbitrary serialized type. This type is a black box within the table ecosystem +and is only deserialized at the edges. + +The raw type is an extension to the SQL standard. + +**Declaration** + ++ ++ +The type can be declared using `RAW('class', 'snapshot')` where `class` is the originating class and +`snapshot` is the serialized `TypeSerializerSnapshot` in Base64 encoding. Usually, the type string is not +declared directly but is generated while persisting the type. + +In the API, the `RAW` type can be declared either by directly supplying a `Class` + `TypeSerializer` or +by passing `Class` and letting the framework extract `Class` + `TypeSerializer` from there. + +**Bridging to JVM Types** + +| Java Type | Input | Output | Remarks | +|:------------------|:-----:|:------:|:-------------------------------------------| +|*class* | X | X | Originating class or subclasses (for input) or+{% highlight text %} +RAW('class', 'snapshot') +{% endhighlight %} ++ ++{% highlight java %} +DataTypes.RAW(class, serializer) + +DataTypes.RAW(class) +{% endhighlight %} ++ +
superclasses (for output). *Default* | +|`byte[]` | | X | | +|`org.apache.flink.table.data.RawValueData` | X | X | Internal data structure. | + #### `NULL` Data type for representing untyped `NULL` values. @@ -1197,45 +1346,122 @@ DataTypes.NULL() |`java.lang.Object` | X | X | *Default* | |*any class* | | (X) | Any non-primitive type. | -#### `RAW` +Data Type Extraction +-------------------- + +At many locations in the API, Flink tries to automatically extract data type from class information using +reflection to avoid repetitive manual schema work. However, extracting a data type reflectively is not always +successful because logical information might be missing. Therefore, it might be necessary to add additional +information close to a class or field declaration for supporting the extraction logic. + +The following table lists classes that can be implicitly mapped to a data type without requiring further information. + +If you intend to implement classes in Scala, *it is recommended to use boxed types* (e.g. `java.lang.Integer`) +instead of Scala's primitives. Scala's primitives (e.g. `Int` or `Double`) are compiled to JVM primitives (e.g. +`int`/`double`) and result in `NOT NULL` semantics as shown in the table below. Furthermore, Scala primitives that +are used in generics (e.g. `java.lang.Map[Int, Double]`) are erased during compilation and lead to class +information similar to `java.lang.Map[java.lang.Object, java.lang.Object]`. + +| Class | Data Type | +|:----------------------------|:------------------------------------| +| `java.lang.String` | `STRING` | +| `java.lang.Boolean` | `BOOLEAN` | +| `boolean` | `BOOLEAN NOT NULL` | +| `java.lang.Byte` | `TINYINT` | +| `byte` | `TINYINT NOT NULL` | +| `java.lang.Short` | `SMALLINT` | +| `short` | `SMALLINT NOT NULL` | +| `java.lang.Integer` | `INT` | +| `int` | `INT NOT NULL` | +| `java.lang.Long` | `BIGINT` | +| `long` | `BIGINT NOT NULL` | +| `java.lang.Float` | `FLOAT` | +| `float` | `FLOAT NOT NULL` | +| `java.lang.Double` | `DOUBLE` | +| `double` | `DOUBLE NOT NULL` | +| `java.sql.Date` | `DATE` | +| `java.time.LocalDate` | `DATE` | +| `java.sql.Time` | `TIME(0)` | +| `java.time.LocalTime` | `TIME(9)` | +| `java.sql.Timestamp` | `TIMESTAMP(9)` | +| `java.time.LocalDateTime` | `TIMESTAMP(9)` | +| `java.time.OffsetDateTime` | `TIMESTAMP(9) WITH TIME ZONE` | +| `java.time.Instant` | `TIMESTAMP(9) WITH LOCAL TIME ZONE` | +| `java.time.Duration` | `INVERVAL SECOND(9)` | +| `java.time.Period` | `INTERVAL YEAR(4) TO MONTH` | +| `byte[]` | `BYTES` | +| `T[]` | `ARRAY` | +| `java.lang.Map -The raw type is an extension to the SQL standard. ++{% highlight java %} +import org.apache.flink.table.annotation.DataTypeHint; -**Declaration** +class User { -{% top %} diff --git a/docs/dev/table/types.zh.md b/docs/dev/table/types.zh.md index 252ff021bf1f4c..819abc83af92c2 100644 --- a/docs/dev/table/types.zh.md +++ b/docs/dev/table/types.zh.md @@ -28,7 +28,7 @@ under the License. 从 Flink 1.9 开始，Table & SQL API 开始启用一种新的类型系统作为长期解决方案，用来保持 API 稳定性和 SQL 标准的兼容性。 -重新设计类型系统是一项涉及几乎所有的面向用户接口的重大工作。因此，它的引入跨越多个版本，社区的目标是在 Flink 1.10 完成这项工作。 +重新设计类型系统是一项涉及几乎所有的面向用户接口的重大工作。因此，它的引入跨越多个版本，社区的目标是在 Flink 1.12 完成这项工作。 同时由于为 Table 编程添加了新的 Planner 详见（[FLINK-11439](https://issues.apache.org/jira/browse/FLINK-11439)）, 并不是每种 Planner 都支持所有的数据类型。此外,Planner 对于数据类型的精度和参数化支持也可能是不完整的。 @@ -112,7 +112,7 @@ DataType t = DataTypes.ARRAY(DataTypes.INT().notNull()).bridgedTo(int[].class); // 而是使用 java.sql.Timestamp val t: DataType = DataTypes.TIMESTAMP(3).bridgedTo(classOf[java.sql.Timestamp]); -// 告诉运行时不要产生或者消费装箱的整数数组 +// 告诉运行时不要产生或者消费装箱的整数数组 // 而是使用基本数据类型的整数数组 val t: DataType = DataTypes.ARRAY(DataTypes.INT().notNull()).bridgedTo(classOf[Array[Int]]); {% endhighlight %} @@ -184,23 +184,22 @@ Flink 1.9 之前引入的旧的 Planner 主要支持类型信息（Type Informat | `DOUBLE` | | | `DATE` | | | `TIME` | 支持的精度仅为 `0`。 | -| `TIMESTAMP` | 支持的精度仅为 `3`。 | -| `TIMESTAMP WITH LOCAL TIME ZONE` | 支持的精度仅为 `3`。 | +| `TIMESTAMP` | | +| `TIMESTAMP WITH LOCAL TIME ZONE` | | | `INTERVAL` | 仅支持 `MONTH` 和 `SECOND(3)` 区间。 | | `ARRAY` | | | `MULTISET` | | | `MAP` | | | `ROW` | | | `RAW` | | +| stuctured types | 暂只能在用户自定义函数里使用。 | 局限性 ----------- **Java 表达式字符串**：Table API 中的 Java 表达式字符串，例如 `table.select("field.cast(STRING)")`，尚未被更新到新的类型系统中，使用[旧的 Planner 章节](#旧的-planner)中声明的字符串来表示。 -**连接器描述符和 SQL 客户端**：描述符字符串的表示形式尚未更新到新的类型系统。使用在[连接到外部系统章节](./connect.html#type-strings)中声明的字符串表示。 - -**用户自定义函数**：用户自定义函数尚不能声明数据类型。 +**用户自定义函数**：用户自定义聚合函数尚不能声明数据类型，标量函数和表函数充分支持数据类型。 数据类型列表 ------------------ @@ -236,10 +235,11 @@ DataTypes.CHAR(n) **JVM 类型** -| Java 类型 | 输入 | 输出 | 备注 | -|:-------------------|:-----:|:------:|:------------------------| -|`java.lang.String` | X | X | *缺省* | -|`byte[]` | X | X | 假设使用 UTF-8 编码。 | +| Java 类型 | 输入 | 输出 | 备注 | +|:----------------------------------------|:-----:|:------:|:------------------------| +|`java.lang.String` | X | X | *缺省* | +|`byte[]` | X | X | 假设使用 UTF-8 编码。 | +|`org.apache.flink.table.data.StringData` | X | X | 内部数据结构。 | #### `VARCHAR` / `STRING` @@ -274,10 +274,11 @@ DataTypes.STRING() **JVM 类型** -| Java 类型 | 输入 | 输出 | 备注 | -|:-------------------|:-----:|:------:|:------------------------| -|`java.lang.String` | X | X | *缺省* | -|`byte[]` | X | X | 假设使用 UTF-8 编码。 | +| Java 类型 | 输入 | 输出 | 备注 | +|:----------------------------------------|:-----:|:------:|:------------------------| +|`java.lang.String` | X | X | *缺省* | +|`byte[]` | X | X | 假设使用 UTF-8 编码。 | +|`org.apache.flink.table.data.StringData` | X | X | 内部数据结构。 | ### 二进制字符串 @@ -389,9 +390,10 @@ DataTypes.DECIMAL(p, s) **JVM 类型** -| Java 类型 | 输入 | 输出 | 备注 | -|:----------------------|:-----:|:------:|:------------------------| -|`java.math.BigDecimal` | X | X | *缺省* | +| Java 类型 | 输入 | 输出 | 备注 | +|:-----------------------------------------|:-----:|:------:|:------------------------| +|`java.math.BigDecimal` | X | X | *缺省* | +|`org.apache.flink.table.data.DecimalData` | X | X | 内部数据结构。 | #### `TINYINT` @@ -690,12 +692,13 @@ DataTypes.TIMESTAMP(p) | Java 类型 | 输入 | 输出 | 备注 | |:-------------------------|:-----:|:------:|:----------------------------------------------------| -|`java.time.LocalDateTime` | X | X | *缺省* | -|`java.sql.Timestamp` | X | X | | +|`java.time.LocalDateTime` | X | X | *缺省* | +|`java.sql.Timestamp` | X | X | | +|`org.apache.flink.table.data.TimestampData` | X | X | 内部数据结构。 | #### `TIMESTAMP WITH TIME ZONE` -*带有*时区的时间戳数据类型，由 `year-month-day hour:minute:second[.fractional] zone` 组成，精度达到纳秒，范围从 `0000-01-01 00:00:00.000000000 +14:59` 到 +*带有*时区的时间戳数据类型，由 `year-month-day hour:minute:second[.fractional] zone` 组成，精度达到纳秒，范围从 `0000-01-01 00:00:00.000000000 +14:59` 到 `9999-12-31 23:59:59.999999999 -14:59`。 与 SQL 标准相比，不支持闰秒（`23:59:60` 和 `23:59:61`），语义上更接近于 `java.time.OffsetDateTime`。 @@ -732,7 +735,7 @@ DataTypes.TIMESTAMP_WITH_TIME_ZONE(p) #### `TIMESTAMP WITH LOCAL TIME ZONE` -*带有本地*时区的时间戳数据类型，由 `year-month-day hour:minute:second[.fractional] zone` 组成，精度达到纳秒，范围从 `0000-01-01 00:00:00.000000000 +14:59` 到 +*带有本地*时区的时间戳数据类型，由 `year-month-day hour:minute:second[.fractional] zone` 组成，精度达到纳秒，范围从 `0000-01-01 00:00:00.000000000 +14:59` 到 `9999-12-31 23:59:59.999999999 -14:59`。 不支持闰秒（`23:59:60` 和 `23:59:61`），语义上更接近于 `java.time.OffsetDateTime`。 @@ -764,13 +767,14 @@ DataTypes.TIMESTAMP_WITH_LOCAL_TIME_ZONE(p) **JVM 类型** -| Java 类型 |输入 |输出 |备注 | -|:-------------------|:-----:|:------:|:--------------------------------------------------| -|`java.time.Instant` | X | X | *缺省* | -|`java.lang.Integer` | X | X | 描述从 Epoch 算起的秒数。 | -|`int` | X | (X) | 描述从 Epoch 算起的秒数。+ // defines an INT data type with a default conversion class `java.lang.Integer` + public @DataTypeHint("INT") Object o; -+ +-{% highlight text %} -RAW('class', 'snapshot') + // defines a TIMESTAMP data type of millisecond precision with an explicit conversion class + public @DataTypeHint(value = "TIMESTAMP(3)", bridgedTo = java.sql.Timestamp.class) Object o; + + // enrich the extraction with forcing using a RAW type + public @DataTypeHint("RAW") Class modelClass; + + // defines that all occurrences of java.math.BigDecimal (also in nested fields) will be + // extracted as DECIMAL(12, 2) + public @DataTypeHint(defaultDecimalPrecision = 12, defaultDecimalScale = 2) AccountStatement stmt; + + // defines that whenever a type cannot be mapped to a data type, instead of throwing + // an exception, always treat it as a RAW type + public @DataTypeHint(allowRawGlobally = HintFlag.TRUE) ComplexModel model; +} {% endhighlight %}-++ // defines an INT data type with a default conversion class `java.lang.Integer` + @DataTypeHint("INT") + var o: AnyRef -The type can be declared using `RAW('class', 'snapshot')` where `class` is the originating class and -`snapshot` is the serialized `TypeSerializerSnapshot` in Base64 encoding. Usually, the type string is not -declared directly but is generated while persisting the type. + // defines a TIMESTAMP data type of millisecond precision with an explicit conversion class + @DataTypeHint(value = "TIMESTAMP(3)", bridgedTo = java.sql.Timestamp.class) + var o: AnyRef -In the API, the `RAW` type can be declared either by directly supplying a `Class` + `TypeSerializer` or -by passing `TypeInformation` and let the framework extract `Class` + `TypeSerializer` from there. + // enrich the extraction with forcing using a RAW type + @DataTypeHint("RAW") + var modelClass: Class[_] -**Bridging to JVM Types** + // defines that all occurrences of java.math.BigDecimal (also in nested fields) will be + // extracted as DECIMAL(12, 2) + @DataTypeHint(defaultDecimalPrecision = 12, defaultDecimalScale = 2) + var stmt: AccountStatement -| Java Type | Input | Output | Remarks | -|:------------------|:-----:|:------:|:-------------------------------------------| -|*class* | X | X | Originating class or subclasses (for input) or superclasses (for output). *Default* | -|`byte[]` | | X | | + // defines that whenever a type cannot be mapped to a data type, instead of throwing + // an exception, always treat it as a RAW type + @DataTypeHint(allowRawGlobally = HintFlag.TRUE) + var model: ComplexModel +} +{% endhighlight %} +{% highlight java %} -DataTypes.RAW(class, serializer) +import org.apache.flink.table.annotation.DataTypeHint -DataTypes.RAW(typeInfo) -{% endhighlight %} -+class User { -
仅当类型不可为空时才输出。 | -|`java.lang.Long` | X | X | 描述从 Epoch 算起的毫秒数。 | -|`long` | X | (X) | 描述从 Epoch 算起的毫秒数。
仅当类型不可为空时才输出。 | +| Java 类型 |输入 |输出 |备注 | +|:-------------------------------------------|:-----:|:------:|:--------------------------------------------------| +|`java.time.Instant` | X | X | *缺省* | +|`java.lang.Integer` | X | X | 描述从 Epoch 算起的秒数。 | +|`int` | X | (X) | 描述从 Epoch 算起的秒数。
仅当类型不可为空时才输出。 | +|`java.lang.Long` | X | X | 描述从 Epoch 算起的毫秒数。 | +|`long` | X | (X) | 描述从 Epoch 算起的毫秒数。
仅当类型不可为空时才输出 | +|`org.apache.flink.table.data.TimestampData` | X | X | 内部数据结构。 | #### `INTERVAL YEAR TO MONTH` @@ -922,9 +926,10 @@ DataTypes.ARRAY(t) **JVM 类型** -| Java 类型 | 输入 | 输出 | 备注 | -|:----------|:-----:|:------:|:----------------------------------| -|*t*`[]` | (X) | (X) | 依赖于子类型。 *缺省* | +| Java 类型 | 输入 | 输出 | 备注 | +|:---------------------------------------|:-----:|:------:|:----------------------------------| +|*t*`[]` | (X) | (X) | 依赖于子类型。 *缺省* | +|`org.apache.flink.table.data.ArrayData` | X | X | 内部数据结构。 | #### `MAP` @@ -956,10 +961,11 @@ DataTypes.MAP(kt, vt) **JVM 类型** -| Java 类型 | 输入 | 输出 | 备注 | -|:--------------------------------------|:-----:|:------:|:----------| -| `java.util.Map+ ++ +**JVM 类型** + +| Java 类型 | 输入 | 输出 | 备注 | +|:-------------------------------------|:-----:|:------:|:------------------------------------------------------| +|*类型* | X | X | 原始类或子类（用于输入）或超类（用于输出）*缺省* | +|`org.apache.flink.types.Row` | X | X | 代表一行数据的结构化类型。 | +|`org.apache.flink.table.data.RowData` | X | X | 内部数据结构。 | ### 其他数据类型 @@ -1071,6 +1165,44 @@ DataTypes.BOOLEAN() |`java.lang.Boolean` | X | X | *缺省* | |`boolean` | X | (X) | 仅当类型不可为空时才输出。 | +#### `RAW` + +任意序列化类型的数据类型。此类型对于 Flink Table 来讲是一个黑盒子，仅在跟外部交互时被反序列化。 + +Raw 类型是 SQL 标准的扩展。 + +**声明** + ++{% highlight java %} +class User { + + // extract fields automatically + public int age; + public String name; + + // enrich the extraction with precision information + public @DataTypeHint("DECIMAL(10, 2)") BigDecimal totalBalance; + + // enrich the extraction with forcing using RAW types + public @DataTypeHint("RAW") Class modelClass; +} + +DataTypes.of(User.class); +{% endhighlight %} ++ ++{% highlight scala %} +case class User( + + // extract fields automatically + age: Int, + name: String, + + // enrich the extraction with precision information + @DataTypeHint("DECIMAL(10, 2)") totalBalance: java.math.BigDecimal, + + // enrich the extraction with forcing using a RAW type + @DataTypeHint("RAW") modelClass: Class[_] +) + +DataTypes.of(classOf[User]) +{% endhighlight %} ++ ++ ++ +此类型用 `RAW('class', 'snapshot')` 声明，其中 `class` 是原始类，`snapshot` 是 Base64 编码的序列化的 `TypeSerializerSnapshot`。通常，类型字符串不是直接声明的，而是在持久化类型时生成的。 + +在 API 中，可以通过直接提供 `Class` + `TypeSerializer` 或通过传递 `TypeInformation` 并让框架从那里提取 `Class` + `TypeSerializer` 来声明 `RAW` 类型。 + +**JVM 类型** + +| Java 类型 | 输入 | 输出 | 备注 | +|:------------------------------------------|:-----:|:------:|:----------------------------------------------------| +|*类型* | X | X | 原始类或子类（用于输入）或超类（用于输出）。 *缺省* | +|`byte[]` | | X | | +|`org.apache.flink.table.data.RawValueData` | X | X | 内部数据结构。 | + #### `NULL` 表示空类型 `NULL` 值的数据类型。 @@ -1106,41 +1238,111 @@ DataTypes.NULL() |`java.lang.Object` | X | X | *缺省* | |*任何类型* | | (X) | 任何非基本数据类型 | -#### `RAW` +数据类型注解 +--------------------- -任意序列化类型的数据类型。此类型是 Table 编程环境中的黑箱，仅在边缘反序列化。 +Flink API 经常尝试使用反射自动从类信息中提取数据类型，以避免重复的手动定义模式工作。然而以反射方式提取数据类型并不总是成功的，因为可能会丢失逻辑信息。因此，可能有必要在类或字段声明附近添加额外信息以支持提取逻辑。 + +下表列出了可以隐式映射到数据类型而无需额外信息的类： + +| 类 | 数据类型 | +|:----------------------------|:------------------------------------| +| `java.lang.String` | `STRING` | +| `java.lang.Boolean` | `BOOLEAN` | +| `boolean` | `BOOLEAN NOT NULL` | +| `java.lang.Byte` | `TINYINT` | +| `byte` | `TINYINT NOT NULL` | +| `java.lang.Short` | `SMALLINT` | +| `short` | `SMALLINT NOT NULL` | +| `java.lang.Integer` | `INT` | +| `int` | `INT NOT NULL` | +| `java.lang.Long` | `BIGINT` | +| `long` | `BIGINT NOT NULL` | +| `java.lang.Float` | `FLOAT` | +| `float` | `FLOAT NOT NULL` | +| `java.lang.Double` | `DOUBLE` | +| `double` | `DOUBLE NOT NULL` | +| `java.sql.Date` | `DATE` | +| `java.time.LocalDate` | `DATE` | +| `java.sql.Time` | `TIME(0)` | +| `java.time.LocalTime` | `TIME(9)` | +| `java.sql.Timestamp` | `TIMESTAMP(9)` | +| `java.time.LocalDateTime` | `TIMESTAMP(9)` | +| `java.time.OffsetDateTime` | `TIMESTAMP(9) WITH TIME ZONE` | +| `java.time.Instant` | `TIMESTAMP(9) WITH LOCAL TIME ZONE` | +| `java.time.Duration` | `INVERVAL SECOND(9)` | +| `java.time.Period` | `INTERVAL YEAR(4) TO MONTH` | +| `byte[]` | `BYTES` | +| `T[]` | `ARRAY+{% highlight text %} +RAW('class', 'snapshot') +{% endhighlight %} ++ ++{% highlight java %} +DataTypes.RAW(class, serializer) + +DataTypes.RAW(class) +{% endhighlight %} ++ +` | +| `java.lang.Map -**声明** ++{% highlight java %} +import org.apache.flink.table.annotation.DataTypeHint; -{% top %} diff --git a/docs/dev/types_serialization.zh.md b/docs/dev/types_serialization.zh.md index 72c70cc477039c..5be877f9e353a3 100644 --- a/docs/dev/types_serialization.zh.md +++ b/docs/dev/types_serialization.zh.md @@ -30,6 +30,206 @@ Apache Flink 以其独特的方式来处理数据类型以及序列化，这种 * This will be replaced by the TOC {:toc} +## Supported Data Types + +Flink places some restrictions on the type of elements that can be in a DataSet or DataStream. +The reason for this is that the system analyzes the types to determine +efficient execution strategies. + +There are seven different categories of data types: + +1. **Java Tuples** and **Scala Case Classes** +2. **Java POJOs** +3. **Primitive Types** +4. **Regular Classes** +5. **Values** +6. **Hadoop Writables** +7. **Special Types** + +#### Tuples and Case Classes + ++class User { -+ +-{% highlight text %} -RAW('class', 'snapshot') + // defines an INT data type with a default conversion class `java.lang.Integer` + public @DataTypeHint("INT") Object o; + + // defines a TIMESTAMP data type of millisecond precision with an explicit conversion class + public @DataTypeHint(value = "TIMESTAMP(3)", bridgedTo = java.sql.Timestamp.class) Object o; + + // enrich the extraction with forcing using a RAW type + public @DataTypeHint("RAW") Class modelClass; + + // defines that all occurrences of java.math.BigDecimal (also in nested fields) will be + // extracted as DECIMAL(12, 2) + public @DataTypeHint(defaultDecimalPrecision = 12, defaultDecimalScale = 2) AccountStatement stmt; + + // defines that whenever a type cannot be mapped to a data type, instead of throwing + // an exception, always treat it as a RAW type + public @DataTypeHint(allowRawGlobally = HintFlag.TRUE) ComplexModel model; +} {% endhighlight %}-++ // defines an INT data type with a default conversion class `java.lang.Integer` + @DataTypeHint("INT") + var o: AnyRef -此类型用 `RAW('class', 'snapshot')` 声明，其中 `class` 是原始类，`snapshot` 是 Base64 编码的序列化的 `TypeSerializerSnapshot`。通常，类型字符串不是直接声明的，而是在保留类型时生成的。 + // defines a TIMESTAMP data type of millisecond precision with an explicit conversion class + @DataTypeHint(value = "TIMESTAMP(3)", bridgedTo = java.sql.Timestamp.class) + var o: AnyRef -在 API 中，可以通过直接提供 `Class` + `TypeSerializer` 或通过传递 `TypeInformation` 并让框架从那里提取 `Class` + `TypeSerializer` 来声明 `RAW` 类型。 + // enrich the extraction with forcing using a RAW type + @DataTypeHint("RAW") + var modelClass: Class[_] -**JVM 类型** + // defines that all occurrences of java.math.BigDecimal (also in nested fields) will be + // extracted as DECIMAL(12, 2) + @DataTypeHint(defaultDecimalPrecision = 12, defaultDecimalScale = 2) + var stmt: AccountStatement -| Java 类型 | 输入 | 输出 | 备注 | -|:------------------|:-----:|:------:|:-------------------------------------------| -|*类型* | X | X | 原始类或子类（用于输入）或超类（用于输出）。 *缺省* | -|`byte[]` | | X | | + // defines that whenever a type cannot be mapped to a data type, instead of throwing + // an exception, always treat it as a RAW type + @DataTypeHint(allowRawGlobally = HintFlag.TRUE) + var model: ComplexModel +} +{% endhighlight %} +{% highlight java %} -DataTypes.RAW(class, serializer) +import org.apache.flink.table.annotation.DataTypeHint -DataTypes.RAW(typeInfo) -{% endhighlight %} -+class User { -++ +#### POJOs + +Java and Scala classes are treated by Flink as a special POJO data type if they fulfill the following requirements: + +- The class must be public. + +- It must have a public constructor without arguments (default constructor). + +- All fields are either public or must be accessible through getter and setter functions. For a field called `foo` the getter and setter methods must be named `getFoo()` and `setFoo()`. + +- The type of a field must be supported by a registered serializer. + +POJOs are generally represented with a `PojoTypeInfo` and serialized with the `PojoSerializer` (using [Kryo](https://github.com/EsotericSoftware/kryo) as configurable fallback). +The exception is when the POJOs are actually Avro types (Avro Specific Records) or produced as "Avro Reflect Types". +In that case the POJO's are represented by an `AvroTypeInfo` and serialized with the `AvroSerializer`. +You can also register your own custom serializer if required; see [Serialization](https://ci.apache.org/projects/flink/flink-docs-stable/dev/types_serialization.html#serialization-of-pojo-types) for further information. + +Flink analyzes the structure of POJO types, i.e., it learns about the fields of a POJO. As a result POJO types are easier to use than general types. Moreover, Flink can process POJOs more efficiently than general types. + +The following example shows a simple POJO with two public fields. + ++ +Tuples are composite types that contain a fixed number of fields with various types. +The Java API provides classes from `Tuple1` up to `Tuple25`. Every field of a tuple +can be an arbitrary Flink type including further tuples, resulting in nested tuples. Fields of a +tuple can be accessed directly using the field's name as `tuple.f4`, or using the generic getter method +`tuple.getField(int position)`. The field indices start at 0. Note that this stands in contrast +to the Scala tuples, but it is more consistent with Java's general indexing. + +{% highlight java %} +DataStream++ +Scala case classes (and Scala tuples which are a special case of case classes), are composite types that contain a fixed number of fields with various types. Tuple fields are addressed by their 1-offset names such as `_1` for the first field. Case class fields are accessed by their name. + +{% highlight scala %} +case class WordCount(word: String, count: Int) +val input = env.fromElements( + WordCount("hello", 1), + WordCount("world", 2)) // Case Class Data Set + +input.keyBy("word")// key by field expression "word" + +val input2 = env.fromElements(("hello", 1), ("world", 2)) // Tuple2 Data Set + +input2.keyBy(0, 1) // key by field positions 0 and 1 +{% endhighlight %} + ++++ +#### Primitive Types + +Flink supports all Java and Scala primitive types such as `Integer`, `String`, and `Double`. + +#### General Class Types + +Flink supports most Java and Scala classes (API and custom). +Restrictions apply to classes containing fields that cannot be serialized, like file pointers, I/O streams, or other native +resources. Classes that follow the Java Beans conventions work well in general. + +All classes that are not identified as POJO types (see POJO requirements above) are handled by Flink as general class types. +Flink treats these data types as black boxes and is not able to access their content (e.g., for efficient sorting). General types are de/serialized using the serialization framework [Kryo](https://github.com/EsotericSoftware/kryo). + +#### Values + +*Value* types describe their serialization and deserialization manually. Instead of going through a +general purpose serialization framework, they provide custom code for those operations by means of +implementing the `org.apache.flinktypes.Value` interface with the methods `read` and `write`. Using +a Value type is reasonable when general purpose serialization would be highly inefficient. An +example would be a data type that implements a sparse vector of elements as an array. Knowing that +the array is mostly zero, one can use a special encoding for the non-zero elements, while the +general purpose serialization would simply write all array elements. + +The `org.apache.flinktypes.CopyableValue` interface supports manual internal cloning logic in a +similar way. + +Flink comes with pre-defined Value types that correspond to basic data types. (`ByteValue`, +`ShortValue`, `IntValue`, `LongValue`, `FloatValue`, `DoubleValue`, `StringValue`, `CharValue`, +`BooleanValue`). These Value types act as mutable variants of the basic data types: Their value can +be altered, allowing programmers to reuse objects and take pressure off the garbage collector. + + +#### Hadoop Writables + +You can use types that implement the `org.apache.hadoop.Writable` interface. The serialization logic +defined in the `write()`and `readFields()` methods will be used for serialization. + +#### Special Types + +You can use special types, including Scala's `Either`, `Option`, and `Try`. +The Java API has its own custom implementation of `Either`. +Similarly to Scala's `Either`, it represents a value of two possible types, *Left* or *Right*. +`Either` can be useful for error handling or operators that need to output two different types of records. + +#### Type Erasure & Type Inference + +*Note: This Section is only relevant for Java.* + +The Java compiler throws away much of the generic type information after compilation. This is +known as *type erasure* in Java. It means that at runtime, an instance of an object does not know +its generic type any more. For example, instances of `DataStream+{% highlight java %} +public class WordWithCount { + + public String word; + public int count; + + public WordWithCount() {} + + public WordWithCount(String word, int count) { + this.word = word; + this.count = count; + } +} + +DataStream+wordCounts = env.fromElements( + new WordWithCount("hello", 1), + new WordWithCount("world", 2)); + +wordCounts.keyBy("word"); // key by field expression "word" + +{% endhighlight %} + +{% highlight scala %} +class WordWithCount(var word: String, var count: Int) { + def this() { + this(null, -1) + } +} + +val input = env.fromElements( + new WordWithCount("hello", 1), + new WordWithCount("world", 2)) // Case Class Data Set + +input.keyBy("word")// key by field expression "word" + +{% endhighlight %} ++` and `DataStream ` look the +same to the JVM. + +Flink requires type information at the time when it prepares the program for execution (when the +main method of the program is called). The Flink Java API tries to reconstruct the type information +that was thrown away in various ways and store it explicitly in the data sets and operators. You can +retrieve the type via `DataStream.getType()`. The method returns an instance of `TypeInformation`, +which is Flink's internal way of representing types. + +The type inference has its limits and needs the "cooperation" of the programmer in some cases. +Examples for that are methods that create data sets from collections, such as +`ExecutionEnvironment.fromCollection(),` where you can pass an argument that describes the type. But +also generic functions like `MapFunction ++ +Rich functions provide, in addition to the user-defined function (map, +reduce, etc), four methods: `open`, `close`, `getRuntimeContext`, and +`setRuntimeContext`. These are useful for parameterizing the function +(see [Passing Parameters to Functions]({{ site.baseurl }}/dev/batch/index.html#passing-parameters-to-functions)), +creating and finalizing local state, accessing broadcast variables (see +[Broadcast Variables]({{ site.baseurl }}/dev/batch/index.html#broadcast-variables)), and for accessing runtime +information such as accumulators and counters (see +[Accumulators and Counters](#accumulators--counters)), and information +on iterations (see [Iterations]({{ site.baseurl }}/dev/batch/iterations.html)). + +{% top %} + +## Accumulators & Counters + +Accumulators are simple constructs with an **add operation** and a **final accumulated result**, +which is available after the job ended. + +The most straightforward accumulator is a **counter**: You can increment it using the +```Accumulator.add(V value)``` method. At the end of the job Flink will sum up (merge) all partial +results and send the result to the client. Accumulators are useful during debugging or if you +quickly want to find out more about your data. + +Flink currently has the following **built-in accumulators**. Each of them implements the +{% gh_link /flink-core/src/main/java/org/apache/flink/api/common/accumulators/Accumulator.java "Accumulator" %} +interface. + +- {% gh_link /flink-core/src/main/java/org/apache/flink/api/common/accumulators/IntCounter.java "__IntCounter__" %}, + {% gh_link /flink-core/src/main/java/org/apache/flink/api/common/accumulators/LongCounter.java "__LongCounter__" %} + and {% gh_link /flink-core/src/main/java/org/apache/flink/api/common/accumulators/DoubleCounter.java "__DoubleCounter__" %}: + See below for an example using a counter. +- {% gh_link /flink-core/src/main/java/org/apache/flink/api/common/accumulators/Histogram.java "__Histogram__" %}: + A histogram implementation for a discrete number of bins. Internally it is just a map from Integer + to Integer. You can use this to compute distributions of values, e.g. the distribution of + words-per-line for a word count program. + +__How to use accumulators:__ + +First you have to create an accumulator object (here a counter) in the user-defined transformation +function where you want to use it. + +{% highlight java %} +private IntCounter numLines = new IntCounter(); +{% endhighlight %} + +Second you have to register the accumulator object, typically in the ```open()``` method of the +*rich* function. Here you also define the name. + +{% highlight java %} +getRuntimeContext().addAccumulator("num-lines", this.numLines); +{% endhighlight %} + +You can now use the accumulator anywhere in the operator function, including in the ```open()``` and +```close()``` methods. + +{% highlight java %} +this.numLines.add(1); +{% endhighlight %} + +The overall result will be stored in the ```JobExecutionResult``` object which is +returned from the `execute()` method of the execution environment +(currently this only works if the execution waits for the +completion of the job). + +{% highlight java %} +myJobExecutionResult.getAccumulatorResult("num-lines") +{% endhighlight %} + +All accumulators share a single namespace per job. Thus you can use the same accumulator in +different operator functions of your job. Flink will internally merge all accumulators with the same +name. + +A note on accumulators and iterations: Currently the result of accumulators is only available after +the overall job has ended. We plan to also make the result of the previous iteration available in the +next iteration. You can use +{% gh_link /flink-java/src/main/java/org/apache/flink/api/java/operators/IterativeDataSet.java#L98 "Aggregators" %} +to compute per-iteration statistics and base the termination of iterations on such statistics. + +__Custom accumulators:__ + +To implement your own accumulator you simply have to write your implementation of the Accumulator +interface. Feel free to create a pull request if you think your custom accumulator should be shipped +with Flink. + +You have the choice to implement either +{% gh_link /flink-core/src/main/java/org/apache/flink/api/common/accumulators/Accumulator.java "Accumulator" %} +or {% gh_link /flink-core/src/main/java/org/apache/flink/api/common/accumulators/SimpleAccumulator.java "SimpleAccumulator" %}. + +```Accumulator+ +## Implementing an interface + +The most basic way is to implement one of the provided interfaces: + +{% highlight java %} +class MyMapFunction implements MapFunction++ + +## Lambda Functions + +As already seen in previous examples all operations accept lambda functions for describing +the operation: +{% highlight scala %} +val data: DataSet[String] = // [...] +data.filter { _.startsWith("http://") } +{% endhighlight %} + +{% highlight scala %} +val data: DataSet[Int] = // [...] +data.reduce { (i1,i2) => i1 + i2 } +// or +data.reduce { _ + _ } +{% endhighlight %} + +## Rich functions + +All transformations that take as argument a lambda function can +instead take as argument a *rich* function. For example, instead of + +{% highlight scala %} +data.map { x => x.toInt } +{% endhighlight %} + +you can write + +{% highlight scala %} +class MyMapFunction extends RichMapFunction[String, Int] { + def map(in: String):Int = { in.toInt } +}; +{% endhighlight %} + +and pass the function to a `map` transformation: + +{% highlight scala %} +data.map(new MyMapFunction()) +{% endhighlight %} + +Rich functions can also be defined as an anonymous class: +{% highlight scala %} +data.map (new RichMapFunction[String, Int] { + def map(in: String):Int = { in.toInt } +}) +{% endhighlight %} ++ ++ + diff --git a/docs/fig/source_reader.svg b/docs/fig/source_reader.svg new file mode 100644 index 00000000000000..1d0f3635b3b263 --- /dev/null +++ b/docs/fig/source_reader.svg @@ -0,0 +1,20 @@ + + + diff --git a/docs/fig/spend-report-console.png b/docs/fig/spend-report-console.png new file mode 100644 index 00000000000000..ade38f3f2647a7 Binary files /dev/null and b/docs/fig/spend-report-console.png differ diff --git a/docs/fig/spend-report-grafana.png b/docs/fig/spend-report-grafana.png new file mode 100644 index 00000000000000..8cca47f8d37d61 Binary files /dev/null and b/docs/fig/spend-report-grafana.png differ diff --git a/docs/fig/stream_aligning.svg b/docs/fig/stream_aligning.svg index de2543597ceb06..2192117b78d152 100644 --- a/docs/fig/stream_aligning.svg +++ b/docs/fig/stream_aligning.svg @@ -1,4 +1,4 @@ - + - \ No newline at end of file diff --git a/docs/fig/stream_unaligning.svg b/docs/fig/stream_unaligning.svg new file mode 100644 index 00000000000000..8a8b26f313ae07 --- /dev/null +++ b/docs/fig/stream_unaligning.svg @@ -0,0 +1,383 @@ + + + + + \ No newline at end of file diff --git a/docs/fig/table_connectors.svg b/docs/fig/table_connectors.svg new file mode 100644 index 00000000000000..f41d0cb9cfc30c --- /dev/null +++ b/docs/fig/table_connectors.svg @@ -0,0 +1,666 @@ + + + diff --git a/docs/flinkDev/ide_setup.md b/docs/flinkDev/ide_setup.md index d944bc688f0864..e715a50c7bfcd7 100644 --- a/docs/flinkDev/ide_setup.md +++ b/docs/flinkDev/ide_setup.md @@ -27,8 +27,8 @@ under the License. The sections below describe how to import the Flink project into an IDE for the development of Flink itself. For writing Flink programs, please -refer to the [Java API]({{ site.baseurl }}/dev/projectsetup/java_api_quickstart.html) -and the [Scala API]({{ site.baseurl }}/dev/projectsetup/scala_api_quickstart.html) +refer to the [Java API]({{ site.baseurl }}/dev/project-configuration.html) +and the [Scala API]({{ site.baseurl }}/dev/project-configuration) quickstart guides. **NOTE:** Whenever something is not working in your IDE, try with the Maven diff --git a/docs/flinkDev/ide_setup.zh.md b/docs/flinkDev/ide_setup.zh.md index 170f26cdf945ab..bdfbc0253ca119 100644 --- a/docs/flinkDev/ide_setup.zh.md +++ b/docs/flinkDev/ide_setup.zh.md @@ -27,8 +27,8 @@ under the License. The sections below describe how to import the Flink project into an IDE for the development of Flink itself. For writing Flink programs, please -refer to the [Java API]({{ site.baseurl }}/dev/projectsetup/java_api_quickstart.html) -and the [Scala API]({{ site.baseurl }}/dev/projectsetup/scala_api_quickstart.html) +refer to the [Java API]({{ site.baseurl }}/dev/project-configuration.html) +and the [Scala API]({{ site.baseurl }}/dev/project-configuration) quickstart guides. **NOTE:** Whenever something is not working in your IDE, try with the Maven diff --git a/docs/getting-started/docker-playgrounds/flink-operations-playground.md b/docs/getting-started/docker-playgrounds/flink-operations-playground.md deleted file mode 100644 index 1e2a569051f2bf..00000000000000 --- a/docs/getting-started/docker-playgrounds/flink-operations-playground.md +++ /dev/null @@ -1,828 +0,0 @@ ---- -title: "Flink Operations Playground" -nav-title: 'Flink Operations Playground' -nav-parent_id: docker-playgrounds -nav-pos: 1 ---- - - -There are many ways to deploy and operate Apache Flink in various environments. Regardless of this -variety, the fundamental building blocks of a Flink Cluster remain the same, and similar -operational principles apply. - -In this playground, you will learn how to manage and run Flink Jobs. You will see how to deploy and -monitor an application, experience how Flink recovers from Job failure, and perform everyday -operational tasks like upgrades and rescaling. - -{% if site.version contains "SNAPSHOT" %} - - - NOTE: The Apache Flink Docker images used for this playground are only available for - released versions of Apache Flink. -
-{% endif %} - -* This will be replaced by the TOC -{:toc} - -## Anatomy of this Playground - -This playground consists of a long living -[Flink Session Cluster]({{ site.baseurl }}/concepts/glossary.html#flink-session-cluster) and a Kafka -Cluster. - -A Flink Cluster always consists of a -[Flink Master]({{ site.baseurl }}/concepts/glossary.html#flink-master) and one or more -[Flink TaskManagers]({{ site.baseurl }}/concepts/glossary.html#flink-taskmanager). The Flink Master -is responsible for handling [Job]({{ site.baseurl }}/concepts/glossary.html#flink-job) submissions, -the supervision of Jobs as well as resource management. The Flink TaskManagers are the worker -processes and are responsible for the execution of the actual -[Tasks]({{ site.baseurl }}/concepts/glossary.html#task) which make up a Flink Job. In this -playground you will start with a single TaskManager, but scale out to more TaskManagers later. -Additionally, this playground comes with a dedicated *client* container, which we use to submit the -Flink Job initially and to perform various operational tasks later on. The *client* container is not -needed by the Flink Cluster itself but only included for ease of use. - -The Kafka Cluster consists of a Zookeeper server and a Kafka Broker. - -
- Since you are currently looking at the latest SNAPSHOT - version of the documentation, all version references below will not work. - Please switch the documentation to the latest released version via the release picker which you - find on the left side below the menu. -
-
-When the playground is started a Flink Job called *Flink Event Count* will be submitted to the
-Flink Master. Additionally, two Kafka Topics *input* and *output* are created.
-
-
-
-The Job consumes `ClickEvent`s from the *input* topic, each with a `timestamp` and a `page`. The
-events are then keyed by `page` and counted in 15 second
-[windows]({{ site.baseurl }}/dev/stream/operators/windows.html). The results are written to the
-*output* topic.
-
-There are six different pages and we generate 1000 click events per page and 15 seconds. Hence, the
-output of the Flink job should show 1000 views per page and window.
-
-{% top %}
-
-## Starting the Playground
-
-The playground environment is set up in just a few steps. We will walk you through the necessary
-commands and show how to validate that everything is running correctly.
-
-We assume that you have [Docker](https://docs.docker.com/) (1.12+) and
-[docker-compose](https://docs.docker.com/compose/) (2.1+) installed on your machine.
-
-The required configuration files are available in the
-[flink-playgrounds](https://github.com/apache/flink-playgrounds) repository. Check it out and spin
-up the environment:
-
-{% highlight bash %}
-git clone --branch release-{{ site.version_title }} https://github.com/apache/flink-playgrounds.git
-cd flink-playgrounds/operations-playground
-docker-compose build
-docker-compose up -d
-{% endhighlight %}
-
-Afterwards, you can inspect the running Docker containers with the following command:
-
-{% highlight bash %}
-docker-compose ps
-
- Name Command State Ports
------------------------------------------------------------------------------------------------------------------------------
-operations-playground_clickevent-generator_1 /docker-entrypoint.sh java ... Up 6123/tcp, 8081/tcp
-operations-playground_client_1 /docker-entrypoint.sh flin ... Exit 0
-operations-playground_jobmanager_1 /docker-entrypoint.sh jobm ... Up 6123/tcp, 0.0.0.0:8081->8081/tcp
-operations-playground_kafka_1 start-kafka.sh Up 0.0.0.0:9094->9094/tcp
-operations-playground_taskmanager_1 /docker-entrypoint.sh task ... Up 6123/tcp, 8081/tcp
-operations-playground_zookeeper_1 /bin/sh -c /usr/sbin/sshd ... Up 2181/tcp, 22/tcp, 2888/tcp, 3888/tcp
-{% endhighlight %}
-
-This indicates that the client container has successfully submitted the Flink Job (`Exit 0`) and all
-cluster components as well as the data generator are running (`Up`).
-
-You can stop the playground environment by calling:
-
-{% highlight bash %}
-docker-compose down -v
-{% endhighlight %}
-
-## Entering the Playground
-
-There are many things you can try and check out in this playground. In the following two sections we
-will show you how to interact with the Flink Cluster and demonstrate some of Flink's key features.
-
-### Flink WebUI
-
-The most natural starting point to observe your Flink Cluster is the WebUI exposed under
-[http://localhost:8081](http://localhost:8081). If everything went well, you'll see that the cluster initially consists of
-one TaskManager and executes a Job called *Click Event Count*.
-
-
-
-The Flink WebUI contains a lot of useful and interesting information about your Flink Cluster and
-its Jobs (JobGraph, Metrics, Checkpointing Statistics, TaskManager Status,...).
-
-### Logs
-
-**JobManager**
-
-The JobManager logs can be tailed via `docker-compose`.
-
-{% highlight bash %}
-docker-compose logs -f jobmanager
-{% endhighlight %}
-
-After the initial startup you should mainly see log messages for every checkpoint completion.
-
-**TaskManager**
-
-The TaskManager log can be tailed in the same way.
-{% highlight bash %}
-docker-compose logs -f taskmanager
-{% endhighlight %}
-
-After the initial startup you should mainly see log messages for every checkpoint completion.
-
-### Flink CLI
-
-The [Flink CLI]({{ site.baseurl }}/ops/cli.html) can be used from within the client container. For
-example, to print the `help` message of the Flink CLI you can run
-{% highlight bash%}
-docker-compose run --no-deps client flink --help
-{% endhighlight %}
-
-### Flink REST API
-
-The [Flink REST API]({{ site.baseurl }}/monitoring/rest_api.html#api) is exposed via
-`localhost:8081` on the host or via `jobmanager:8081` from the client container, e.g. to list all
-currently running jobs, you can run:
-{% highlight bash%}
-curl localhost:8081/jobs
-{% endhighlight %}
-
-{% if site.version contains "SNAPSHOT" %}
-- Note: If the curl command is not available on your machine, you can run it from the client - container (similar to the Flink CLI): -{% highlight bash%} -docker-compose run --no-deps client curl jobmanager:8081/jobs -{% endhighlight %} -
-{% endif %} - -### Kafka Topics - -You can look at the records that are written to the Kafka Topics by running -{% highlight bash%} -//input topic (1000 records/s) -docker-compose exec kafka kafka-console-consumer.sh \ - --bootstrap-server localhost:9092 --topic input - -//output topic (24 records/min) -docker-compose exec kafka kafka-console-consumer.sh \ - --bootstrap-server localhost:9092 --topic output -{% endhighlight %} - -{% top %} - -## Time to Play! - -Now that you learned how to interact with Flink and the Docker containers, let's have a look at -some common operational tasks that you can try out on our playground. -All of these tasks are independent of each other, i.e. you can perform them in any order. -Most tasks can be executed via the [CLI](#flink-cli) and the [REST API](#flink-rest-api). - -### Listing Running Jobs - --- -The JobID is assigned to a Job upon submission and is needed to perform actions on the Job via the -CLI or REST API. - -### Observing Failure & Recovery - -Flink provides exactly-once processing guarantees under (partial) failure. In this playground you -can observe and - to some extent - verify this behavior. - -#### Step 1: Observing the Output - -As described [above](#anatomy-of-this-playground), the events in this playground are generate such -that each window contains exactly one thousand records. So, in order to verify that Flink -successfully recovers from a TaskManager failure without data loss or duplication you can tail the -output topic and check that - after recovery - all windows are present and the count is correct. - -For this, start reading from the *output* topic and leave this command running until after -recovery (Step 3). - -{% highlight bash%} -docker-compose exec kafka kafka-console-consumer.sh \ - --bootstrap-server localhost:9092 --topic output -{% endhighlight %} - -#### Step 2: Introducing a Fault - -In order to simulate a partial failure you can kill a TaskManager. In a production setup, this -could correspond to a loss of the TaskManager process, the TaskManager machine or simply a transient -exception being thrown from the framework or user code (e.g. due to the temporary unavailability of -an external resource). - -{% highlight bash%} -docker-compose kill taskmanager -{% endhighlight %} - -After a few seconds, the Flink Master will notice the loss of the TaskManager, cancel the affected Job, and -immediately resubmit it for recovery. -When the Job gets restarted, its tasks remain in the `SCHEDULED` state, which is indicated by the -purple colored squares (see screenshot below). - --**Command** -{% highlight bash %} -docker-compose run --no-deps client flink list -{% endhighlight %} -**Expected Output** -{% highlight plain %} -Waiting for response... ------------------- Running/Restarting Jobs ------------------- -16.07.2019 16:37:55 :-: Click Event Count (RUNNING) --------------------------------------------------------------- -No scheduled jobs. -{% endhighlight %} - -**Request** -{% highlight bash %} -curl localhost:8081/jobs -{% endhighlight %} -**Expected Response (pretty-printed)** -{% highlight bash %} -{ - "jobs": [ - { - "id": "-", - "status": "RUNNING" - } - ] -} -{% endhighlight %} - 
-
-- Note: Even though the tasks of the job are in SCHEDULED state and not RUNNING yet, the overall - status of a Job is shown as RUNNING. -
- -At this point, the tasks of the Job cannot move from the `SCHEDULED` state to `RUNNING` because there -are no resources (TaskSlots provided by TaskManagers) to the run the tasks. -Until a new TaskManager becomes available, the Job will go through a cycle of cancellations and resubmissions. - -In the meantime, the data generator keeps pushing `ClickEvent`s into the *input* topic. This is -similar to a real production setup where data is produced while the Job to process it is down. - -#### Step 3: Recovery - -Once you restart the TaskManager, it reconnects to the Master. - -{% highlight bash%} -docker-compose up -d taskmanager -{% endhighlight %} - -When the Master is notified about the new TaskManager, it schedules the tasks of the -recovering Job to the newly available TaskSlots. Upon restart, the tasks recover their state from -the last successful [checkpoint]({{ site.baseurl }}/internals/stream_checkpointing.html) that was taken -before the failure and switch to the `RUNNING` state. - -The Job will quickly process the full backlog of input events (accumulated during the outage) -from Kafka and produce output at a much higher rate (> 24 records/minute) until it reaches -the head of the stream. In the *output* you will see that all keys (`page`s) are present for all time -windows and that every count is exactly one thousand. Since we are using the -[FlinkKafkaProducer]({{ site.baseurl }}/dev/connectors/kafka.html#kafka-producers-and-fault-tolerance) -in its "at-least-once" mode, there is a chance that you will see some duplicate output records. - -- Note: Most production setups rely on a resource manager (Kubernetes, Yarn, Mesos) to - automatically restart failed processes. -
- -### Upgrading & Rescaling a Job - -Upgrading a Flink Job always involves two steps: First, the Flink Job is gracefully stopped with a -[Savepoint]({{ site.baseurl }}/ops/state/savepoints.html). A Savepoint is a consistent snapshot of -the complete application state at a well-defined, globally consistent point in time (similar to a -checkpoint). Second, the upgraded Flink Job is started from the Savepoint. In this context "upgrade" -can mean different things including the following: - -* An upgrade to the configuration (incl. the parallelism of the Job) -* An upgrade to the topology of the Job (added/removed Operators) -* An upgrade to the user-defined functions of the Job - -Before starting with the upgrade you might want to start tailing the *output* topic, in order to -observe that no data is lost or corrupted in the course the upgrade. - -{% highlight bash%} -docker-compose exec kafka kafka-console-consumer.sh \ - --bootstrap-server localhost:9092 --topic output -{% endhighlight %} - -#### Step 1: Stopping the Job - -To gracefully stop the Job, you need to use the "stop" command of either the CLI or the REST API. -For this you will need the JobID of the Job, which you can obtain by -[listing all running Jobs](#listing-running-jobs) or from the WebUI. With the JobID you can proceed -to stopping the Job: - --- -#### Step 2a: Restart Job without Changes - -You can now restart the upgraded Job from this Savepoint. For simplicity, you can start by -restarting it without any changes. - --**Command** -{% highlight bash %} -docker-compose run --no-deps client flink stop--{% endhighlight %} -**Expected Output** -{% highlight bash %} -Suspending job " " with a savepoint. -Suspended job " " with a savepoint. -{% endhighlight %} - -The Savepoint has been stored to the `state.savepoint.dir` configured in the *flink-conf.yaml*, -which is mounted under */tmp/flink-savepoints-directory/* on your local machine. You will need the -path to this Savepoint in the next step. In case of the REST API this path was already part of the -response, you will need to have a look at the filesystem directly. - -**Command** -{% highlight bash %} -ls -lia /tmp/flink-savepoints-directory -{% endhighlight %} - -**Expected Output** -{% highlight bash %} -total 0 - 17 drwxr-xr-x 3 root root 60 17 jul 17:05 . - 2 drwxrwxrwt 135 root root 3420 17 jul 17:09 .. -1002 drwxr-xr-x 2 root root 140 17 jul 17:05 savepoint- - -{% endhighlight %} - - - **Request** -{% highlight bash %} -# triggering stop -curl -X POST localhost:8081/jobs/-/stop -d '{"drain": false}' -{% endhighlight %} - -**Expected Response (pretty-printed)** -{% highlight json %} -{ - "request-id": " " -} -{% endhighlight %} - -**Request** -{% highlight bash %} -# check status of stop action and retrieve savepoint path - curl localhost:8081/jobs/ /savepoints/ -{% endhighlight %} - -**Expected Response (pretty-printed)** -{% highlight json %} -{ - "status": { - "id": "COMPLETED" - }, - "operation": { - "location": " " - } - -{% endhighlight %} - -- -Once the Job is `RUNNING` again, you will see in the *output* Topic that records are produced at a -higher rate while the Job is processing the backlog accumulated during the outage. Additionally, -you will see that no data was lost during the upgrade: all windows are present with a count of -exactly one thousand. - -#### Step 2b: Restart Job with a Different Parallelism (Rescaling) - -Alternatively, you could also rescale the Job from this Savepoint by passing a different parallelism -during resubmission. - --**Command** -{% highlight bash %} -docker-compose run --no-deps client flink run -s-\ - -d /opt/ClickCountJob.jar \ - --bootstrap.servers kafka:9092 --checkpointing --event-time -{% endhighlight %} -**Expected Output** -{% highlight bash %} -Starting execution of program -Job has been submitted with JobID -{% endhighlight %} - - -**Request** -{% highlight bash %} -# Uploading the JAR from the Client container -docker-compose run --no-deps client curl -X POST -H "Expect:" \ - -F "jarfile=@/opt/ClickCountJob.jar" http://jobmanager:8081/jars/upload -{% endhighlight %} - -**Expected Response (pretty-printed)** -{% highlight json %} -{ - "filename": "/tmp/flink-web--/flink-web-upload/ ", - "status": "success" -} - -{% endhighlight %} - -**Request** -{% highlight bash %} -# Submitting the Job -curl -X POST http://localhost:8081/jars/ /run \ - -d '{"programArgs": "--bootstrap.servers kafka:9092 --checkpointing --event-time", "savepointPath": " "}' -{% endhighlight %} -**Expected Response (pretty-printed)** -{% highlight json %} -{ - "jobid": " " -} -{% endhighlight %} - --Now, the Job has been resubmitted, but it will not start as there are not enough TaskSlots to -execute it with the increased parallelism (2 available, 3 needed). With -{% highlight bash %} -docker-compose scale taskmanager=2 -{% endhighlight %} -you can add a second TaskManager with two TaskSlots to the Flink Cluster, which will automatically register with the -Flink Master. Shortly after adding the TaskManager the Job should start running again. - -Once the Job is "RUNNING" again, you will see in the *output* Topic that now data was lost during -rescaling: all windows are present with a count of exactly one thousand. - -### Querying the Metrics of a Job - -The Flink Master exposes system and user [metrics]({{ site.baseurl }}/monitoring/metrics.html) -via its REST API. - -The endpoint depends on the scope of these metrics. Metrics scoped to a Job can be listed via -`jobs/-**Command** -{% highlight bash %} -docker-compose run --no-deps client flink run -p 3 -s-\ - -d /opt/ClickCountJob.jar \ - --bootstrap.servers kafka:9092 --checkpointing --event-time -{% endhighlight %} -**Expected Output** -{% highlight bash %} -Starting execution of program -Job has been submitted with JobID -{% endhighlight %} - - -**Request** -{% highlight bash %} -# Uploading the JAR from the Client container -docker-compose run --no-deps client curl -X POST -H "Expect:" \ - -F "jarfile=@/opt/ClickCountJob.jar" http://jobmanager:8081/jars/upload -{% endhighlight %} - -**Expected Response (pretty-printed)** -{% highlight json %} -{ - "filename": "/tmp/flink-web--/flink-web-upload/ ", - "status": "success" -} - -{% endhighlight %} - -**Request** -{% highlight bash %} -# Submitting the Job -curl -X POST http://localhost:8081/jars/ /run \ - -d '{"parallelism": 3, "programArgs": "--bootstrap.servers kafka:9092 --checkpointing --event-time", "savepointPath": " "}' -{% endhighlight %} -**Expected Response (pretty-printed** -{% highlight json %} -{ - "jobid": " " -} -{% endhighlight %} - /metrics`. The actual value of a metric can be queried via the `get` query parameter. - -**Request** -{% highlight bash %} -curl "localhost:8081/jobs/ /metrics?get=lastCheckpointSize" -{% endhighlight %} -**Expected Response (pretty-printed; no placeholders)** -{% highlight json %} -[ - { - "id": "lastCheckpointSize", - "value": "9378" - } -] -{% endhighlight %} - -The REST API can not only be used to query metrics, but you can also retrieve detailed information -about the status of a running Job. - -**Request** -{% highlight bash %} -# find the vertex-id of the vertex of interest -curl localhost:8081/jobs/ -{% endhighlight %} - -**Expected Response (pretty-printed)** -{% highlight json %} -{ - "jid": " ", - "name": "Click Event Count", - "isStoppable": false, - "state": "RUNNING", - "start-time": 1564467066026, - "end-time": -1, - "duration": 374793, - "now": 1564467440819, - "timestamps": { - "CREATED": 1564467066026, - "FINISHED": 0, - "SUSPENDED": 0, - "FAILING": 0, - "CANCELLING": 0, - "CANCELED": 0, - "RECONCILING": 0, - "RUNNING": 1564467066126, - "FAILED": 0, - "RESTARTING": 0 - }, - "vertices": [ - { - "id": " ", - "name": "ClickEvent Source", - "parallelism": 2, - "status": "RUNNING", - "start-time": 1564467066423, - "end-time": -1, - "duration": 374396, - "tasks": { - "CREATED": 0, - "FINISHED": 0, - "DEPLOYING": 0, - "RUNNING": 2, - "CANCELING": 0, - "FAILED": 0, - "CANCELED": 0, - "RECONCILING": 0, - "SCHEDULED": 0 - }, - "metrics": { - "read-bytes": 0, - "read-bytes-complete": true, - "write-bytes": 5033461, - "write-bytes-complete": true, - "read-records": 0, - "read-records-complete": true, - "write-records": 166351, - "write-records-complete": true - } - }, - { - "id": " ", - "name": "Timestamps/Watermarks", - "parallelism": 2, - "status": "RUNNING", - "start-time": 1564467066441, - "end-time": -1, - "duration": 374378, - "tasks": { - "CREATED": 0, - "FINISHED": 0, - "DEPLOYING": 0, - "RUNNING": 2, - "CANCELING": 0, - "FAILED": 0, - "CANCELED": 0, - "RECONCILING": 0, - "SCHEDULED": 0 - }, - "metrics": { - "read-bytes": 5066280, - "read-bytes-complete": true, - "write-bytes": 5033496, - "write-bytes-complete": true, - "read-records": 166349, - "read-records-complete": true, - "write-records": 166349, - "write-records-complete": true - } - }, - { - "id": " ", - "name": "ClickEvent Counter", - "parallelism": 2, - "status": "RUNNING", - "start-time": 1564467066469, - "end-time": -1, - "duration": 374350, - "tasks": { - "CREATED": 0, - "FINISHED": 0, - "DEPLOYING": 0, - "RUNNING": 2, - "CANCELING": 0, - "FAILED": 0, - "CANCELED": 0, - "RECONCILING": 0, - "SCHEDULED": 0 - }, - "metrics": { - "read-bytes": 5085332, - "read-bytes-complete": true, - "write-bytes": 316, - "write-bytes-complete": true, - "read-records": 166305, - "read-records-complete": true, - "write-records": 6, - "write-records-complete": true - } - }, - { - "id": " ", - "name": "ClickEventStatistics Sink", - "parallelism": 2, - "status": "RUNNING", - "start-time": 1564467066476, - "end-time": -1, - "duration": 374343, - "tasks": { - "CREATED": 0, - "FINISHED": 0, - "DEPLOYING": 0, - "RUNNING": 2, - "CANCELING": 0, - "FAILED": 0, - "CANCELED": 0, - "RECONCILING": 0, - "SCHEDULED": 0 - }, - "metrics": { - "read-bytes": 20668, - "read-bytes-complete": true, - "write-bytes": 0, - "write-bytes-complete": true, - "read-records": 6, - "read-records-complete": true, - "write-records": 0, - "write-records-complete": true - } - } - ], - "status-counts": { - "CREATED": 0, - "FINISHED": 0, - "DEPLOYING": 0, - "RUNNING": 4, - "CANCELING": 0, - "FAILED": 0, - "CANCELED": 0, - "RECONCILING": 0, - "SCHEDULED": 0 - }, - "plan": { - "jid": " ", - "name": "Click Event Count", - "nodes": [ - { - "id": " ", - "parallelism": 2, - "operator": "", - "operator_strategy": "", - "description": "ClickEventStatistics Sink", - "inputs": [ - { - "num": 0, - "id": " ", - "ship_strategy": "FORWARD", - "exchange": "pipelined_bounded" - } - ], - "optimizer_properties": {} - }, - { - "id": " ", - "parallelism": 2, - "operator": "", - "operator_strategy": "", - "description": "ClickEvent Counter", - "inputs": [ - { - "num": 0, - "id": " ", - "ship_strategy": "HASH", - "exchange": "pipelined_bounded" - } - ], - "optimizer_properties": {} - }, - { - "id": " ", - "parallelism": 2, - "operator": "", - "operator_strategy": "", - "description": "Timestamps/Watermarks", - "inputs": [ - { - "num": 0, - "id": " ", - "ship_strategy": "FORWARD", - "exchange": "pipelined_bounded" - } - ], - "optimizer_properties": {} - }, - { - "id": " ", - "parallelism": 2, - "operator": "", - "operator_strategy": "", - "description": "ClickEvent Source", - "optimizer_properties": {} - } - ] - } -} -{% endhighlight %} - -Please consult the [REST API reference]({{ site.baseurl }}/monitoring/rest_api.html#api) -for a complete list of possible queries including how to query metrics of different scopes (e.g. -TaskManager metrics); - -{% top %} - -## Variants - -You might have noticed that the *Click Event Count* application was always started with `--checkpointing` -and `--event-time` program arguments. By omitting these in the command of the *client* container in the -`docker-compose.yaml`, you can change the behavior of the Job. - -* `--checkpointing` enables [checkpoint]({{ site.baseurl }}/internals/stream_checkpointing.html), -which is Flink's fault-tolerance mechanism. If you run without it and go through -[failure and recovery](#observing-failure--recovery), you should will see that data is actually -lost. - -* `--event-time` enables [event time semantics]({{ site.baseurl }}/dev/event_time.html) for your -Job. When disabled, the Job will assign events to windows based on the wall-clock time instead of -the timestamp of the `ClickEvent`. Consequently, the number of events per window will not be exactly -one thousand anymore. - -The *Click Event Count* application also has another option, turned off by default, that you can -enable to explore the behavior of this job under backpressure. You can add this option in the -command of the *client* container in `docker-compose.yaml`. - -* `--backpressure` adds an additional operator into the middle of the job that causes severe backpressure -during even-numbered minutes (e.g., during 10:12, but not during 10:13). This can be observed by -inspecting various [network metrics]({{ site.baseurl }}/monitoring/metrics.html#default-shuffle-service) -such as `outputQueueLength` and `outPoolUsage`, and/or by using the -[backpressure monitoring]({{ site.baseurl }}/monitoring/back_pressure.html#monitoring-back-pressure) -available in the WebUI. diff --git a/docs/getting-started/docker-playgrounds/index.md b/docs/getting-started/docker-playgrounds/index.md deleted file mode 100644 index e3d5fd87a3a488..00000000000000 --- a/docs/getting-started/docker-playgrounds/index.md +++ /dev/null @@ -1,25 +0,0 @@ ---- -title: Docker Playgrounds -nav-id: docker-playgrounds -nav-title: 'Docker Playgrounds' -nav-parent_id: getting-started -nav-pos: 10 ---- - diff --git a/docs/getting-started/docker-playgrounds/index.zh.md b/docs/getting-started/docker-playgrounds/index.zh.md deleted file mode 100644 index e3d5fd87a3a488..00000000000000 --- a/docs/getting-started/docker-playgrounds/index.zh.md +++ /dev/null @@ -1,25 +0,0 @@ ---- -title: Docker Playgrounds -nav-id: docker-playgrounds -nav-title: 'Docker Playgrounds' -nav-parent_id: getting-started -nav-pos: 10 ---- - diff --git a/docs/getting-started/index.md b/docs/getting-started/index.md deleted file mode 100644 index fffc502aee58e6..00000000000000 --- a/docs/getting-started/index.md +++ /dev/null @@ -1,87 +0,0 @@ ---- -title: "Getting Started" -nav-id: getting-started -nav-title: ' Getting Started' -nav-parent_id: root -section-break: true -nav-show_overview: true -nav-pos: 1 -always-expand: true ---- - - -There are many ways to get started with Apache Flink. Which one is the best for -you depends on your goals and prior experience: - -* take a look at the **Docker Playgrounds** if you want to see what Flink can do, via a hands-on, - docker-based introduction to specific Flink concepts -* explore one of the **Code Walkthroughs** if you want a quick, end-to-end - introduction to one of Flink's APIs -* work your way through the **Hands-on Training** for a comprehensive, - step-by-step introduction to Flink -* use **Project Setup** if you already know the basics of Flink and want a - project template for Java or Scala, or need help setting up the dependencies - -### Taking a first look at Flink - -The **Docker Playgrounds** provide sandboxed Flink environments that are set up in just a few minutes and which allow you to explore and play with Flink. - -* The [**Operations Playground**]({% link getting-started/docker-playgrounds/flink-operations-playground.md %}) shows you how to operate streaming applications with Flink. You can experience how Flink recovers application from failures, upgrade and scale streaming applications up and down, and query application metrics. - - - -### First steps with one of Flink's APIs - -The **Code Walkthroughs** are a great way to get started quickly with a step-by-step introduction to -one of Flink's APIs. Each walkthrough provides instructions for bootstrapping a small skeleton -project, and then shows how to extend it to a simple application. - -* The [**DataStream API** code walkthrough]({% link getting-started/walkthroughs/datastream_api.md %}) shows how - to implement a simple DataStream application and how to extend it to be stateful and use timers. - The DataStream API is Flink's main abstraction for implementing stateful streaming applications - with sophisticated time semantics in Java or Scala. - -* Flink's **Table API** is a relational API used for writing SQL-like queries in Java, Scala, or - Python, which are then automatically optimized, and can be executed on batch or streaming data - with identical syntax and semantics. The [Table API code walkthrough for Java and Scala]({% link - getting-started/walkthroughs/table_api.md %}) shows how to implement a simple Table API query on a - batch source and how to evolve it into a continuous query on a streaming source. There's also a - similar [code walkthrough for the Python Table API]({% link - getting-started/walkthroughs/python_table_api.md %}). - -### Taking a Deep Dive with the Hands-on Training - -The [**Hands-on Training**]({% link training/index.md %}) is a self-paced training course with -a set of lessons and hands-on exercises. This step-by-step introduction to Flink focuses -on learning how to use the DataStream API to meet the needs of common, real-world use cases, -and provides a complete introduction to the fundamental concepts: parallel dataflows, -stateful stream processing, event time and watermarking, and fault tolerance via state snapshots. - - diff --git a/docs/getting-started/index.zh.md b/docs/getting-started/index.zh.md deleted file mode 100644 index cb81bbc8bab041..00000000000000 --- a/docs/getting-started/index.zh.md +++ /dev/null @@ -1,62 +0,0 @@ ---- -title: 入门指南 -nav-id: getting-started -nav-title: ' Getting Started' -nav-parent_id: root -section-break: true -nav-show_overview: true -nav-pos: 1 -always-expand: true ---- - - -上手使用 Apache Flink 有很多方式，哪一个最适合你取决于你的目标和以前的经验。 - -### 初识 Flink - -通过 **Docker Playgrounds** 提供沙箱的Flink环境，你只需花几分钟做些简单设置，就可以开始探索和使用 Flink。 - -* [**Operations Playground**](./docker-playgrounds/flink-operations-playground.html) 向你展示如何使用 Flink 编写数据流应用程序。你可以体验 Flink 如何从故障中恢复应用程序，升级、提高并行度、降低并行度和监控运行的状态指标等特性。 - - - -### Flink API 入门 - -**代码练习**是入门的最佳方式，通过代码练习可以逐步深入理解 Flink API。 -下边的例子演示了如何使用 Flink 的代码框架开始构建一个基础的 Flink 项目，和如何逐步将其扩展为一个简单的应用程序。 - - -* [**DataStream API 示例**](./walkthroughs/datastream_api.html) 展示了如何编写一个基本的 DataStream 应用程序。 DataStream API 是 Flink 的主要抽象，用于通过 Java 或 Scala 实现具有复杂时间语义的有状态数据流处理的应用程序。 - -* [**Table API 示例**](./walkthroughs/table_api.html) 演示了如何在批处中使用简单的 Table API 进行查询，以及如何将其扩展为流处理中的查询。Table API 是 Flink 的语言嵌入式关系 API，用于在 Java 或 Scala 中编写类 SQL 的查询，这些查询会自动进行优化。Table API 查询可以使用一致的语法和语义同时在批处理或流数据上运行。 - - diff --git a/docs/getting-started/project-setup/dependencies.md b/docs/getting-started/project-setup/dependencies.md deleted file mode 100644 index 51f1dd020d22d3..00000000000000 --- a/docs/getting-started/project-setup/dependencies.md +++ /dev/null @@ -1,237 +0,0 @@ ---- -title: "Configuring Dependencies, Connectors, Libraries" -nav-parent_id: project-setup -nav-pos: 2 ---- - - -Every Flink application depends on a set of Flink libraries. At the bare minimum, the application depends -on the Flink APIs. Many applications depend in addition on certain connector libraries (like Kafka, Cassandra, etc.). -When running Flink applications (either in a distributed deployment, or in the IDE for testing), the Flink -runtime library must be available as well. - - -## Flink Core and Application Dependencies - -As with most systems that run user-defined applications, there are two broad categories of dependencies and libraries in Flink: - - - **Flink Core Dependencies**: Flink itself consists of a set of classes and dependencies that are needed to run the system, for example - coordination, networking, checkpoints, failover, APIs, operations (such as windowing), resource management, etc. - The set of all these classes and dependencies forms the core of Flink's runtime and must be present when a Flink - application is started. - - These core classes and dependencies are packaged in the `flink-dist` jar. They are part of Flink's `lib` folder and - part of the basic Flink container images. Think of these dependencies as similar to Java's core library (`rt.jar`, `charsets.jar`, etc.), - which contains the classes like `String` and `List`. - - The Flink Core Dependencies do not contain any connectors or libraries (CEP, SQL, ML, etc.) in order to avoid having an excessive - number of dependencies and classes in the classpath by default. In fact, we try to keep the core dependencies as slim as possible - to keep the default classpath small and avoid dependency clashes. - - - The **User Application Dependencies** are all connectors, formats, or libraries that a specific user application needs. - - The user application is typically packaged into an *application jar*, which contains the application code and the required - connector and library dependencies. - - The user application dependencies explicitly do not include the Flink DataSet / DataStream APIs and runtime dependencies, - because those are already part of Flink's Core Dependencies. - - -## Setting up a Project: Basic Dependencies - -Every Flink application needs as the bare minimum the API dependencies, to develop against. -For Maven, you can use the [Java Project Template]({{ site.baseurl }}/dev/projectsetup/java_api_quickstart.html) -or [Scala Project Template]({{ site.baseurl }}/dev/projectsetup/scala_api_quickstart.html) to create -a program skeleton with these initial dependencies. - -When setting up a project manually, you need to add the following dependencies for the Java/Scala API -(here presented in Maven syntax, but the same dependencies apply to other build tools (Gradle, SBT, etc.) as well. - - -- -**Important:** Please note that all these dependencies have their scope set to *provided*. -That means that they are needed to compile against, but that they should not be packaged into the -project's resulting application jar file - these dependencies are Flink Core Dependencies, -which are already available in any setup. - -It is highly recommended to keep the dependencies in scope *provided*. If they are not set to *provided*, -the best case is that the resulting JAR becomes excessively large, because it also contains all Flink core -dependencies. The worst case is that the Flink core dependencies that are added to the application's jar file -clash with some of your own dependency versions (which is normally avoided through inverted classloading). - -**Note on IntelliJ:** To make the applications run within IntelliJ IDEA it is necessary to tick the -`Include dependencies with "Provided" scope` box in the run configuration. -If this option is not available (possibly due to using an older IntelliJ IDEA version), then a simple workaround -is to create a test that calls the applications `main()` method. - - -## Adding Connector and Library Dependencies - -Most applications need specific connectors or libraries to run, for example a connector to Kafka, Cassandra, etc. -These connectors are not part of Flink's core dependencies and must hence be added as dependencies to the application - -Below is an example adding the connector for Kafka 0.10 as a dependency (Maven syntax): -{% highlight xml %} --{% highlight xml %} --- -org.apache.flink -flink-java -{{site.version }} -provided -- -{% endhighlight %} -org.apache.flink -flink-streaming-java{{ site.scala_version_suffix }} -{{site.version }} -provided --{% highlight xml %} --- -org.apache.flink -flink-scala{{ site.scala_version_suffix }} -{{site.version }} -provided -- -{% endhighlight %} -org.apache.flink -flink-streaming-scala{{ site.scala_version_suffix }} -{{site.version }} -provided -- -{% endhighlight %} - -We recommend to package the application code and all its required dependencies into one *jar-with-dependencies* which -we refer to as the *application jar*. The application jar can be submitted to an already running Flink cluster, -or added to a Flink application container image. - -Projects created from the [Java Project Template]({{ site.baseurl }}/dev/projectsetup/java_api_quickstart.html) or -[Scala Project Template]({{ site.baseurl }}/dev/projectsetup/scala_api_quickstart.html) are configured to automatically include -the application dependencies into the application jar when running `mvn clean package`. For projects that are -not set up from those templates, we recommend to add the Maven Shade Plugin (as listed in the Appendix below) -to build the application jar with all required dependencies. - -**Important:** For Maven (and other build tools) to correctly package the dependencies into the application jar, -these application dependencies must be specified in scope *compile* (unlike the core dependencies, which -must be specified in scope *provided*). - - -## Scala Versions - -Scala versions (2.10, 2.11, 2.12, etc.) are not binary compatible with one another. -For that reason, Flink for Scala 2.11 cannot be used with an application that uses -Scala 2.12. - -All Flink dependencies that (transitively) depend on Scala are suffixed with the -Scala version that they are built for, for example `flink-streaming-scala_2.11`. - -Developers that only use Java can pick any Scala version, Scala developers need to -pick the Scala version that matches their application's Scala version. - -Please refer to the [build guide]({{ site.baseurl }}/flinkDev/building.html#scala-versions) -for details on how to build Flink for a specific Scala version. - -## Hadoop Dependencies - -**General rule: It should never be necessary to add Hadoop dependencies directly to your application.** -*(The only exception being when using existing Hadoop input-/output formats with Flink's Hadoop compatibility wrappers)* - -If you want to use Flink with Hadoop, you need to have a Flink setup that includes the Hadoop dependencies, rather than -adding Hadoop as an application dependency. Please refer to the [Hadoop Setup Guide]({{ site.baseurl }}/ops/deployment/hadoop.html) -for details. - -There are two main reasons for that design: - - - Some Hadoop interaction happens in Flink's core, possibly before the user application is started, for example - setting up HDFS for checkpoints, authenticating via Hadoop's Kerberos tokens, or deployment on YARN. - - - Flink's inverted classloading approach hides many transitive dependencies from the core dependencies. That applies not only - to Flink's own core dependencies, but also to Hadoop's dependencies when present in the setup. - That way, applications can use different versions of the same dependencies without running into dependency conflicts (and - trust us, that's a big deal, because Hadoops dependency tree is huge.) - -If you need Hadoop dependencies during testing or development inside the IDE (for example for HDFS access), please configure -these dependencies similar to the scope of the dependencies to *test* or to *provided*. - - -## Appendix: Template for building a Jar with Dependencies - -To build an application JAR that contains all dependencies required for declared connectors and libraries, -you can use the following shade plugin definition: - -{% highlight xml %} -org.apache.flink -flink-connector-kafka-0.10{{ site.scala_version_suffix }} -{{site.version }} -- -{% endhighlight %} - -{% top %} - diff --git a/docs/getting-started/project-setup/dependencies.zh.md b/docs/getting-started/project-setup/dependencies.zh.md deleted file mode 100644 index 310000fb9ce854..00000000000000 --- a/docs/getting-started/project-setup/dependencies.zh.md +++ /dev/null @@ -1,200 +0,0 @@ ---- -title: "配置依赖、连接器、类库" -nav-parent_id: project-setup -nav-pos: 2 ---- - - -每个 Flink 应用都需要依赖一组 Flink 类库。Flink 应用至少需要依赖 Flink APIs。许多应用还会额外依赖连接器类库(比如 Kafka、Cassandra 等)。 -当用户运行 Flink 应用时(无论是在 IDE 环境下进行测试，还是部署在分布式环境下)，运行时类库都必须可用。 - -## Flink 核心依赖以及应用依赖 - -与其他运行用户自定义应用的大多数系统一样，Flink 中有两大类依赖类库 - - - **Flink 核心依赖**：Flink 本身包含运行所需的一组类和依赖，比如协调、网络通讯、checkpoint、容错处理、API、算子(如窗口操作)、 - 资源管理等，这些类和依赖形成了 Flink 运行时的核心。当 Flink 应用启动时，这些依赖必须可用。 - - 这些核心类和依赖被打包在 `flink-dist` jar 里。它们是 Flink `lib` 文件夹下的一部分，也是 Flink 基本容器镜像的一部分。 - 这些依赖类似 Java `String` 和 `List` 的核心类库(`rt.jar`, `charsets.jar`等)。 - - Flink 核心依赖不包含连接器和类库（如 CEP、SQL、ML 等），这样做的目的是默认情况下避免在类路径中具有过多的依赖项和类。 - 实际上，我们希望尽可能保持核心依赖足够精简，以保证一个较小的默认类路径，并且避免依赖冲突。 - - - **用户应用依赖** 是指特定的应用程序需要的类库，如连接器，formats等。 - - 用户应用代码和所需的连接器以及其他类库依赖通常被打包到 *application jar* 中。 - - 用户应用程序依赖项不需包括 Flink DataSet / DataStream API 以及运行时依赖项，因为它们已经是 Flink 核心依赖项的一部分。 - -## 搭建一个项目: 基础依赖 - -开发 Flink 应用程序需要最低限度的 API 依赖。Maven 用户，可以使用 -[Java 项目模板]({{ site.baseurl }}/zh/dev/projectsetup/java_api_quickstart.html)或者 -[Scala 项目模板]({{ site.baseurl }}/zh/dev/projectsetup/scala_api_quickstart.html)来创建一个包含最初依赖的程序骨架。 - -手动设置项目时，需要为 Java 或 Scala API 添加以下依赖项（这里以 Maven 语法为例，但也适用于其他构建工具（Gradle、 SBT 等））。 - -- -- -org.apache.maven.plugins -maven-shade-plugin -3.1.1 -- -- -package -- -shade -- -- -- -com.google.code.findbugs:jsr305 -org.slf4j:* -log4j:* -- -- - -*:* -- -META-INF/*.SF -META-INF/*.DSA -META-INF/*.RSA -- -- -my.programs.main.clazz --- -**注意事项:** 所有这些依赖项的作用域都应该设置为 *provided* 。 -这意味着需要这些依赖进行编译，但不应将它们打包到项目生成的应用程序jar文件中-- -因为这些依赖项是 Flink 的核心依赖，在应用启动前已经是可用的状态了。 - -我们强烈建议保持这些依赖的作用域为 *provided* 。 如果它们的作用域未设置为 *provided* ，则典型的情况是因为包含了 Flink 的核心依赖而导致生成的jar包变得过大。 -最糟糕的情况是添加到应用程序的 Flink 核心依赖项与你自己的一些依赖项版本冲突（通常通过反向类加载来避免）。 - -**IntelliJ 上的一些注意事项:** 为了可以让 Flink 应用在 IntelliJ IDEA 中运行，需要在 Run 配置界面将 `Include dependencies with "Provided" scope` 选项勾选上。 -如果这一选项还不可用（可能是因为你正在使用一个老版本的 IntelliJ IDEA），那么一个简单的解决方案是创建一个测试，并在测试中调用应用程序的 `main()` 方法。 - -## 添加连接器以及类库依赖 - -大多数应用需要依赖特定的连接器或其他类库，例如 Kafka、Cassandra 的连接器等。这些连接器不是 Flink 核心依赖的一部分，因此必须作为依赖项手动添加到应用程序中。 - -下面是添加 Kafka 0.10 连接器依赖（Maven 语法）的示例： -{% highlight xml %} --{% highlight xml %} --- -org.apache.flink -flink-java -{{site.version }} -provided -- -{% endhighlight %} -org.apache.flink -flink-streaming-java{{ site.scala_version_suffix }} -{{site.version }} -provided --{% highlight xml %} --- -org.apache.flink -flink-scala{{ site.scala_version_suffix }} -{{site.version }} -provided -- -{% endhighlight %} -org.apache.flink -flink-streaming-scala{{ site.scala_version_suffix }} -{{site.version }} -provided -- -{% endhighlight %} - -我们建议将应用程序代码及其所有需要的依赖项打包到一个 *jar-with-dependencies* 的 jar 包中。 -这个打包好的应用 jar 可以提交到已经运行的 Flink 集群中，或者添加到 Flink 应用容器镜像中。 - -通过[Java 项目模板]({{ site.baseurl }}/zh/dev/projectsetup/java_api_quickstart.html) 或者 -[Scala 项目模板]({{ site.baseurl }}/zh/dev/projectsetup/scala_api_quickstart.html) 创建的应用， -当使用命令 `mvn clean package` 打包的时候会自动将应用依赖类库打包进应用 jar 包。 -对于不是通过上面模板创建的应用，我们推荐添加 Maven Shade Plugin 去构建应用。(下面的附录会给出具体配置) - -**注意:** 要使 Maven（以及其他构建工具）正确地将依赖项打包到应用程序 jar 中，必须将这些依赖项的作用域设置为 *compile* （与核心依赖项不同，后者作用域应该设置为 *provided* ）。 - -## Scala 版本 - -Scala 版本(2.10、2.11、2.12等)互相是不兼容的。因此，依赖 Scala 2.11 的 Flink 环境是不可以运行依赖 Scala 2.12 应用的。 - -所有依赖 Scala 的 Flink 类库都以它们依赖的 Scala 版本为后缀，例如 `flink-streaming-scala_2.11`。 - -只使用 Java 的开发人员可以选择任何 Scala 版本，Scala 开发人员需要选择与其应用程序相匹配的 Scala 版本。 - -对于指定的 Scala 版本如何构建 Flink 应用可以参考 [构建指南]({{ site.baseurl }}/zh/flinkDev/building.html#scala-versions)。 - -## Hadoop 版本 - -**一般规则：永远不需要将 Hadoop 依赖项直接添加到你的应用程序中** -*(唯一的例外是使用 Flink 的 Hadoop 兼容包装器来处理 Hadoop 格式的输入/输出时)* - -如果你想要在 Flink 应用中使用 Hadoop，你需要使用包含 Hadoop 依赖的 Flink，而非将 Hadoop 作为应用依赖进行添加。 -请参考[Hadoop 构建指南]({{ site.baseurl }}/zh/ops/deployment/hadoop.html) - -这样设计是出于两个主要原因: - - - 可能在用户程序启动之前，一些 Hadoop 交互操作就已经发生在 Flink 核心中了，比如为 checkpoint 设置 HDFS 路径，通过 Hadoop's Kerberos tokens 进行权限认证以及进行 YARN 部署等。 - - - Flink 的反向类加载方法隐藏了核心依赖关系中的许多传递依赖关系。这不仅适用于 Flink 自己的核心依赖项，也适用于 Hadoop 在启动中存在的依赖项。 - 通过这种方式，应用程序可以使用相同依赖项的不同版本，而不会引起依赖项的冲突（若非如此可能会引起严重依赖问题，因为 Hadoops 依赖树十分庞大。） - -如果在 IDE 内部进行开发或测试的过程中需要 Hadoop 依赖项（例如用于 HDFS 访问），请将这些依赖项的作用域设置为 *test* 或 *provided* 。 - -## 附录：构建带有依赖的应用 jar 包模板 - -可以通过下面的 shade plugin 配置来构建包含所有依赖项的应用 jar 包 - -{% highlight xml %} -org.apache.flink -flink-connector-kafka-0.10{{ site.scala_version_suffix }} -{{site.version }} -- -{% endhighlight %} - -{% top %} diff --git a/docs/getting-started/project-setup/index.md b/docs/getting-started/project-setup/index.md deleted file mode 100644 index 354ce40a8bf597..00000000000000 --- a/docs/getting-started/project-setup/index.md +++ /dev/null @@ -1,25 +0,0 @@ ---- -title: "Project Setup" -nav-id: project-setup -nav-title: 'Project Setup' -nav-parent_id: getting-started -nav-pos: 30 ---- - diff --git a/docs/getting-started/project-setup/index.zh.md b/docs/getting-started/project-setup/index.zh.md deleted file mode 100644 index ae9512e9cf07bc..00000000000000 --- a/docs/getting-started/project-setup/index.zh.md +++ /dev/null @@ -1,25 +0,0 @@ ---- -title: "项目构建设置" -nav-id: project-setup -nav-title: '项目构建设置' -nav-parent_id: getting-started -nav-pos: 30 ---- - diff --git a/docs/getting-started/project-setup/java_api_quickstart.md b/docs/getting-started/project-setup/java_api_quickstart.md deleted file mode 100644 index 07fc8efb62b944..00000000000000 --- a/docs/getting-started/project-setup/java_api_quickstart.md +++ /dev/null @@ -1,375 +0,0 @@ ---- -title: "Project Template for Java" -nav-title: Project Template for Java -nav-parent_id: project-setup -nav-pos: 0 ---- - - -* This will be replaced by the TOC -{:toc} - - -## Build Tools - -Flink projects can be built with different build tools. -In order to get started quickly, Flink provides project templates for the following build tools: - -- [Maven](#maven) -- [Gradle](#gradle) - -These templates help you to set up the project structure and to create the initial build files. - -## Maven - -### Requirements - -The only requirements are working __Maven 3.0.4__ (or higher) and __Java 8.x__ installations. - -### Create Project - -Use one of the following commands to __create a project__: - - -- -- -org.apache.maven.plugins -maven-shade-plugin -3.1.1 -- -- -package -- -shade -- -- -- -com.google.code.findbugs:jsr305 -org.slf4j:* -log4j:* -- -- - -*:* -- -META-INF/*.SF -META-INF/*.DSA -META-INF/*.RSA -- -- -my.programs.main.clazz --- -### Inspect Project - -There will be a new directory in your working directory. If you've used -the _curl_ approach, the directory is called `quickstart`. Otherwise, -it has the name of your `artifactId`: - -{% highlight bash %} -$ tree quickstart/ -quickstart/ -├── pom.xml -└── src - └── main - ├── java - │ └── org - │ └── myorg - │ └── quickstart - │ ├── BatchJob.java - │ └── StreamingJob.java - └── resources - └── log4j2.properties -{% endhighlight %} - -The sample project is a __Maven project__, which contains two classes: _StreamingJob_ and _BatchJob_ are the basic skeleton programs for a *DataStream* and *DataSet* program. -The _main_ method is the entry point of the program, both for in-IDE testing/execution and for proper deployments. - -We recommend you __import this project into your IDE__ to develop and -test it. IntelliJ IDEA supports Maven projects out of the box. -If you use Eclipse, the [m2e plugin](http://www.eclipse.org/m2e/) -allows to [import Maven projects](http://books.sonatype.com/m2eclipse-book/reference/creating-sect-importing-projects.html#fig-creating-import). -Some Eclipse bundles include that plugin by default, others require you -to install it manually. - -*Please note*: The default JVM heapsize for Java may be too -small for Flink. You have to manually increase it. -In Eclipse, choose `Run Configurations -> Arguments` and write into the `VM Arguments` box: `-Xmx800m`. -In IntelliJ IDEA recommended way to change JVM options is from the `Help | Edit Custom VM Options` menu. See [this article](https://intellij-support.jetbrains.com/hc/en-us/articles/206544869-Configuring-JVM-options-and-platform-properties) for details. - -### Build Project - -If you want to __build/package your project__, go to your project directory and -run the '`mvn clean package`' command. -You will __find a JAR file__ that contains your application, plus connectors and libraries -that you may have added as dependencies to the application: `target/-{% highlight bash %} -$ mvn archetype:generate \ - -DarchetypeGroupId=org.apache.flink \ - -DarchetypeArtifactId=flink-quickstart-java \{% unless site.is_stable %} - -DarchetypeCatalog=https://repository.apache.org/content/repositories/snapshots/ \{% endunless %} - -DarchetypeVersion={{site.version}} -{% endhighlight %} - This allows you to name your newly created project. It will interactively ask you for the groupId, artifactId, and package name. ---{% highlight bash %} -{% if site.is_stable %} -$ curl https://flink.apache.org/q/quickstart.sh | bash -s {{site.version}} -{% else %} -$ curl https://flink.apache.org/q/quickstart-SNAPSHOT.sh | bash -s {{site.version}} -{% endif %} -{% endhighlight %} - -- {% unless site.is_stable %} -- Note: For Maven 3.0 or higher, it is no longer possible to specify the repository (-DarchetypeCatalog) via the command line. For details about this change, please refer to Maven official document - If you wish to use the snapshot repository, you need to add a repository entry to your settings.xml. For example: -{% highlight bash %} -
- {% endunless %} -- -{% endhighlight %} -- -apache -- -- -apache -- -- -apache-snapshots -https://repository.apache.org/content/repositories/snapshots/ -- .jar`. - -__Note:__ If you use a different class than *StreamingJob* as the application's main class / entry point, -we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, Flink -can run the application from the JAR file without additionally specifying the main class. - -## Gradle - -### Requirements - -The only requirements are working __Gradle 3.x__ (or higher) and __Java 8.x__ installations. - -### Create Project - -Use one of the following commands to __create a project__: - - - -- -### Inspect Project - -There will be a new directory in your working directory based on the -project name you provided, e.g. for `quickstart`: - -{% highlight bash %} -$ tree quickstart/ -quickstart/ -├── README -├── build.gradle -├── settings.gradle -└── src - └── main - ├── java - │ └── org - │ └── myorg - │ └── quickstart - │ ├── BatchJob.java - │ └── StreamingJob.java - └── resources - └── log4j2.properties -{% endhighlight %} - -The sample project is a __Gradle project__, which contains two classes: _StreamingJob_ and _BatchJob_ are the basic skeleton programs for a *DataStream* and *DataSet* program. -The _main_ method is the entry point of the program, both for in-IDE testing/execution and for proper deployments. - -We recommend you __import this project into your IDE__ to develop and -test it. IntelliJ IDEA supports Gradle projects after installing the `Gradle` plugin. -Eclipse does so via the [Eclipse Buildship](https://projects.eclipse.org/projects/tools.buildship) plugin -(make sure to specify a Gradle version >= 3.0 in the last step of the import wizard; the `shadow` plugin requires it). -You may also use [Gradle's IDE integration](https://docs.gradle.org/current/userguide/userguide.html#ide-integration) -to create project files from Gradle. - - -*Please note*: The default JVM heapsize for Java may be too -small for Flink. You have to manually increase it. -In Eclipse, choose `Run Configurations -> Arguments` and write into the `VM Arguments` box: `-Xmx800m`. -In IntelliJ IDEA recommended way to change JVM options is from the `Help | Edit Custom VM Options` menu. See [this article](https://intellij-support.jetbrains.com/hc/en-us/articles/206544869-Configuring-JVM-options-and-platform-properties) for details. - -### Build Project - -If you want to __build/package your project__, go to your project directory and -run the '`gradle clean shadowJar`' command. -You will __find a JAR file__ that contains your application, plus connectors and libraries -that you may have added as dependencies to the application: `build/libs/- - -- -- ---{% highlight gradle %} -buildscript { - repositories { - jcenter() // this applies only to the Gradle 'Shadow' plugin - } - dependencies { - classpath 'com.github.jengelman.gradle.plugins:shadow:2.0.4' - } -} - -plugins { - id 'java' - id 'application' - // shadow plugin to produce fat JARs - id 'com.github.johnrengelman.shadow' version '2.0.4' -} - - -// artifact properties -group = 'org.myorg.quickstart' -version = '0.1-SNAPSHOT' -mainClassName = 'org.myorg.quickstart.StreamingJob' -description = """Flink Quickstart Job""" - -ext { - javaVersion = '1.8' - flinkVersion = '{{ site.version }}' - scalaBinaryVersion = '{{ site.scala_version }}' - slf4jVersion = '1.7.15' - log4jVersion = '2.12.1' -} - - -sourceCompatibility = javaVersion -targetCompatibility = javaVersion -tasks.withType(JavaCompile) { - options.encoding = 'UTF-8' -} - -applicationDefaultJvmArgs = ["-Dlog4j.configurationFile=log4j2.properties"] - -task wrapper(type: Wrapper) { - gradleVersion = '3.1' -} - -// declare where to find the dependencies of your project -repositories { - mavenCentral() - maven { url "https://repository.apache.org/content/repositories/snapshots/" } -} - -// NOTE: We cannot use "compileOnly" or "shadow" configurations since then we could not run code -// in the IDE or with "gradle run". We also cannot exclude transitive dependencies from the -// shadowJar yet (see https://github.com/johnrengelman/shadow/issues/159). -// -> Explicitly define the // libraries we want to be included in the "flinkShadowJar" configuration! -configurations { - flinkShadowJar // dependencies which go into the shadowJar - - // always exclude these (also from transitive dependencies) since they are provided by Flink - flinkShadowJar.exclude group: 'org.apache.flink', module: 'force-shading' - flinkShadowJar.exclude group: 'com.google.code.findbugs', module: 'jsr305' - flinkShadowJar.exclude group: 'org.slf4j' - flinkShadowJar.exclude group: 'org.apache.logging.log4j' -} - -// declare the dependencies for your production and test code -dependencies { - // -------------------------------------------------------------- - // Compile-time dependencies that should NOT be part of the - // shadow jar and are provided in the lib folder of Flink - // -------------------------------------------------------------- - compile "org.apache.flink:flink-java:${flinkVersion}" - compile "org.apache.flink:flink-streaming-java_${scalaBinaryVersion}:${flinkVersion}" - - // -------------------------------------------------------------- - // Dependencies that should be part of the shadow jar, e.g. - // connectors. These must be in the flinkShadowJar configuration! - // -------------------------------------------------------------- - //flinkShadowJar "org.apache.flink:flink-connector-kafka-0.11_${scalaBinaryVersion}:${flinkVersion}" - - compile "org.apache.logging.log4j:log4j-api:${log4jVersion}" - compile "org.apache.logging.log4j:log4j-core:${log4jVersion}" - compile "org.apache.logging.log4j:log4j-slf4j-impl:${log4jVersion}" - compile "org.slf4j:slf4j-log4j12:${slf4jVersion}" - - // Add test dependencies here. - // testCompile "junit:junit:4.12" -} - -// make compileOnly dependencies available for tests: -sourceSets { - main.compileClasspath += configurations.flinkShadowJar - main.runtimeClasspath += configurations.flinkShadowJar - - test.compileClasspath += configurations.flinkShadowJar - test.runtimeClasspath += configurations.flinkShadowJar - - javadoc.classpath += configurations.flinkShadowJar -} - -run.classpath = sourceSets.main.runtimeClasspath - -jar { - manifest { - attributes 'Built-By': System.getProperty('user.name'), - 'Build-Jdk': System.getProperty('java.version') - } -} - -shadowJar { - configurations = [project.configurations.flinkShadowJar] -} -{% endhighlight %} ---{% highlight gradle %} -rootProject.name = 'quickstart' -{% endhighlight %} ---{% highlight bash %} -bash -c "$(curl https://flink.apache.org/q/gradle-quickstart.sh)" -- {{site.version}} {{site.scala_version}} -{% endhighlight %} - This allows you to name your newly created project. It will interactively ask - you for the project name, organization (also used for the package name), project version, - Scala and Flink version. --- -all.jar`. - -__Note:__ If you use a different class than *StreamingJob* as the application's main class / entry point, -we recommend you change the `mainClassName` setting in the `build.gradle` file accordingly. That way, Flink -can run the application from the JAR file without additionally specifying the main class. - -## Next Steps - -Write your application! - -If you are writing a streaming application and you are looking for inspiration what to write, -take a look at the [Stream Processing Application Tutorial]({{ site.baseurl }}/getting-started/walkthroughs/datastream_api.html). - -If you are writing a batch processing application and you are looking for inspiration what to write, -take a look at the [Batch Application Examples]({{ site.baseurl }}/dev/batch/examples.html). - -For a complete overview over the APIs, have a look at the -[DataStream API]({{ site.baseurl }}/dev/datastream_api.html) and -[DataSet API]({{ site.baseurl }}/dev/batch/index.html) sections. - -[Here]({{ site.baseurl }}/ops/deployment/local.html) you can find out how to run an application outside the IDE on a local cluster. - -If you have any trouble, ask on our -[Mailing List](http://mail-archives.apache.org/mod_mbox/flink-user/). -We are happy to provide help. - -{% top %} diff --git a/docs/getting-started/project-setup/java_api_quickstart.zh.md b/docs/getting-started/project-setup/java_api_quickstart.zh.md deleted file mode 100644 index a5a0493a983781..00000000000000 --- a/docs/getting-started/project-setup/java_api_quickstart.zh.md +++ /dev/null @@ -1,360 +0,0 @@ ---- -title: "Java 项目模板" -nav-title: Java 项目模板 -nav-parent_id: project-setup -nav-pos: 0 ---- - - -* This will be replaced by the TOC -{:toc} - - -## 构建工具 - -Flink项目可以使用不同的构建工具进行构建。 -为了能够快速入门，Flink 为以下构建工具提供了项目模版： - -- [Maven](#maven) -- [Gradle](#gradle) - -这些模版可以帮助你搭建项目结构并创建初始构建文件。 - -## Maven - -### 环境要求 - -唯一的要求是使用 __Maven 3.0.4__ （或更高版本）和安装 __Java 8.x__。 - -### 创建项目 - -使用以下命令之一来 __创建项目__： - - - -- -### 检查项目 - -项目创建后，工作目录将多出一个新目录。如果你使用的是 _curl_ 方式创建项目，目录名为 `quickstart`； -如果你使用的是 _Maven archetypes_ 方式创建项目，则目录名为你指定的 `artifactId`： - -{% highlight bash %} -$ tree quickstart/ -quickstart/ -├── pom.xml -└── src - └── main - ├── java - │ └── org - │ └── myorg - │ └── quickstart - │ ├── BatchJob.java - │ └── StreamingJob.java - └── resources - └── log4j2.properties -{% endhighlight %} - -示例项目是一个 __Maven project__，它包含了两个类：_StreamingJob_ 和 _BatchJob_ 分别是 *DataStream* and *DataSet* 程序的基础骨架程序。 -_main_ 方法是程序的入口，既可用于IDE测试/执行，也可用于部署。 - -我们建议你将 __此项目导入IDE__ 来开发和测试它。 -IntelliJ IDEA 支持 Maven 项目开箱即用。如果你使用的是 Eclipse，使用[m2e 插件](http://www.eclipse.org/m2e/) 可以 -[导入 Maven 项目](http://books.sonatype.com/m2eclipse-book/reference/creating-sect-importing-projects.html#fig-creating-import)。 -一些 Eclipse 捆绑包默认包含该插件，其他情况需要你手动安装。 - -*请注意*：对 Flink 来说，默认的 JVM 堆内存可能太小，你应当手动增加堆内存。 -在 Eclipse 中，选择 `Run Configurations -> Arguments` 并在 `VM Arguments` 对应的输入框中写入：`-Xmx800m`。 -在 IntelliJ IDEA 中，推荐从菜单 `Help | Edit Custom VM Options` 来修改 JVM 选项。有关详细信息，请参阅[这篇文章](https://intellij-support.jetbrains.com/hc/en-us/articles/206544869-Configuring-JVM-options-and-platform-properties)。 - -### 构建项目 - -如果你想要 __构建/打包你的项目__，请在项目目录下运行 '`mvn clean package`' 命令。 -命令执行后，你将 __找到一个JAR文件__，里面包含了你的应用程序，以及已作为依赖项添加到应用程序的连接器和库：`target/-{% highlight bash %} -$ mvn archetype:generate \ - -DarchetypeGroupId=org.apache.flink \ - -DarchetypeArtifactId=flink-quickstart-java \{% unless site.is_stable %} - -DarchetypeCatalog=https://repository.apache.org/content/repositories/snapshots/ \{% endunless %} - -DarchetypeVersion={{site.version}} -{% endhighlight %} - 这种方式允许你为新项目命名。它将以交互式的方式询问你项目的 groupId、artifactId 和 package 名称。 ---{% highlight bash %} -{% if site.is_stable %} -$ curl https://flink.apache.org/q/quickstart.sh | bash -s {{site.version}} -{% else %} -$ curl https://flink.apache.org/q/quickstart-SNAPSHOT.sh | bash -s {{site.version}} -{% endif %} -{% endhighlight %} - -- {% unless site.is_stable %} -- 注意：Maven 3.0 及更高版本，不再支持通过命令行指定仓库（-DarchetypeCatalog）。有关这个改动的详细信息， - 如果你希望使用快照仓库，则需要在 settings.xml 文件中添加一个仓库条目。有关这个改动的详细信息， - 请参阅 Maven 官方文档 请参阅 Maven 官方文档 - 如果你希望使用快照仓库，则需要在 settings.xml 文件中添加一个仓库条目。例如： -{% highlight bash %} -
- {% endunless %} -- -{% endhighlight %} -- -apache -- -- -apache -- -- -apache-snapshots -https://repository.apache.org/content/repositories/snapshots/ -- .jar`。 - -__注意：__ 如果你使用其他类而不是 *StreamingJob* 作为应用程序的主类/入口， -我们建议你相应地修改 `pom.xml` 文件中的 `mainClass` 配置。这样， -Flink 可以从 JAR 文件运行应用程序，而无需另外指定主类。 - -## Gradle - -### 环境要求 - -唯一的要求是使用 __Gradle 3.x__ (或更高版本) 和安装 __Java 8.x__ 。 - -### 创建项目 - -使用以下命令之一来 __创建项目__： - - - -- -### 检查项目 - -根据你提供的项目名称，工作目录中将多出一个新目录，例如 `quickstart`： - -{% highlight bash %} -$ tree quickstart/ -quickstart/ -├── README -├── build.gradle -├── settings.gradle -└── src - └── main - ├── java - │ └── org - │ └── myorg - │ └── quickstart - │ ├── BatchJob.java - │ └── StreamingJob.java - └── resources - └── log4j2.properties -{% endhighlight %} - -示例项目是一个 __Gradle 项目__，它包含了两个类：_StreamingJob_ 和 _BatchJob_ 是 *DataStream* 和 *DataSet* 程序的基础骨架程序。 -_main_ 方法是程序的入口，即可用于IDE测试/执行，也可用于部署。 - -我们建议你将 __此项目导入你的 IDE__ 来开发和测试它。 -IntelliJ IDEA 在安装 `Gradle` 插件后支持 Gradle 项目。Eclipse 则通过 [Eclipse Buildship](https://projects.eclipse.org/projects/tools.buildship) 插件支持 Gradle 项目（鉴于 `shadow` 插件对 Gradle 版本有要求，请确保在导入向导的最后一步指定 Gradle 版本 >= 3.0）。 -你也可以使用 [Gradle’s IDE integration](https://docs.gradle.org/current/userguide/userguide.html#ide-integration) 从 Gradle -创建项目文件。 - - -*请注意*：对 Flink 来说，默认的 JVM 堆内存可能太小，你应当手动增加堆内存。 -在 Eclipse中，选择 `Run Configurations -> Arguments` 并在 `VM Arguments` 对应的输入框中写入：`-Xmx800m`。 -在 IntelliJ IDEA 中，推荐从菜单 `Help | Edit Custom VM Options` 来修改 JVM 选项。有关详细信息，请参阅[此文章](https://intellij-support.jetbrains.com/hc/en-us/articles/206544869-Configuring-JVM-options-and-platform-properties)。 - -### 构建项目 - -如果你想要 __构建/打包项目__，请在项目目录下运行 '`gradle clean shadowJar`' 命令。 -命令执行后，你将 __找到一个 JAR 文件__，里面包含了你的应用程序，以及已作为依赖项添加到应用程序的连接器和库：`build/libs/- - -- -- ---{% highlight gradle %} -buildscript { - repositories { - jcenter() // this applies only to the Gradle 'Shadow' plugin - } - dependencies { - classpath 'com.github.jengelman.gradle.plugins:shadow:2.0.4' - } -} - -plugins { - id 'java' - id 'application' - // shadow plugin to produce fat JARs - id 'com.github.johnrengelman.shadow' version '2.0.4' -} - - -// artifact properties -group = 'org.myorg.quickstart' -version = '0.1-SNAPSHOT' -mainClassName = 'org.myorg.quickstart.StreamingJob' -description = """Flink Quickstart Job""" - -ext { - javaVersion = '1.8' - flinkVersion = '{{ site.version }}' - scalaBinaryVersion = '{{ site.scala_version }}' - slf4jVersion = '1.7.15' - log4jVersion = '2.12.1' -} - - -sourceCompatibility = javaVersion -targetCompatibility = javaVersion -tasks.withType(JavaCompile) { - options.encoding = 'UTF-8' -} - -applicationDefaultJvmArgs = ["-Dlog4j.configurationFile=log4j2.properties"] - -task wrapper(type: Wrapper) { - gradleVersion = '3.1' -} - -// declare where to find the dependencies of your project -repositories { - mavenCentral() - maven { url "https://repository.apache.org/content/repositories/snapshots/" } -} - -// 注意：我们不能使用 "compileOnly" 或者 "shadow" 配置，这会使我们无法在 IDE 中或通过使用 "gradle run" 命令运行代码。 -// 我们也不能从 shadowJar 中排除传递依赖（请查看 https://github.com/johnrengelman/shadow/issues/159)。 -// -> 显式定义我们想要包含在 "flinkShadowJar" 配置中的类库! -configurations { - flinkShadowJar // dependencies which go into the shadowJar - - // 总是排除这些依赖（也来自传递依赖），因为 Flink 会提供这些依赖。 - flinkShadowJar.exclude group: 'org.apache.flink', module: 'force-shading' - flinkShadowJar.exclude group: 'com.google.code.findbugs', module: 'jsr305' - flinkShadowJar.exclude group: 'org.slf4j' - flinkShadowJar.exclude group: 'org.apache.logging.log4j' -} - -// declare the dependencies for your production and test code -dependencies { - // -------------------------------------------------------------- - // 编译时依赖不应该包含在 shadow jar 中， - // 这些依赖会在 Flink 的 lib 目录中提供。 - // -------------------------------------------------------------- - compile "org.apache.flink:flink-java:${flinkVersion}" - compile "org.apache.flink:flink-streaming-java_${scalaBinaryVersion}:${flinkVersion}" - - // -------------------------------------------------------------- - // 应该包含在 shadow jar 中的依赖，例如：连接器。 - // 它们必须在 flinkShadowJar 的配置中！ - // -------------------------------------------------------------- - //flinkShadowJar "org.apache.flink:flink-connector-kafka-0.11_${scalaBinaryVersion}:${flinkVersion}" - - compile "org.apache.logging.log4j:log4j-api:${log4jVersion}" - compile "org.apache.logging.log4j:log4j-core:${log4jVersion}" - compile "org.apache.logging.log4j:log4j-slf4j-impl:${log4jVersion}" - compile "org.slf4j:slf4j-log4j12:${slf4jVersion}" - - // Add test dependencies here. - // testCompile "junit:junit:4.12" -} - -// make compileOnly dependencies available for tests: -sourceSets { - main.compileClasspath += configurations.flinkShadowJar - main.runtimeClasspath += configurations.flinkShadowJar - - test.compileClasspath += configurations.flinkShadowJar - test.runtimeClasspath += configurations.flinkShadowJar - - javadoc.classpath += configurations.flinkShadowJar -} - -run.classpath = sourceSets.main.runtimeClasspath - -jar { - manifest { - attributes 'Built-By': System.getProperty('user.name'), - 'Build-Jdk': System.getProperty('java.version') - } -} - -shadowJar { - configurations = [project.configurations.flinkShadowJar] -} -{% endhighlight %} ---{% highlight gradle %} -rootProject.name = 'quickstart' -{% endhighlight %} ---{% highlight bash %} -bash -c "$(curl https://flink.apache.org/q/gradle-quickstart.sh)" -- {{site.version}} {{site.scala_version}} -{% endhighlight %} - 这种方式允许你为新项目命名。它将以交互式的方式询问你的项目名称、组织机构（也用于包名）、项目版本、Scala 和 Flink 版本。 --- -all.jar`。 - -__注意：__ 如果你使用其他类而不是 *StreamingJob* 作为应用程序的主类/入口， -我们建议你相应地修改 `build.gradle` 文件中的 `mainClassName` 配置。 -这样，Flink 可以从 JAR 文件运行应用程序，而无需另外指定主类。 - -## 下一步 - -开始编写应用！ - -如果你准备编写流处理应用，正在寻找灵感来写什么， -可以看看[流处理应用程序教程]({{ site.baseurl }}/zh/getting-started/walkthroughs/datastream_api.html) - -如果你准备编写批处理应用，正在寻找灵感来写什么， -可以看看[批处理应用程序示例]({{ site.baseurl }}/zh/dev/batch/examples.html) - -有关 API 的完整概述，请查看 -[DataStream API]({{ site.baseurl }}/zh/dev/datastream_api.html) 和 -[DataSet API]({{ site.baseurl }}/zh/dev/batch/index.html) 章节。 - -在[这里]({{ site.baseurl }}/zh/ops/deployment/local.html)，你可以找到如何在 IDE 之外的本地集群中运行应用程序。 - -如果你有任何问题，请发信至我们的[邮箱列表](http://mail-archives.apache.org/mod_mbox/flink-user/)，我们很乐意提供帮助。 - -{% top %} diff --git a/docs/getting-started/project-setup/scala_api_quickstart.md b/docs/getting-started/project-setup/scala_api_quickstart.md deleted file mode 100644 index 82e13683433cb9..00000000000000 --- a/docs/getting-started/project-setup/scala_api_quickstart.md +++ /dev/null @@ -1,249 +0,0 @@ ---- -title: "Project Template for Scala" -nav-title: Project Template for Scala -nav-parent_id: project-setup -nav-pos: 1 ---- - - -* This will be replaced by the TOC -{:toc} - - -## Build Tools - -Flink projects can be built with different build tools. -In order to get started quickly, Flink provides project templates for the following build tools: - -- [SBT](#sbt) -- [Maven](#maven) - -These templates help you to set up the project structure and to create the initial build files. - -## SBT - -### Create Project - -You can scaffold a new project via either of the following two methods: - - - - -- -### Build Project - -In order to build your project you simply have to issue the `sbt clean assembly` command. -This will create the fat-jar __your-project-name-assembly-0.1-SNAPSHOT.jar__ in the directory __target/scala_your-major-scala-version/__. - -### Run Project - -In order to run your project you have to issue the `sbt run` command. - -Per default, this will run your job in the same JVM as `sbt` is running. -In order to run your job in a distinct JVM, add the following line to `build.sbt` - -{% highlight scala %} -fork in run := true -{% endhighlight %} - - -#### IntelliJ - -We recommend using [IntelliJ](https://www.jetbrains.com/idea/) for your Flink job development. -In order to get started, you have to import your newly created project into IntelliJ. -You can do this via `File -> New -> Project from Existing Sources...` and then choosing your project's directory. -IntelliJ will then automatically detect the `build.sbt` file and set everything up. - -In order to run your Flink job, it is recommended to choose the `mainRunner` module as the classpath of your __Run/Debug Configuration__. -This will ensure, that all dependencies which are set to _provided_ will be available upon execution. -You can configure the __Run/Debug Configurations__ via `Run -> Edit Configurations...` and then choose `mainRunner` from the _Use classpath of module_ dropbox. - -#### Eclipse - -In order to import the newly created project into [Eclipse](https://eclipse.org/), you first have to create Eclipse project files for it. -These project files can be created via the [sbteclipse](https://github.com/typesafehub/sbteclipse) plugin. -Add the following line to your `PROJECT_DIR/project/plugins.sbt` file: - -{% highlight bash %} -addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "4.0.0") -{% endhighlight %} - -In `sbt` use the following command to create the Eclipse project files - -{% highlight bash %} -> eclipse -{% endhighlight %} - -Now you can import the project into Eclipse via `File -> Import... -> Existing Projects into Workspace` and then select the project directory. - -## Maven - -### Requirements - -The only requirements are working __Maven 3.0.4__ (or higher) and __Java 8.x__ installations. - - -### Create Project - -Use one of the following commands to __create a project__: - - - --{% highlight bash %} -$ sbt new tillrohrmann/flink-project.g8 -{% endhighlight %} - This will prompt you for a couple of parameters (project name, flink version...) and then create a Flink project from the flink-project template. - You need sbt >= 0.13.13 to execute this command. You can follow this installation guide to obtain it if necessary. ---{% highlight bash %} -$ bash <(curl https://flink.apache.org/q/sbt-quickstart.sh) -{% endhighlight %} - This will create a Flink project in the specified project directory. ---- - -### Inspect Project - -There will be a new directory in your working directory. If you've used -the _curl_ approach, the directory is called `quickstart`. Otherwise, -it has the name of your `artifactId`: - -{% highlight bash %} -$ tree quickstart/ -quickstart/ -├── pom.xml -└── src - └── main - ├── resources - │ └── log4j2.properties - └── scala - └── org - └── myorg - └── quickstart - ├── BatchJob.scala - └── StreamingJob.scala -{% endhighlight %} - -The sample project is a __Maven project__, which contains two classes: _StreamingJob_ and _BatchJob_ are the basic skeleton programs for a *DataStream* and *DataSet* program. -The _main_ method is the entry point of the program, both for in-IDE testing/execution and for proper deployments. - -We recommend you __import this project into your IDE__. - -IntelliJ IDEA supports Maven out of the box and offers a plugin for Scala development. -From our experience, IntelliJ provides the best experience for developing Flink applications. - -For Eclipse, you need the following plugins, which you can install from the provided Eclipse Update Sites: - -* _Eclipse 4.x_ - * [Scala IDE](http://download.scala-ide.org/sdk/lithium/e44/scala211/stable/site) - * [m2eclipse-scala](http://alchim31.free.fr/m2e-scala/update-site) - * [Build Helper Maven Plugin](https://repo1.maven.org/maven2/.m2e/connectors/m2eclipse-buildhelper/0.15.0/N/0.15.0.201207090124/) -* _Eclipse 3.8_ - * [Scala IDE for Scala 2.11](http://download.scala-ide.org/sdk/helium/e38/scala211/stable/site) or [Scala IDE for Scala 2.10](http://download.scala-ide.org/sdk/helium/e38/scala210/stable/site) - * [m2eclipse-scala](http://alchim31.free.fr/m2e-scala/update-site) - * [Build Helper Maven Plugin](https://repository.sonatype.org/content/repositories/forge-sites/m2e-extras/0.14.0/N/0.14.0.201109282148/) - -### Build Project - -If you want to __build/package your project__, go to your project directory and -run the '`mvn clean package`' command. -You will __find a JAR file__ that contains your application, plus connectors and libraries -that you may have added as dependencies to the application: `target/-{% highlight bash %} -$ mvn archetype:generate \ - -DarchetypeGroupId=org.apache.flink \ - -DarchetypeArtifactId=flink-quickstart-scala \{% unless site.is_stable %} - -DarchetypeCatalog=https://repository.apache.org/content/repositories/snapshots/ \{% endunless %} - -DarchetypeVersion={{site.version}} -{% endhighlight %} - This allows you to name your newly created project. It will interactively ask you for the groupId, artifactId, and package name. ---{% highlight bash %} -{% if site.is_stable %} -$ curl https://flink.apache.org/q/quickstart-scala.sh | bash -s {{site.version}} -{% else %} -$ curl https://flink.apache.org/q/quickstart-scala-SNAPSHOT.sh | bash -s {{site.version}} -{% endif %} -{% endhighlight %} -- {% unless site.is_stable %} -- Note: For Maven 3.0 or higher, it is no longer possible to specify the repository (-DarchetypeCatalog) via the command line. For details about this change, please refer to Maven official document - If you wish to use the snapshot repository, you need to add a repository entry to your settings.xml. For example: -{% highlight bash %} -
- {% endunless %} -- -{% endhighlight %} -- -apache -- -- -apache -- -- -apache-snapshots -https://repository.apache.org/content/repositories/snapshots/ -- .jar`. - -__Note:__ If you use a different class than *StreamingJob* as the application's main class / entry point, -we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, the Flink -can run time application from the JAR file without additionally specifying the main class. - - -## Next Steps - -Write your application! - -If you are writing a streaming application and you are looking for inspiration what to write, -take a look at the [Stream Processing Application Tutorial]({{ site.baseurl }}/getting-started/walkthroughs/datastream_api.html) - -If you are writing a batch processing application and you are looking for inspiration what to write, -take a look at the [Batch Application Examples]({{ site.baseurl }}/dev/batch/examples.html) - -For a complete overview over the APIs, have a look at the -[DataStream API]({{ site.baseurl }}/dev/datastream_api.html) and -[DataSet API]({{ site.baseurl }}/dev/batch/index.html) sections. - -[Here]({{ site.baseurl }}/ops/deployment/local.html) you can find out how to run an application outside the IDE on a local cluster. - -If you have any trouble, ask on our -[Mailing List](http://mail-archives.apache.org/mod_mbox/flink-user/). -We are happy to provide help. - -{% top %} diff --git a/docs/getting-started/project-setup/scala_api_quickstart.zh.md b/docs/getting-started/project-setup/scala_api_quickstart.zh.md deleted file mode 100644 index 8628d078a221ce..00000000000000 --- a/docs/getting-started/project-setup/scala_api_quickstart.zh.md +++ /dev/null @@ -1,241 +0,0 @@ ---- -title: "Scala 项目模板" -nav-title: Scala 项目模板 -nav-parent_id: project-setup -nav-pos: 1 ---- - - -* This will be replaced by the TOC -{:toc} - - -## 构建工具 - -可以使用不同的构建工具来构建Flink项目。 -为了快速入门，Flink为以下构建工具提供了项目模板： - -- [SBT](#sbt) -- [Maven](#maven) - -这些模板将帮助你建立项目的框架并创建初始化的构建文件。 - -## SBT - -### 创建项目 - -你可以通过以下两种方法之一构建新项目: - - - - -- -### 构建项目 - -为了构建你的项目，仅需简单的运行 `sbt clean assembly` 命令。 -这将在 __target/scala_your-major-scala-version/__ 目录中创建一个 fat-jar __your-project-name-assembly-0.1-SNAPSHOT.jar__。 - -### 运行项目 - -为了构建你的项目，需要运行 `sbt run`。 - -默认情况下，这会在运行 `sbt` 的 JVM 中运行你的作业。 -为了在不同的 JVM 中运行，请添加以下内容添加到 `build.sbt` - -{% highlight scala %} -fork in run := true -{% endhighlight %} - - -#### IntelliJ - -我们建议你使用 [IntelliJ](https://www.jetbrains.com/idea/) 来开发Flink作业。 -开始，你需要将新建的项目导入到 IntelliJ。 -通过 `File -> New -> Project from Existing Sources...` 操作路径，然后选择项目目录。之后 IntelliJ 将自动检测到 `build.sbt` 文件并设置好所有内容。 - -为了运行你的Flink作业，建议选择 `mainRunner` 模块作为 __Run/Debug Configuration__ 的类路径。 -这将确保在作业执行时可以使用所有设置为 _provided_ 的依赖项。 -你可以通过 `Run -> Edit Configurations...` 配置 __Run/Debug Configurations__，然后从 _Use classpath of module_ 下拉框中选择 `mainRunner`。 - -#### Eclipse - -为了将新建的项目导入 [Eclipse](https://eclipse.org/), 首先需要创建一个 Eclipse 项目文件。 -通过插件 [sbteclipse](https://github.com/typesafehub/sbteclipse) 创建项目文件，并将下面的内容添加到 `PROJECT_DIR/project/plugins.sbt` 文件中: - -{% highlight bash %} -addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "4.0.0") -{% endhighlight %} - -在 `sbt` 中使用以下命令创建 Eclipse 项目文件 - -{% highlight bash %} -> eclipse -{% endhighlight %} - -现在你可以通过 `File -> Import... -> Existing Projects into Workspace` 将项目导入 Eclipse，然后选择项目目录。 - -## Maven - -### 环境要求 - -唯一的要求是安装 __Maven 3.0.4__ (或更高版本) 和 __Java 8.x__。 - - -### 创建项目 - -使用以下命令之一来 __创建项目__: - - - --{% highlight bash %} -$ sbt new tillrohrmann/flink-project.g8 -{% endhighlight %} - 这里将提示你输入几个参数 (项目名称，Flink版本...) 然后从 Flink项目模版创建一个Flink项目。 - 你的sbt版本需要不小于0.13.13才能执行这个命令。如有必要，你可以参考这个安装指南获取合适版本的sbt。 ---{% highlight bash %} -$ bash <(curl https://flink.apache.org/q/sbt-quickstart.sh) -{% endhighlight %} - 这将在指定的目录创建一个Flink项目。 ---- - -### 检查项目 - -项目创建后，工作目录中将多出一个新目录。如果你使用的是 _curl_ 方式创建项目，目录称为 `quickstart`，如果是另外一种创建方式，目录则称为你指定的 `artifactId`。 - -{% highlight bash %} -$ tree quickstart/ -quickstart/ -├── pom.xml -└── src - └── main - ├── resources - │ └── log4j2.properties - └── scala - └── org - └── myorg - └── quickstart - ├── BatchJob.scala - └── StreamingJob.scala -{% endhighlight %} - -样例项目是一个 __Maven 项目__, 包含了两个类： _StreamingJob_ 和 _BatchJob_ 是 *DataStream* 和 *DataSet* 程序的基本框架程序. -_main_ 方法是程序的入口, 既用于 IDE 内的测试/执行，也用于合理部署。 - -我们建议你将 __此项目导入你的 IDE__。 - -IntelliJ IDEA 支持 Maven 开箱即用，并为Scala开发提供插件。 -从我们的经验来看，IntelliJ 提供了最好的Flink应用程序开发体验。 - -对于 Eclipse，需要以下的插件，你可以从提供的 Eclipse Update Sites 安装这些插件： - -* _Eclipse 4.x_ - * [Scala IDE](http://download.scala-ide.org/sdk/lithium/e44/scala211/stable/site) - * [m2eclipse-scala](http://alchim31.free.fr/m2e-scala/update-site) - * [Build Helper Maven Plugin](https://repo1.maven.org/maven2/.m2e/connectors/m2eclipse-buildhelper/0.15.0/N/0.15.0.201207090124/) -* _Eclipse 3.8_ - * [Scala IDE for Scala 2.11](http://download.scala-ide.org/sdk/helium/e38/scala211/stable/site) 或者 [Scala IDE for Scala 2.10](http://download.scala-ide.org/sdk/helium/e38/scala210/stable/site) - * [m2eclipse-scala](http://alchim31.free.fr/m2e-scala/update-site) - * [Build Helper Maven Plugin](https://repository.sonatype.org/content/repositories/forge-sites/m2e-extras/0.14.0/N/0.14.0.201109282148/) - -### 构建 - -如果你想要 __构建/打包你的项目__, 进入到你的项目目录，并执行命令‘`mvn clean package`’。 -你将 __找到一个 JAR 文件__，其中包含了你的应用程序，以及已作为依赖项添加到应用程序的连接器和库：`target/-{% highlight bash %} -$ mvn archetype:generate \ - -DarchetypeGroupId=org.apache.flink \ - -DarchetypeArtifactId=flink-quickstart-scala \{% unless site.is_stable %} - -DarchetypeCatalog=https://repository.apache.org/content/repositories/snapshots/ \{% endunless %} - -DarchetypeVersion={{site.version}} -{% endhighlight %} - 这将允许你 为新项目命名。同时以交互式的方式询问你项目的 groupId，artifactId 和 package 名称. ---{% highlight bash %} -{% if site.is_stable %} -$ curl https://flink.apache.org/q/quickstart-scala.sh | bash -s {{site.version}} -{% else %} -$ curl https://flink.apache.org/q/quickstart-scala-SNAPSHOT.sh | bash -s {{site.version}} -{% endif %} -{% endhighlight %} -- {% unless site.is_stable %} -- 注意：Maven 3.0 及更高版本，不再支持通过命令行指定仓库（-DarchetypeCatalog）。有关这个改动的详细信息， - 请参阅 Maven 官方文档 - 如果你希望使用快照仓库，则需要在 settings.xml 文件中添加一个仓库条目。例如： -{% highlight bash %} -
- {% endunless %} -- -{% endhighlight %} -- -apache -- -- -apache -- -- -apache-snapshots -https://repository.apache.org/content/repositories/snapshots/ -- .jar`。 - -__注意:__ 如果你使用其他类而不是 *StreamingJob* 作为应用程序的主类/入口，我们建议你相应地更改 `pom.xml` 文件中 `mainClass` 的设置。这样，Flink 运行应用程序时无需另外指定主类。 - - -## 下一步 - -开始编写你的应用！ - -如果你准备编写流处理应用，正在寻找灵感来写什么， -可以看看[流处理应用程序教程]({{ site.baseurl }}/zh/getting-started/walkthroughs/datastream_api.html) - -如果你准备编写批处理应用，正在寻找灵感来写什么， -可以看看[批处理应用程序示例]({{ site.baseurl }}/zh/dev/batch/examples.html) - -有关 API 的完整概述，请查看 -[DataStream API]({{ site.baseurl }}/zh/dev/datastream_api.html) 和 -[DataSet API]({{ site.baseurl }}/zh/dev/batch/index.html) 部分。 - -在[这里]({{ site.baseurl }}/zh/ops/deployment/local.html)，你可以找到如何在IDE外的本地集群中运行应用程序。 - -如果你有任何问题，请发信至我们的[邮箱列表](http://mail-archives.apache.org/mod_mbox/flink-user/)。 -我们很乐意提供帮助。 - -{% top %} diff --git a/docs/getting-started/walkthroughs/datastream_api.md b/docs/getting-started/walkthroughs/datastream_api.md deleted file mode 100644 index 7281a65767b2e4..00000000000000 --- a/docs/getting-started/walkthroughs/datastream_api.md +++ /dev/null @@ -1,944 +0,0 @@ ---- -title: "DataStream API" -nav-id: datastreamwalkthrough -nav-title: 'DataStream API' -nav-parent_id: walkthroughs -nav-pos: 1 ---- - - -Apache Flink offers a DataStream API for building robust, stateful streaming applications. -It provides fine-grained control over state and time, which allows for the implementation of advanced event-driven systems. -In this step-by-step guide you'll learn how to build a stateful streaming application with Flink's DataStream API. - -* This will be replaced by the TOC -{:toc} - -## What Are You Building? - -Credit card fraud is a growing concern in the digital age. -Criminals steal credit card numbers by running scams or hacking into insecure systems. -Stolen numbers are tested by making one or more small purchases, often for a dollar or less. -If that works, they then make more significant purchases to get items they can sell or keep for themselves. - -In this tutorial, you will build a fraud detection system for alerting on suspicious credit card transactions. -Using a simple set of rules, you will see how Flink allows us to implement advanced business logic and act in real-time. - -## Prerequisites - -This walkthrough assumes that you have some familiarity with Java or Scala, but you should be able to follow along even if you are coming from a different programming language. - -## Help, I’m Stuck! - -If you get stuck, check out the [community support resources](https://flink.apache.org/gettinghelp.html). -In particular, Apache Flink's [user mailing list](https://flink.apache.org/community.html#mailing-lists) is consistently ranked as one of the most active of any Apache project and a great way to get help quickly. - -## How to Follow Along - -If you want to follow along, you will require a computer with: - -* Java 8 or 11 -* Maven - -A provided Flink Maven Archetype will create a skeleton project with all the necessary dependencies quickly, so you only need to focus on filling out the business logic. -These dependencies include `flink-streaming-java` which is the core dependency for all Flink streaming applications and `flink-walkthrough-common` that has data generators and other classes specific to this walkthrough. - -{% panel **Note:** Each code block within this walkthrough may not contain the full surrounding class for brevity. The full code is available [at the bottom of the page](#final-application). %} - - -- -{% unless site.is_stable %} --{% highlight bash %} -$ mvn archetype:generate \ - -DarchetypeGroupId=org.apache.flink \ - -DarchetypeArtifactId=flink-walkthrough-datastream-java \{% unless site.is_stable %} - -DarchetypeCatalog=https://repository.apache.org/content/repositories/snapshots/ \{% endunless %} - -DarchetypeVersion={{ site.version }} \ - -DgroupId=frauddetection \ - -DartifactId=frauddetection \ - -Dversion=0.1 \ - -Dpackage=spendreport \ - -DinteractiveMode=false -{% endhighlight %} ---{% highlight bash %} -$ mvn archetype:generate \ - -DarchetypeGroupId=org.apache.flink \ - -DarchetypeArtifactId=flink-walkthrough-datastream-scala \{% unless site.is_stable %} - -DarchetypeCatalog=https://repository.apache.org/content/repositories/snapshots/ \{% endunless %} - -DarchetypeVersion={{ site.version }} \ - -DgroupId=frauddetection \ - -DartifactId=frauddetection \ - -Dversion=0.1 \ - -Dpackage=spendreport \ - -DinteractiveMode=false -{% endhighlight %} --- Note: For Maven 3.0 or higher, it is no longer possible to specify the repository (-DarchetypeCatalog) via the command line. For details about this change, please refer to Maven official document - If you wish to use the snapshot repository, you need to add a repository entry to your settings.xml. For example: -{% highlight bash %} -
-{% endunless %} - -You can edit the `groupId`, `artifactId` and `package` if you like. With the above parameters, -Maven will create a folder named `frauddetection` that contains a project with all the dependencies to complete this tutorial. -After importing the project into your editor, you can find a file `FraudDetectionJob.java` (or `FraudDetectionJob.scala`) with the following code which you can run directly inside your IDE. -Try setting break points through out the data stream and run the code in DEBUG mode to get a feeling for how everything works. - -- -{% endhighlight %} -- -apache -- -- -apache -- -- -apache-snapshots -https://repository.apache.org/content/repositories/snapshots/ --- -## Breaking Down the Code - -Let's walk step-by-step through the code of these two files. The `FraudDetectionJob` class defines the data flow of the application and the `FraudDetector` class defines the business logic of the function that detects fraudulent transactions. - -We start describing how the Job is assembled in the `main` method of the `FraudDetectionJob` class. - -#### The Execution Environment - -The first line sets up your `StreamExecutionEnvironment`. -The execution environment is how you set properties for your Job, create your sources, and finally trigger the execution of the Job. - --#### FraudDetectionJob.java - -{% highlight java %} -package spendreport; - -import org.apache.flink.streaming.api.datastream.DataStream; -import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; -import org.apache.flink.walkthrough.common.sink.AlertSink; -import org.apache.flink.walkthrough.common.entity.Alert; -import org.apache.flink.walkthrough.common.entity.Transaction; -import org.apache.flink.walkthrough.common.source.TransactionSource; - -public class FraudDetectionJob { - - public static void main(String[] args) throws Exception { - StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); - - DataStream- -transactions = env - .addSource(new TransactionSource()) - .name("transactions"); - - DataStream alerts = transactions - .keyBy(Transaction::getAccountId) - .process(new FraudDetector()) - .name("fraud-detector"); - - alerts - .addSink(new AlertSink()) - .name("send-alerts"); - - env.execute("Fraud Detection"); - } -} -{% endhighlight %} - -#### FraudDetector.java -{% highlight java %} -package spendreport; - -import org.apache.flink.streaming.api.functions.KeyedProcessFunction; -import org.apache.flink.util.Collector; -import org.apache.flink.walkthrough.common.entity.Alert; -import org.apache.flink.walkthrough.common.entity.Transaction; - -public class FraudDetector extends KeyedProcessFunction collector) throws Exception { - - Alert alert = new Alert(); - alert.setId(transaction.getAccountId()); - - collector.collect(alert); - } -} -{% endhighlight %} - -#### FraudDetectionJob.scala - -{% highlight scala %} -package spendreport - -import org.apache.flink.streaming.api.scala._ -import org.apache.flink.walkthrough.common.sink.AlertSink -import org.apache.flink.walkthrough.common.entity.Alert -import org.apache.flink.walkthrough.common.entity.Transaction -import org.apache.flink.walkthrough.common.source.TransactionSource - -object FraudDetectionJob { - - @throws[Exception] - def main(args: Array[String]): Unit = { - val env: StreamExecutionEnvironment = StreamExecutionEnvironment.getExecutionEnvironment - - val transactions: DataStream[Transaction] = env - .addSource(new TransactionSource) - .name("transactions") - - val alerts: DataStream[Alert] = transactions - .keyBy(transaction => transaction.getAccountId) - .process(new FraudDetector) - .name("fraud-detector") - - alerts - .addSink(new AlertSink) - .name("send-alerts") - - env.execute("Fraud Detection") - } -} -{% endhighlight %} - -#### FraudDetector.scala - -{% highlight scala %} -package spendreport - -import org.apache.flink.streaming.api.functions.KeyedProcessFunction -import org.apache.flink.util.Collector -import org.apache.flink.walkthrough.common.entity.Alert -import org.apache.flink.walkthrough.common.entity.Transaction - -object FraudDetector { - val SMALL_AMOUNT: Double = 1.00 - val LARGE_AMOUNT: Double = 500.00 - val ONE_MINUTE: Long = 60 * 1000L -} - -@SerialVersionUID(1L) -class FraudDetector extends KeyedProcessFunction[Long, Transaction, Alert] { - - @throws[Exception] - def processElement( - transaction: Transaction, - context: KeyedProcessFunction[Long, Transaction, Alert]#Context, - collector: Collector[Alert]): Unit = { - - val alert = new Alert - alert.setId(transaction.getAccountId) - - collector.collect(alert) - } -} - -{% endhighlight %} ---- -#### Creating a Source - -Sources ingest data from external systems, such as Apache Kafka, Rabbit MQ, or Apache Pulsar, into Flink Jobs. -This walkthrough uses a source that generates an infinite stream of credit card transactions for you to process. -Each transaction contains an account ID (`accountId`), timestamp (`timestamp`) of when the transaction occurred, and US$ amount (`amount`). -The `name` attached to the source is just for debugging purposes, so if something goes wrong, we will know where the error originated. - --{% highlight java %} -StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); -{% endhighlight %} -- --{% highlight scala %} -val env: StreamExecutionEnvironment = StreamExecutionEnvironment.getExecutionEnvironment -{% endhighlight %} ---- - -#### Partitioning Events & Detecting Fraud - -The `transactions` stream contains a lot of transactions from a large number of users, such that it needs to be processed in parallel by multiple fraud detection tasks. Since fraud occurs on a per-account basis, you must ensure that all transactions for the same account are processed by the same parallel task of the fraud detector operator. - -To ensure that the same physical task processes all records for a particular key, you can partition a stream using `DataStream#keyBy`. -The `process()` call adds an operator that applies a function to each partitioned element in the stream. -It is common to say the operator immediately after a `keyBy`, in this case `FraudDetector`, is executed within a _keyed context_. - - --{% highlight java %} -DataStream- -transactions = env - .addSource(new TransactionSource()) - .name("transactions"); -{% endhighlight %} - -{% highlight scala %} -val transactions: DataStream[Transaction] = env - .addSource(new TransactionSource) - .name("transactions") -{% endhighlight %} ---- -#### Outputting Results - -A sink writes a `DataStream` to an external system; such as Apache Kafka, Cassandra, and AWS Kinesis. -The `AlertSink` logs each `Alert` record with log level **INFO**, instead of writing it to persistent storage, so you can easily see your results. - --{% highlight java %} -DataStream- -alerts = transactions - .keyBy(Transaction::getAccountId) - .process(new FraudDetector()) - .name("fraud-detector"); -{% endhighlight %} - -{% highlight scala %} -val alerts: DataStream[Alert] = transactions - .keyBy(transaction => transaction.getAccountId) - .process(new FraudDetector) - .name("fraud-detector") -{% endhighlight %} ---- -#### Executing the Job - -Flink applications are built lazily and shipped to the cluster for execution only once fully formed. -Call `StreamExecutionEnvironment#execute` to begin the execution of our Job and give it a name. - --{% highlight java %} -alerts.addSink(new AlertSink()); -{% endhighlight %} -- --{% highlight scala %} -alerts.addSink(new AlertSink) -{% endhighlight %} ---- -#### The Fraud Detector - -The fraud detector is implemented as a `KeyedProcessFunction`. -Its method `KeyedProcessFunction#processElement` is called for every transaction event. -This first version produces an alert on every transaction, which some may say is overly conservative. - -The next steps of this tutorial will guide you to expand the fraud detector with more meaningful business logic. - --{% highlight java %} -env.execute("Fraud Detection"); -{% endhighlight %} -- --{% highlight scala %} -env.execute("Fraud Detection") -{% endhighlight %} ---- -## Writing a Real Application (v1) - -For the first version, the fraud detector should output an alert for any account that makes a small transaction immediately followed by a large one. Where small is anything less than $1.00 and large is more than $500. -Imagine your fraud detector processes the following stream of transactions for a particular account. - --{% highlight java %} -public class FraudDetector extends KeyedProcessFunction- -collector) throws Exception { - - Alert alert = new Alert(); - alert.setId(transaction.getAccountId()); - - collector.collect(alert); - } -} -{% endhighlight %} - -{% highlight scala %} -object FraudDetector { - val SMALL_AMOUNT: Double = 1.00 - val LARGE_AMOUNT: Double = 500.00 - val ONE_MINUTE: Long = 60 * 1000L -} - -@SerialVersionUID(1L) -class FraudDetector extends KeyedProcessFunction[Long, Transaction, Alert] { - - @throws[Exception] - def processElement( - transaction: Transaction, - context: KeyedProcessFunction[Long, Transaction, Alert]#Context, - collector: Collector[Alert]): Unit = { - - val alert = new Alert - alert.setId(transaction.getAccountId) - - collector.collect(alert) - } -} -{% endhighlight %} ---
- -Transactions 3 and 4 should be marked as fraudulent because it is a small transaction, $0.09, followed by a large one, $510. -Alternatively, transactions 7, 8, and 9 are not fraud because the small amount of $0.02 is not immediately followed by the large one; instead, there is an intermediate transaction that breaks the pattern. - -To do this, the fraud detector must _remember_ information across events; a large transaction is only fraudulent if the previous one was small. -Remembering information across events requires [state]({{ site.baseurl }}/concepts/glossary.html#managed-state), and that is why we decided to use a [KeyedProcessFunction]({{ site.baseurl }}/dev/stream/operators/process_function.html). -It provides fine-grained control over both state and time, which will allow us to evolve our algorithm with more complex requirements throughout this walkthrough. - -The most straightforward implementation is a boolean flag that is set whenever a small transaction is processed. -When a large transaction comes through, you can simply check if the flag is set for that account. - -However, merely implementing the flag as a member variable in the `FraudDetector` class will not work. -Flink processes the transactions of multiple accounts with the same object instance of `FraudDetector`, which means if accounts A and B are routed through the same instance of `FraudDetector`, a transaction for account A could set the flag to true and then a transaction for account B could set off a false alert. -We could of course use a data structure like a `Map` to keep track of the flags for individual keys, however, a simple member variable would not be fault-tolerant and all its information be lost in case of a failure. -Hence, the fraud detector would possibly miss alerts if the application ever had to restart to recover from a failure. - -To address these challenges, Flink provides primitives for fault-tolerant state that are almost as easy to use as regular member variables. - -The most basic type of state in Flink is [ValueState]({{ site.baseurl }}/dev/stream/state/state.html#using-managed-keyed-state), a data type that adds fault tolerance to any variable it wraps. -`ValueState` is a form of _keyed state_, meaning it is only available in operators that are applied in a _keyed context_; any operator immediately following `DataStream#keyBy`. -A _keyed state_ of an operator is automatically scoped to the key of the record that is currently processed. -In this example, the key is the account id for the current transaction (as declared by `keyBy()`), and `FraudDetector` maintains an independent state for each account. -`ValueState` is created using a `ValueStateDescriptor` which contains metadata about how Flink should manage the variable. The state should be registered before the function starts processing data. -The right hook for this is the `open()` method. - -
--- -`ValueState` is a wrapper class, similar to `AtomicReference` or `AtomicLong` in the Java standard library. -It provides three methods for interacting with its contents; `update` sets the state, `value` gets the current value, and `clear` deletes its contents. -If the state for a particular key is empty, such as at the beginning of an application or after calling `ValueState#clear`, then `ValueState#value` will return `null`. -Modifications to the object returned by `ValueState#value` are not guaranteed to be recognized by the system, and so all changes must be performed with `ValueState#update`. -Otherwise, fault tolerance is managed automatically by Flink under the hood, and so you can interact with it like with any standard variable. - -Below, you can see an example of how you can use a flag state to track potential fraudulent transactions. - --{% highlight java %} -public class FraudDetector extends KeyedProcessFunction- -flagState; - - @Override - public void open(Configuration parameters) { - ValueStateDescriptor flagDescriptor = new ValueStateDescriptor<>( - "flag", - Types.BOOLEAN); - flagState = getRuntimeContext().getState(flagDescriptor); - } -{% endhighlight %} - -{% highlight scala %} -@SerialVersionUID(1L) -class FraudDetector extends KeyedProcessFunction[Long, Transaction, Alert] { - - @transient private var flagState: ValueState[java.lang.Boolean] = _ - - @throws[Exception] - override def open(parameters: Configuration): Unit = { - val flagDescriptor = new ValueStateDescriptor("flag", Types.BOOLEAN) - flagState = getRuntimeContext.getState(flagDescriptor) - } -{% endhighlight %} ---- -For every transaction, the fraud detector checks the state of the flag for that account. -Remember, `ValueState` is always scoped to the current key, i.e., account. -If the flag is non-null, then the last transaction seen for that account was small, and so if the amount for this transaction is large, then the detector outputs a fraud alert. - -After that check, the flag state is unconditionally cleared. -Either the current transaction caused a fraud alert, and the pattern is over, or the current transaction did not cause an alert, and the pattern is broken and needs to be restarted. - -Finally, the transaction amount is checked to see if it is small. -If so, then the flag is set so that it can be checked by the next event. -Notice that `ValueState-{% highlight java %} -@Override -public void processElement( - Transaction transaction, - Context context, - Collector- -collector) throws Exception { - - // Get the current state for the current key - Boolean lastTransactionWasSmall = flagState.value(); - - // Check if the flag is set - if (lastTransactionWasSmall != null) { - if (transaction.getAmount() > LARGE_AMOUNT) { - // Output an alert downstream - Alert alert = new Alert(); - alert.setId(transaction.getAccountId()); - - collector.collect(alert); - } - - // Clean up our state - flagState.clear(); - } - - if (transaction.getAmount() < SMALL_AMOUNT) { - // Set the flag to true - flagState.update(true); - } -} -{% endhighlight %} - -{% highlight scala %} -override def processElement( - transaction: Transaction, - context: KeyedProcessFunction[Long, Transaction, Alert]#Context, - collector: Collector[Alert]): Unit = { - - // Get the current state for the current key - val lastTransactionWasSmall = flagState.value - - // Check if the flag is set - if (lastTransactionWasSmall != null) { - if (transaction.getAmount > FraudDetector.LARGE_AMOUNT) { - // Output an alert downstream - val alert = new Alert - alert.setId(transaction.getAccountId) - - collector.collect(alert) - } - // Clean up our state - flagState.clear() - } - - if (transaction.getAmount < FraudDetector.SMALL_AMOUNT) { - // set the flag to true - flagState.update(true) - } -} -{% endhighlight %} --` actually has three states, unset ( `null`), `true`, and `false`, because all `ValueState`'s are nullable. -This job only makes use of unset ( `null`) and `true` to check whether the flag is set or not. - -## Fraud Detector v2: State + Time = ❤️ - -Scammers don't wait long to make their large purchase to reduce the chances their test transaction is noticed. -For example, suppose you wanted to set a 1 minute timeout to your fraud detector; i.e., in the previous example transactions 3 and 4 would only be considered fraud if they occurred within 1 minute of each other. -Flink's `KeyedProcessFunction` allows you to set timers which invoke a callback method at some point in time in the future. - -Let's see how we can modify our Job to comply with our new requirements: - -* Whenever the flag is set to `true`, also set a timer for 1 minute in the future. -* When the timer fires, reset the flag by clearing its state. -* If the flag is ever cleared the timer should be canceled. - -To cancel a timer, you have to remember what time it is set for, and remembering implies state, so you will begin by creating a timer state along with your flag state. - - -- -`KeyedProcessFunction#processElement` is called with a `Context` that contains a timer service. -The timer service can be used to query the current time, register timers, and delete timers. -With this, you can set a timer for 1 minute in the future every time the flag is set and store the timestamp in `timerState`. - --{% highlight java %} -private transient ValueState- -flagState; -private transient ValueState timerState; - -@Override -public void open(Configuration parameters) { - ValueStateDescriptor flagDescriptor = new ValueStateDescriptor<>( - "flag", - Types.BOOLEAN); - flagState = getRuntimeContext().getState(flagDescriptor); - - ValueStateDescriptor timerDescriptor = new ValueStateDescriptor<>( - "timer-state", - Types.LONG); - timerState = getRuntimeContext().getState(timerDescriptor); -} -{% endhighlight %} - -{% highlight scala %} -@SerialVersionUID(1L) -class FraudDetector extends KeyedProcessFunction[Long, Transaction, Alert] { - - @transient private var flagState: ValueState[java.lang.Boolean] = _ - @transient private var timerState: ValueState[java.lang.Long] = _ - - @throws[Exception] - override def open(parameters: Configuration): Unit = { - val flagDescriptor = new ValueStateDescriptor("flag", Types.BOOLEAN) - flagState = getRuntimeContext.getState(flagDescriptor) - - val timerDescriptor = new ValueStateDescriptor("timer-state", Types.LONG) - timerState = getRuntimeContext.getState(timerDescriptor) - } -{% endhighlight %} ---- -Processing time is wall clock time, and is determined by the system clock of the machine running the operator. - -When a timer fires, it calls `KeyedProcessFunction#onTimer`. -Overriding this method is how you can implement your callback to reset the flag. - --{% highlight java %} -if (transaction.getAmount() < SMALL_AMOUNT) { - // set the flag to true - flagState.update(true); - - // set the timer and timer state - long timer = context.timerService().currentProcessingTime() + ONE_MINUTE; - context.timerService().registerProcessingTimeTimer(timer); - timerState.update(timer); -} -{% endhighlight %} ---{% highlight scala %} -if (transaction.getAmount < FraudDetector.SMALL_AMOUNT) { - // set the flag to true - flagState.update(true) - - // set the timer and timer state - val timer = context.timerService.currentProcessingTime + FraudDetector.ONE_MINUTE - context.timerService.registerProcessingTimeTimer(timer) - timerState.update(timer) -} -{% endhighlight %} ---- -Finally, to cancel the timer, you need to delete the registered timer and delete the timer state. -You can wrap this in a helper method and call this method instead of `flagState.clear()`. - --{% highlight java %} -@Override -public void onTimer(long timestamp, OnTimerContext ctx, Collector-out) { - // remove flag after 1 minute - timerState.clear(); - flagState.clear(); -} -{% endhighlight %} - -{% highlight scala %} -override def onTimer( - timestamp: Long, - ctx: KeyedProcessFunction[Long, Transaction, Alert]#OnTimerContext, - out: Collector[Alert]): Unit = { - // remove flag after 1 minute - timerState.clear() - flagState.clear() -} -{% endhighlight %} ---- -And that's it, a fully functional, stateful, distributed streaming application! - -## Final Application - --{% highlight java %} -private void cleanUp(Context ctx) throws Exception { - // delete timer - Long timer = timerState.value(); - ctx.timerService().deleteProcessingTimeTimer(timer); - - // clean up all state - timerState.clear(); - flagState.clear(); -} -{% endhighlight %} -- --{% highlight scala %} -@throws[Exception] -private def cleanUp(ctx: KeyedProcessFunction[Long, Transaction, Alert]#Context): Unit = { - // delete timer - val timer = timerState.value - ctx.timerService.deleteProcessingTimeTimer(timer) - - // clean up all states - timerState.clear() - flagState.clear() -} -{% endhighlight %} ---- -### Expected Output - -Running this code with the provided `TransactionSource` will emit fraud alerts for account 3. -You should see the following output in your task manager logs: - -{% highlight bash %} -2019-08-19 14:22:06,220 INFO org.apache.flink.walkthrough.common.sink.AlertSink - Alert{id=3} -2019-08-19 14:22:11,383 INFO org.apache.flink.walkthrough.common.sink.AlertSink - Alert{id=3} -2019-08-19 14:22:16,551 INFO org.apache.flink.walkthrough.common.sink.AlertSink - Alert{id=3} -2019-08-19 14:22:21,723 INFO org.apache.flink.walkthrough.common.sink.AlertSink - Alert{id=3} -2019-08-19 14:22:26,896 INFO org.apache.flink.walkthrough.common.sink.AlertSink - Alert{id=3} -{% endhighlight %} diff --git a/docs/getting-started/walkthroughs/datastream_api.zh.md b/docs/getting-started/walkthroughs/datastream_api.zh.md deleted file mode 100644 index f21004d495a01a..00000000000000 --- a/docs/getting-started/walkthroughs/datastream_api.zh.md +++ /dev/null @@ -1,946 +0,0 @@ ---- -title: "DataStream API" -nav-id: datastreamwalkthrough -nav-title: 'DataStream API' -nav-parent_id: walkthroughs -nav-pos: 1 ---- - - -Apache Flink 提供了 DataStream API 来实现稳定可靠的、有状态的流处理应用程序。 -Flink 支持对状态和时间的细粒度控制，以此来实现复杂的事件驱动数据处理系统。 -这个入门指导手册讲述了如何通过 Flink DataStream API 来实现一个有状态流处理程序。 - -* This will be replaced by the TOC -{:toc} - -## 你要搭建一个什么系统 - -在当今数字时代，信用卡欺诈行为越来越被重视。 -罪犯可以通过诈骗或者入侵安全级别较低系统来盗窃信用卡卡号。 -用盗得的信用卡进行很小额度的例如一美元或者更小额度的消费进行测试。 -如果测试消费成功，那么他们就会用这个信用卡进行大笔消费，来购买一些他们希望得到的，或者可以倒卖的财物。 - -在这个教程中，你将会建立一个针对可疑信用卡交易行为的反欺诈检测系统。 -通过使用一组简单的规则，你将了解到 Flink 如何为我们实现复杂业务逻辑并实时执行。 - -## 准备条件 - -这个代码练习假定你对 Java 或 Scala 有一定的了解，当然，如果你之前使用的是其他开发语言，你也应该能够跟随本教程进行学习。 - -## 困难求助 - -如果遇到困难，可以参考 [社区支持资源](https://flink.apache.org/zh/gettinghelp.html)。 -当然也可以在邮件列表提问，Flink 的 [用户邮件列表](https://flink.apache.org/zh/community.html#mailing-lists) 一直被评为所有Apache项目中最活跃的一个，这也是快速获得帮助的好方法。 - -## 怎样跟着教程练习 - -首先，你需要在你的电脑上准备以下环境： - -* Java 8 or 11 -* Maven - -一个准备好的 Flink Maven Archetype 能够快速创建一个包含了必要依赖的 Flink 程序骨架，基于此，你可以把精力集中在编写业务逻辑上即可。 -这些已包含的依赖包括 `flink-streaming-java`、`flink-walkthrough-common` 等，他们分别是 Flink 应用程序的核心依赖项和这个代码练习需要的数据生成器，当然还包括其他本代码练习所依赖的类。 - -{% panel **说明:** 为简洁起见，本练习中的代码块中可能不包含完整的类路径。完整的类路径可以在文档底部 [链接](#完整的程序) 中找到。 %} - --{% highlight java %} -package spendreport; - -import org.apache.flink.api.common.state.ValueState; -import org.apache.flink.api.common.state.ValueStateDescriptor; -import org.apache.flink.api.common.typeinfo.Types; -import org.apache.flink.configuration.Configuration; -import org.apache.flink.streaming.api.functions.KeyedProcessFunction; -import org.apache.flink.util.Collector; -import org.apache.flink.walkthrough.common.entity.Alert; -import org.apache.flink.walkthrough.common.entity.Transaction; - -public class FraudDetector extends KeyedProcessFunction- -flagState; - private transient ValueState timerState; - - @Override - public void open(Configuration parameters) { - ValueStateDescriptor flagDescriptor = new ValueStateDescriptor<>( - "flag", - Types.BOOLEAN); - flagState = getRuntimeContext().getState(flagDescriptor); - - ValueStateDescriptor timerDescriptor = new ValueStateDescriptor<>( - "timer-state", - Types.LONG); - timerState = getRuntimeContext().getState(timerDescriptor); - } - - @Override - public void processElement( - Transaction transaction, - Context context, - Collector collector) throws Exception { - - // Get the current state for the current key - Boolean lastTransactionWasSmall = flagState.value(); - - // Check if the flag is set - if (lastTransactionWasSmall != null) { - if (transaction.getAmount() > LARGE_AMOUNT) { - //Output an alert downstream - Alert alert = new Alert(); - alert.setId(transaction.getAccountId()); - - collector.collect(alert); - } - // Clean up our state - cleanUp(context); - } - - if (transaction.getAmount() < SMALL_AMOUNT) { - // set the flag to true - flagState.update(true); - - long timer = context.timerService().currentProcessingTime() + ONE_MINUTE; - context.timerService().registerProcessingTimeTimer(timer); - - timerState.update(timer); - } - } - - @Override - public void onTimer(long timestamp, OnTimerContext ctx, Collector out) { - // remove flag after 1 minute - timerState.clear(); - flagState.clear(); - } - - private void cleanUp(Context ctx) throws Exception { - // delete timer - Long timer = timerState.value(); - ctx.timerService().deleteProcessingTimeTimer(timer); - - // clean up all state - timerState.clear(); - flagState.clear(); - } -} -{% endhighlight %} - -{% highlight scala %} -package spendreport - -import org.apache.flink.api.common.state.{ValueState, ValueStateDescriptor} -import org.apache.flink.api.scala.typeutils.Types -import org.apache.flink.configuration.Configuration -import org.apache.flink.streaming.api.functions.KeyedProcessFunction -import org.apache.flink.util.Collector -import org.apache.flink.walkthrough.common.entity.Alert -import org.apache.flink.walkthrough.common.entity.Transaction - -object FraudDetector { - val SMALL_AMOUNT: Double = 1.00 - val LARGE_AMOUNT: Double = 500.00 - val ONE_MINUTE: Long = 60 * 1000L -} - -@SerialVersionUID(1L) -class FraudDetector extends KeyedProcessFunction[Long, Transaction, Alert] { - - @transient private var flagState: ValueState[java.lang.Boolean] = _ - @transient private var timerState: ValueState[java.lang.Long] = _ - - @throws[Exception] - override def open(parameters: Configuration): Unit = { - val flagDescriptor = new ValueStateDescriptor("flag", Types.BOOLEAN) - flagState = getRuntimeContext.getState(flagDescriptor) - - val timerDescriptor = new ValueStateDescriptor("timer-state", Types.LONG) - timerState = getRuntimeContext.getState(timerDescriptor) - } - - override def processElement( - transaction: Transaction, - context: KeyedProcessFunction[Long, Transaction, Alert]#Context, - collector: Collector[Alert]): Unit = { - - // Get the current state for the current key - val lastTransactionWasSmall = flagState.value - - // Check if the flag is set - if (lastTransactionWasSmall != null) { - if (transaction.getAmount > FraudDetector.LARGE_AMOUNT) { - // Output an alert downstream - val alert = new Alert - alert.setId(transaction.getAccountId) - - collector.collect(alert) - } - // Clean up our state - cleanUp(context) - } - - if (transaction.getAmount < FraudDetector.SMALL_AMOUNT) { - // set the flag to true - flagState.update(true) - val timer = context.timerService.currentProcessingTime + FraudDetector.ONE_MINUTE - - context.timerService.registerProcessingTimeTimer(timer) - timerState.update(timer) - } - } - - override def onTimer( - timestamp: Long, - ctx: KeyedProcessFunction[Long, Transaction, Alert]#OnTimerContext, - out: Collector[Alert]): Unit = { - // remove flag after 1 minute - timerState.clear() - flagState.clear() - } - - @throws[Exception] - private def cleanUp(ctx: KeyedProcessFunction[Long, Transaction, Alert]#Context): Unit = { - // delete timer - val timer = timerState.value - ctx.timerService.deleteProcessingTimeTimer(timer) - - // clean up all states - timerState.clear() - flagState.clear() - } -} -{% endhighlight %} ---- -{% unless site.is_stable %} --{% highlight bash %} -$ mvn archetype:generate \ - -DarchetypeGroupId=org.apache.flink \ - -DarchetypeArtifactId=flink-walkthrough-datastream-java \{% unless site.is_stable %} - -DarchetypeCatalog=https://repository.apache.org/content/repositories/snapshots/ \{% endunless %} - -DarchetypeVersion={{ site.version }} \ - -DgroupId=frauddetection \ - -DartifactId=frauddetection \ - -Dversion=0.1 \ - -Dpackage=spendreport \ - -DinteractiveMode=false -{% endhighlight %} ---{% highlight bash %} -$ mvn archetype:generate \ - -DarchetypeGroupId=org.apache.flink \ - -DarchetypeArtifactId=flink-walkthrough-datastream-scala \{% unless site.is_stable %} - -DarchetypeCatalog=https://repository.apache.org/content/repositories/snapshots/ \{% endunless %} - -DarchetypeVersion={{ site.version }} \ - -DgroupId=frauddetection \ - -DartifactId=frauddetection \ - -Dversion=0.1 \ - -Dpackage=spendreport \ - -DinteractiveMode=false -{% endhighlight %} --- 注意：Maven 3.0 及更高版本，不再支持通过命令行指定仓库（-DarchetypeCatalog）。有关这个改动的详细信息， - 请参阅 Maven 官方文档 - 如果你希望使用快照仓库，则需要在 settings.xml 文件中添加一个仓库条目。例如： -{% highlight bash %} -
-{% endunless %} - -你可以根据自己的情况修改 `groupId`、 `artifactId` 和 `package`。通过这三个参数， -Maven 将会创建一个名为 `frauddetection` 的文件夹，包含了所有依赖的整个工程项目将会位于该文件夹下。 -将工程目录导入到你的开发环境之后，你可以找到 `FraudDetectionJob.java` （或 `FraudDetectionJob.scala`） 代码文件，文件中的代码如下所示。你可以在 IDE 中直接运行这个文件。 -同时，你可以试着在数据流中设置一些断点或者以 DEBUG 模式来运行程序，体验 Flink 是如何运行的。 - -- -{% endhighlight %} -- -apache -- -- -apache -- -- -apache-snapshots -https://repository.apache.org/content/repositories/snapshots/ --- -## 代码分析 - -让我们一步步地来分析一下这两个代码文件。`FraudDetectionJob` 类定义了程序的数据流，而 `FraudDetector` 类定义了欺诈交易检测的业务逻辑。 - -下面我们开始讲解整个 Job 是如何组装到 `FraudDetectionJob` 类的 `main` 函数中的。 - -#### 执行环境 - -第一行的 `StreamExecutionEnvironment` 用于设置你的执行环境。 -任务执行环境用于定义任务的属性、创建数据源以及最终启动任务的执行。 - --#### FraudDetectionJob.java - -{% highlight java %} -package spendreport; - -import org.apache.flink.streaming.api.datastream.DataStream; -import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; -import org.apache.flink.walkthrough.common.sink.AlertSink; -import org.apache.flink.walkthrough.common.entity.Alert; -import org.apache.flink.walkthrough.common.entity.Transaction; -import org.apache.flink.walkthrough.common.source.TransactionSource; - -public class FraudDetectionJob { - - public static void main(String[] args) throws Exception { - StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); - - DataStream- -transactions = env - .addSource(new TransactionSource()) - .name("transactions"); - - DataStream alerts = transactions - .keyBy(Transaction::getAccountId) - .process(new FraudDetector()) - .name("fraud-detector"); - - alerts - .addSink(new AlertSink()) - .name("send-alerts"); - - env.execute("Fraud Detection"); - } -} -{% endhighlight %} - -#### FraudDetector.java -{% highlight java %} -package spendreport; - -import org.apache.flink.streaming.api.functions.KeyedProcessFunction; -import org.apache.flink.util.Collector; -import org.apache.flink.walkthrough.common.entity.Alert; -import org.apache.flink.walkthrough.common.entity.Transaction; - -public class FraudDetector extends KeyedProcessFunction collector) throws Exception { - - Alert alert = new Alert(); - alert.setId(transaction.getAccountId()); - - collector.collect(alert); - } -} -{% endhighlight %} - -#### FraudDetectionJob.scala - -{% highlight scala %} -package spendreport - -import org.apache.flink.streaming.api.scala._ -import org.apache.flink.walkthrough.common.sink.AlertSink -import org.apache.flink.walkthrough.common.entity.Alert -import org.apache.flink.walkthrough.common.entity.Transaction -import org.apache.flink.walkthrough.common.source.TransactionSource - -object FraudDetectionJob { - - @throws[Exception] - def main(args: Array[String]): Unit = { - val env: StreamExecutionEnvironment = StreamExecutionEnvironment.getExecutionEnvironment - - val transactions: DataStream[Transaction] = env - .addSource(new TransactionSource) - .name("transactions") - - val alerts: DataStream[Alert] = transactions - .keyBy(transaction => transaction.getAccountId) - .process(new FraudDetector) - .name("fraud-detector") - - alerts - .addSink(new AlertSink) - .name("send-alerts") - - env.execute("Fraud Detection") - } -} -{% endhighlight %} - -#### FraudDetector.scala - -{% highlight scala %} -package spendreport - -import org.apache.flink.streaming.api.functions.KeyedProcessFunction -import org.apache.flink.util.Collector -import org.apache.flink.walkthrough.common.entity.Alert -import org.apache.flink.walkthrough.common.entity.Transaction - -object FraudDetector { - val SMALL_AMOUNT: Double = 1.00 - val LARGE_AMOUNT: Double = 500.00 - val ONE_MINUTE: Long = 60 * 1000L -} - -@SerialVersionUID(1L) -class FraudDetector extends KeyedProcessFunction[Long, Transaction, Alert] { - - @throws[Exception] - def processElement( - transaction: Transaction, - context: KeyedProcessFunction[Long, Transaction, Alert]#Context, - collector: Collector[Alert]): Unit = { - - val alert = new Alert - alert.setId(transaction.getAccountId) - - collector.collect(alert) - } -} - -{% endhighlight %} ---- -#### 创建数据源 - -数据源从外部系统例如 Apache Kafka、Rabbit MQ 或者 Apache Pulsar 接收数据，然后将数据送到 Flink 程序中。 -这个代码练习使用的是一个能够无限循环生成信用卡模拟交易数据的数据源。 -每条交易数据包括了信用卡 ID （`accountId`），交易发生的时间 （`timestamp`） 以及交易的金额（`amount`）。 -绑定到数据源上的 `name` 属性是为了调试方便，如果发生一些异常，我们能够通过它快速定位问题发生在哪里。 - --{% highlight java %} -StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); -{% endhighlight %} -- --{% highlight scala %} -val env: StreamExecutionEnvironment = StreamExecutionEnvironment.getExecutionEnvironment -{% endhighlight %} ---- - -#### 对事件分区 & 欺诈检测 - -`transactions` 这个数据流包含了大量的用户交易数据，需要被划分到多个并发上进行欺诈检测处理。由于欺诈行为的发生是基于某一个账户的，所以，必须要要保证同一个账户的所有交易行为数据要被同一个并发的 task 进行处理。 - -为了保证同一个 task 处理同一个 key 的所有数据，你可以使用 `DataStream#keyBy` 对流进行分区。 -`process()` 函数对流绑定了一个操作，这个操作将会对流上的每一个消息调用所定义好的函数。 -通常，一个操作会紧跟着 `keyBy` 被调用，在这个例子中，这个操作是`FraudDetector`，该操作是在一个 _keyed context_ 上执行的。 - - --{% highlight java %} -DataStream- -transactions = env - .addSource(new TransactionSource()) - .name("transactions"); -{% endhighlight %} - -{% highlight scala %} -val transactions: DataStream[Transaction] = env - .addSource(new TransactionSource) - .name("transactions") -{% endhighlight %} ---- -#### 输出结果 - -sink 会将 `DataStream` 写出到外部系统，例如 Apache Kafka、Cassandra 或者 AWS Kinesis 等。 -`AlertSink` 使用 **INFO** 的日志级别打印每一个 `Alert` 的数据记录，而不是将其写入持久存储，以便你可以方便地查看结果。 - --{% highlight java %} -DataStream- -alerts = transactions - .keyBy(Transaction::getAccountId) - .process(new FraudDetector()) - .name("fraud-detector"); -{% endhighlight %} - -{% highlight scala %} -val alerts: DataStream[Alert] = transactions - .keyBy(transaction => transaction.getAccountId) - .process(new FraudDetector) - .name("fraud-detector") -{% endhighlight %} ---- -#### 运行作业 - -Flink 程序是懒加载的，并且只有在完全搭建好之后，才能够发布到集群上执行。 -调用 `StreamExecutionEnvironment#execute` 时给任务传递一个任务名参数，就可以开始运行任务。 - --{% highlight java %} -alerts.addSink(new AlertSink()); -{% endhighlight %} -- --{% highlight scala %} -alerts.addSink(new AlertSink) -{% endhighlight %} ---- -#### 欺诈检测器 - -欺诈检查类 `FraudDetector` 是 `KeyedProcessFunction` 接口的一个实现。 -他的方法 `KeyedProcessFunction#processElement` 将会在每个交易事件上被调用。 -这个程序里边会对每笔交易发出警报，有人可能会说这做报过于保守了。 - -本教程的后续步骤将指导你对这个欺诈检测器进行更有意义的业务逻辑扩展。 - --{% highlight java %} -env.execute("Fraud Detection"); -{% endhighlight %} -- --{% highlight scala %} -env.execute("Fraud Detection") -{% endhighlight %} ---- -## 实现一个真正的应用程序 - -我们先实现第一版报警程序，对于一个账户，如果出现小于 $1 美元的交易后紧跟着一个大于 $500 的交易，就输出一个报警信息。 - -假设你的欺诈检测器所处理的交易数据如下： - --{% highlight java %} -public class FraudDetector extends KeyedProcessFunction- -collector) throws Exception { - - Alert alert = new Alert(); - alert.setId(transaction.getAccountId()); - - collector.collect(alert); - } -} -{% endhighlight %} - -{% highlight scala %} -object FraudDetector { - val SMALL_AMOUNT: Double = 1.00 - val LARGE_AMOUNT: Double = 500.00 - val ONE_MINUTE: Long = 60 * 1000L -} - -@SerialVersionUID(1L) -class FraudDetector extends KeyedProcessFunction[Long, Transaction, Alert] { - - @throws[Exception] - def processElement( - transaction: Transaction, - context: KeyedProcessFunction[Long, Transaction, Alert]#Context, - collector: Collector[Alert]): Unit = { - - val alert = new Alert - alert.setId(transaction.getAccountId) - - collector.collect(alert) - } -} -{% endhighlight %} ---
- -交易 3 和交易 4 应该被标记为欺诈行为，因为交易 3 是一个 $0.09 的小额交易，而紧随着的交易 4 是一个 $510 的大额交易。 -另外，交易 7、8 和 交易 9 就不属于欺诈交易了，因为在交易 7 这个 $0.02 的小额交易之后，并没有跟随一个大额交易，而是一个金额适中的交易，这使得交易 7 到 交易 9 不属于欺诈行为。 - -欺诈检测器需要在多个交易事件之间记住一些信息。仅当一个大额的交易紧随一个小额交易的情况发生时，这个大额交易才被认为是欺诈交易。 -在多个事件之间存储信息就需要使用到 [状态]({{ site.baseurl }}/zh/concepts/glossary.html#managed-state)，这也是我们选择使用 [KeyedProcessFunction]({{ site.baseurl }}/zh/dev/stream/operators/process_function.html) 的原因。 -它能够同时提供对状态和时间的细粒度操作，这使得我们能够在接下来的代码练习中实现更复杂的算法。 - -最直接的实现方式是使用一个 boolean 型的标记状态来表示是否刚处理过一个小额交易。 -当处理到该账户的一个大额交易时，你只需要检查这个标记状态来确认上一个交易是是否小额交易即可。 - -然而，仅使用一个标记作为 `FraudDetector` 的类成员来记录账户的上一个交易状态是不准确的。 -Flink 会在同一个 `FraudDetector` 的并发实例中处理多个账户的交易数据，假设，当账户 A 和账户 B 的数据被分发的同一个并发实例上处理时，账户 A 的小额交易行为可能会将标记状态设置为真，随后账户 B 的大额交易可能会被误判为欺诈交易。 -当然，我们可以使用如 `Map` 这样的数据结构来保存每一个账户的状态，但是常规的类成员变量是无法做到容错处理的，当任务失败重启后，之前的状态信息将会丢失。 -这样的话，如果程序曾出现过失败重启的情况，将会漏掉一些欺诈报警。 - -为了应对这个问题，Flink 提供了一套支持容错状态的原语，这些原语几乎与常规成员变量一样易于使用。 - -Flink 中最基础的状态类型是 [ValueState]({{ site.baseurl }}/zh/dev/stream/state/state.html#using-managed-keyed-state)，这是一种能够为被其封装的变量添加容错能力的类型。 -`ValueState` 是一种 _keyed state_，也就是说它只能被用于 _keyed context_ 提供的 operator 中，即所有能够紧随 `DataStream#keyBy` 之后被调用的operator。 -一个 operator 中的 _keyed state_ 的作用域默认是属于它所属的 key 的。 -这个例子中，key 就是当前正在处理的交易行为所属的信用卡账户（key 传入 keyBy() 函数调用），而 `FraudDetector` 维护了每个帐户的标记状态。 -`ValueState` 需要使用 `ValueStateDescriptor` 来创建，`ValueStateDescriptor` 包含了 Flink 如何管理变量的一些元数据信息。状态在使用之前需要先被注册。 -状态需要使用 `open()` 函数来注册状态。 - -
--- -`ValueState` 是一个包装类，类似于 Java 标准库里边的 `AtomicReference` 和 `AtomicLong`。 -它提供了三个用于交互的方法。`update` 用于更新状态，`value` 用于获取状态值，还有 `clear` 用于清空状态。 -如果一个 key 还没有状态，例如当程序刚启动或者调用过 `ValueState#clear` 方法时，`ValueState#value` 将会返回 `null`。 -如果需要更新状态，需要调用 `ValueState#update` 方法，直接更改 `ValueState#value` 的返回值可能不会被系统识别。 -容错处理将在 Flink 后台自动管理，你可以像与常规变量那样与状态变量进行交互。 - -下边的示例，说明了如何使用标记状态来追踪可能的欺诈交易行为。 - --{% highlight java %} -public class FraudDetector extends KeyedProcessFunction- -flagState; - - @Override - public void open(Configuration parameters) { - ValueStateDescriptor flagDescriptor = new ValueStateDescriptor<>( - "flag", - Types.BOOLEAN); - flagState = getRuntimeContext().getState(flagDescriptor); - } -{% endhighlight %} - -{% highlight scala %} -@SerialVersionUID(1L) -class FraudDetector extends KeyedProcessFunction[Long, Transaction, Alert] { - - @transient private var flagState: ValueState[java.lang.Boolean] = _ - - @throws[Exception] - override def open(parameters: Configuration): Unit = { - val flagDescriptor = new ValueStateDescriptor("flag", Types.BOOLEAN) - flagState = getRuntimeContext.getState(flagDescriptor) - } -{% endhighlight %} ---- -对于每笔交易，欺诈检测器都会检查该帐户的标记状态。 -请记住，`ValueState` 的作用域始终限于当前的 key，即信用卡帐户。 -如果标记状态不为空，则该帐户的上一笔交易是小额的，因此，如果当前这笔交易的金额很大，那么检测程序将输出报警信息。 - -在检查之后，不论是什么状态，都需要被清空。 -不管是当前交易触发了欺诈报警而造成模式的结束，还是当前交易没有触发报警而造成模式的中断，都需要重新开始新的模式检测。 - -最后，检查当前交易的金额是否属于小额交易。 -如果是，那么需要设置标记状态，以便可以在下一个事件中对其进行检查。 -注意，`ValueState-{% highlight java %} -@Override -public void processElement( - Transaction transaction, - Context context, - Collector- -collector) throws Exception { - - // Get the current state for the current key - Boolean lastTransactionWasSmall = flagState.value(); - - // Check if the flag is set - if (lastTransactionWasSmall != null) { - if (transaction.getAmount() > LARGE_AMOUNT) { - // Output an alert downstream - Alert alert = new Alert(); - alert.setId(transaction.getAccountId()); - - collector.collect(alert); - } - - // Clean up our state - flagState.clear(); - } - - if (transaction.getAmount() < SMALL_AMOUNT) { - // Set the flag to true - flagState.update(true); - } -} -{% endhighlight %} - -{% highlight scala %} -override def processElement( - transaction: Transaction, - context: KeyedProcessFunction[Long, Transaction, Alert]#Context, - collector: Collector[Alert]): Unit = { - - // Get the current state for the current key - val lastTransactionWasSmall = flagState.value - - // Check if the flag is set - if (lastTransactionWasSmall != null) { - if (transaction.getAmount > FraudDetector.LARGE_AMOUNT) { - // Output an alert downstream - val alert = new Alert - alert.setId(transaction.getAccountId) - - collector.collect(alert) - } - // Clean up our state - flagState.clear() - } - - if (transaction.getAmount < FraudDetector.SMALL_AMOUNT) { - // set the flag to true - flagState.update(true) - } -} -{% endhighlight %} --` 实际上有 3 种状态：unset (`null`)，`true`，和 `false`，`ValueState` 是允许空值的。 -我们的程序只使用了 unset (`null`) 和 `true` 两种来判断标记状态被设置了与否。 - -## 欺诈检测器 v2：状态 + 时间 = ❤️ - -骗子们在小额交易后不会等很久就进行大额消费，这样可以降低小额测试交易被发现的几率。 -比如，假设你为欺诈检测器设置了一分钟的超时，对于上边的例子，交易 3 和 交易 4 只有间隔在一分钟之内才被认为是欺诈交易。 -Flink 中的 `KeyedProcessFunction` 允许您设置计时器，该计时器在将来的某个时间点执行回调函数。 - -让我们看看如何修改程序以符合我们的新要求： - -* 当标记状态被设置为 `true` 时，设置一个在当前时间一分钟后触发的定时器。 -* 当定时器被触发时，重置标记状态。 -* 当标记状态被重置时，删除定时器。 - -要删除一个定时器，你需要记录这个定时器的触发时间，这同样需要状态来实现，所以你需要在标记状态后也创建一个记录定时器时间的状态。 - - -- -`KeyedProcessFunction#processElement` 需要使用提供了定时器服务的 `Context` 来调用。 -定时器服务可以用于查询当前时间、注册定时器和删除定时器。 -使用它，你可以在标记状态被设置时，也设置一个当前时间一分钟后触发的定时器，同时，将触发时间保存到 `timerState` 状态中。 - --{% highlight java %} -private transient ValueState- -flagState; -private transient ValueState timerState; - -@Override -public void open(Configuration parameters) { - ValueStateDescriptor flagDescriptor = new ValueStateDescriptor<>( - "flag", - Types.BOOLEAN); - flagState = getRuntimeContext().getState(flagDescriptor); - - ValueStateDescriptor timerDescriptor = new ValueStateDescriptor<>( - "timer-state", - Types.LONG); - timerState = getRuntimeContext().getState(timerDescriptor); -} -{% endhighlight %} - -{% highlight scala %} -@SerialVersionUID(1L) -class FraudDetector extends KeyedProcessFunction[Long, Transaction, Alert] { - - @transient private var flagState: ValueState[java.lang.Boolean] = _ - @transient private var timerState: ValueState[java.lang.Long] = _ - - @throws[Exception] - override def open(parameters: Configuration): Unit = { - val flagDescriptor = new ValueStateDescriptor("flag", Types.BOOLEAN) - flagState = getRuntimeContext.getState(flagDescriptor) - - val timerDescriptor = new ValueStateDescriptor("timer-state", Types.LONG) - timerState = getRuntimeContext.getState(timerDescriptor) - } -{% endhighlight %} ---- -处理时间是本地时钟时间，这是由运行任务的服务器的系统时间来决定的。 - -当定时器触发时，将会调用 `KeyedProcessFunction#onTimer` 方法。 -通过重写这个方法来实现一个你自己的重置状态的回调逻辑。 - --{% highlight java %} -if (transaction.getAmount() < SMALL_AMOUNT) { - // set the flag to true - flagState.update(true); - - // set the timer and timer state - long timer = context.timerService().currentProcessingTime() + ONE_MINUTE; - context.timerService().registerProcessingTimeTimer(timer); - timerState.update(timer); -} -{% endhighlight %} ---{% highlight scala %} -if (transaction.getAmount < FraudDetector.SMALL_AMOUNT) { - // set the flag to true - flagState.update(true) - - // set the timer and timer state - val timer = context.timerService.currentProcessingTime + FraudDetector.ONE_MINUTE - context.timerService.registerProcessingTimeTimer(timer) - timerState.update(timer) -} -{% endhighlight %} ---- -最后，如果要取消定时器，你需要删除已经注册的定时器，并同时清空保存定时器的状态。 -你可以把这些逻辑封装到一个助手函数中，而不是直接调用 `flagState.clear()`。 - --{% highlight java %} -@Override -public void onTimer(long timestamp, OnTimerContext ctx, Collector-out) { - // remove flag after 1 minute - timerState.clear(); - flagState.clear(); -} -{% endhighlight %} - -{% highlight scala %} -override def onTimer( - timestamp: Long, - ctx: KeyedProcessFunction[Long, Transaction, Alert]#OnTimerContext, - out: Collector[Alert]): Unit = { - // remove flag after 1 minute - timerState.clear() - flagState.clear() -} -{% endhighlight %} ---- -这就是一个功能完备的，有状态的分布式流处理程序了。 - -## 完整的程序 - --{% highlight java %} -private void cleanUp(Context ctx) throws Exception { - // delete timer - Long timer = timerState.value(); - ctx.timerService().deleteProcessingTimeTimer(timer); - - // clean up all state - timerState.clear(); - flagState.clear(); -} -{% endhighlight %} -- --{% highlight scala %} -@throws[Exception] -private def cleanUp(ctx: KeyedProcessFunction[Long, Transaction, Alert]#Context): Unit = { - // delete timer - val timer = timerState.value - ctx.timerService.deleteProcessingTimeTimer(timer) - - // clean up all states - timerState.clear() - flagState.clear() -} -{% endhighlight %} ---- -### 期望的结果 - -使用已准备好的 `TransactionSource` 数据源运行这个代码，将会检测到账户 3 的欺诈行为，并输出报警信息。 -你将能够在你的 task manager 的日志中看到下边输出： - -{% highlight bash %} -2019-08-19 14:22:06,220 INFO org.apache.flink.walkthrough.common.sink.AlertSink - Alert{id=3} -2019-08-19 14:22:11,383 INFO org.apache.flink.walkthrough.common.sink.AlertSink - Alert{id=3} -2019-08-19 14:22:16,551 INFO org.apache.flink.walkthrough.common.sink.AlertSink - Alert{id=3} -2019-08-19 14:22:21,723 INFO org.apache.flink.walkthrough.common.sink.AlertSink - Alert{id=3} -2019-08-19 14:22:26,896 INFO org.apache.flink.walkthrough.common.sink.AlertSink - Alert{id=3} -{% endhighlight %} diff --git a/docs/getting-started/walkthroughs/index.md b/docs/getting-started/walkthroughs/index.md deleted file mode 100644 index 055f1b85050c4e..00000000000000 --- a/docs/getting-started/walkthroughs/index.md +++ /dev/null @@ -1,25 +0,0 @@ ---- -title: "Code Walkthroughs" -nav-id: walkthroughs -nav-title: 'Code Walkthroughs' -nav-parent_id: getting-started -nav-pos: 20 ---- - \ No newline at end of file diff --git a/docs/getting-started/walkthroughs/index.zh.md b/docs/getting-started/walkthroughs/index.zh.md deleted file mode 100644 index 055f1b85050c4e..00000000000000 --- a/docs/getting-started/walkthroughs/index.zh.md +++ /dev/null @@ -1,25 +0,0 @@ ---- -title: "Code Walkthroughs" -nav-id: walkthroughs -nav-title: 'Code Walkthroughs' -nav-parent_id: getting-started -nav-pos: 20 ---- - \ No newline at end of file diff --git a/docs/getting-started/walkthroughs/table_api.md b/docs/getting-started/walkthroughs/table_api.md deleted file mode 100644 index 0c99dbca10a1b7..00000000000000 --- a/docs/getting-started/walkthroughs/table_api.md +++ /dev/null @@ -1,513 +0,0 @@ ---- -title: "Table API" -nav-id: tableapiwalkthrough -nav-title: 'Table API' -nav-parent_id: walkthroughs -nav-pos: 2 ---- - - -Apache Flink offers a Table API as a unified, relational API for batch and stream processing, i.e., queries are executed with the same semantics on unbounded, real-time streams or bounded, batch data sets and produce the same results. -The Table API in Flink is commonly used to ease the definition of data analytics, data pipelining, and ETL applications. - -* This will be replaced by the TOC -{:toc} - -## What Will You Be Building? - -In this tutorial, you will learn how to build a continuous ETL pipeline for tracking financial transactions by account over time. -You will start by building your report as a nightly batch job, and then migrate to a streaming pipeline. - -## Prerequisites - -This walkthrough assumes that you have some familiarity with Java or Scala, but you should be able to follow along even if you are coming from a different programming language. -It also assumes that you are familiar with basic relational concepts such as `SELECT` and `GROUP BY` clauses. - -## Help, I’m Stuck! - -If you get stuck, check out the [community support resources](https://flink.apache.org/community.html). -In particular, Apache Flink's [user mailing list](https://flink.apache.org/community.html#mailing-lists) is consistently ranked as one of the most active of any Apache project and a great way to get help quickly. - -## How To Follow Along - -If you want to follow along, you will require a computer with: - -* Java 8 or 11 -* Maven - -A provided Flink Maven Archetype will create a skeleton project with all the necessary dependencies quickly: - --{% highlight java %} -package spendreport; - -import org.apache.flink.api.common.state.ValueState; -import org.apache.flink.api.common.state.ValueStateDescriptor; -import org.apache.flink.api.common.typeinfo.Types; -import org.apache.flink.configuration.Configuration; -import org.apache.flink.streaming.api.functions.KeyedProcessFunction; -import org.apache.flink.util.Collector; -import org.apache.flink.walkthrough.common.entity.Alert; -import org.apache.flink.walkthrough.common.entity.Transaction; - -public class FraudDetector extends KeyedProcessFunction- -flagState; - private transient ValueState timerState; - - @Override - public void open(Configuration parameters) { - ValueStateDescriptor flagDescriptor = new ValueStateDescriptor<>( - "flag", - Types.BOOLEAN); - flagState = getRuntimeContext().getState(flagDescriptor); - - ValueStateDescriptor timerDescriptor = new ValueStateDescriptor<>( - "timer-state", - Types.LONG); - timerState = getRuntimeContext().getState(timerDescriptor); - } - - @Override - public void processElement( - Transaction transaction, - Context context, - Collector collector) throws Exception { - - // Get the current state for the current key - Boolean lastTransactionWasSmall = flagState.value(); - - // Check if the flag is set - if (lastTransactionWasSmall != null) { - if (transaction.getAmount() > LARGE_AMOUNT) { - //Output an alert downstream - Alert alert = new Alert(); - alert.setId(transaction.getAccountId()); - - collector.collect(alert); - } - // Clean up our state - cleanUp(context); - } - - if (transaction.getAmount() < SMALL_AMOUNT) { - // set the flag to true - flagState.update(true); - - long timer = context.timerService().currentProcessingTime() + ONE_MINUTE; - context.timerService().registerProcessingTimeTimer(timer); - - timerState.update(timer); - } - } - - @Override - public void onTimer(long timestamp, OnTimerContext ctx, Collector out) { - // remove flag after 1 minute - timerState.clear(); - flagState.clear(); - } - - private void cleanUp(Context ctx) throws Exception { - // delete timer - Long timer = timerState.value(); - ctx.timerService().deleteProcessingTimeTimer(timer); - - // clean up all state - timerState.clear(); - flagState.clear(); - } -} -{% endhighlight %} - -{% highlight scala %} -package spendreport - -import org.apache.flink.api.common.state.{ValueState, ValueStateDescriptor} -import org.apache.flink.api.scala.typeutils.Types -import org.apache.flink.configuration.Configuration -import org.apache.flink.streaming.api.functions.KeyedProcessFunction -import org.apache.flink.util.Collector -import org.apache.flink.walkthrough.common.entity.Alert -import org.apache.flink.walkthrough.common.entity.Transaction - -object FraudDetector { - val SMALL_AMOUNT: Double = 1.00 - val LARGE_AMOUNT: Double = 500.00 - val ONE_MINUTE: Long = 60 * 1000L -} - -@SerialVersionUID(1L) -class FraudDetector extends KeyedProcessFunction[Long, Transaction, Alert] { - - @transient private var flagState: ValueState[java.lang.Boolean] = _ - @transient private var timerState: ValueState[java.lang.Long] = _ - - @throws[Exception] - override def open(parameters: Configuration): Unit = { - val flagDescriptor = new ValueStateDescriptor("flag", Types.BOOLEAN) - flagState = getRuntimeContext.getState(flagDescriptor) - - val timerDescriptor = new ValueStateDescriptor("timer-state", Types.LONG) - timerState = getRuntimeContext.getState(timerDescriptor) - } - - override def processElement( - transaction: Transaction, - context: KeyedProcessFunction[Long, Transaction, Alert]#Context, - collector: Collector[Alert]): Unit = { - - // Get the current state for the current key - val lastTransactionWasSmall = flagState.value - - // Check if the flag is set - if (lastTransactionWasSmall != null) { - if (transaction.getAmount > FraudDetector.LARGE_AMOUNT) { - // Output an alert downstream - val alert = new Alert - alert.setId(transaction.getAccountId) - - collector.collect(alert) - } - // Clean up our state - cleanUp(context) - } - - if (transaction.getAmount < FraudDetector.SMALL_AMOUNT) { - // set the flag to true - flagState.update(true) - val timer = context.timerService.currentProcessingTime + FraudDetector.ONE_MINUTE - - context.timerService.registerProcessingTimeTimer(timer) - timerState.update(timer) - } - } - - override def onTimer( - timestamp: Long, - ctx: KeyedProcessFunction[Long, Transaction, Alert]#OnTimerContext, - out: Collector[Alert]): Unit = { - // remove flag after 1 minute - timerState.clear() - flagState.clear() - } - - @throws[Exception] - private def cleanUp(ctx: KeyedProcessFunction[Long, Transaction, Alert]#Context): Unit = { - // delete timer - val timer = timerState.value - ctx.timerService.deleteProcessingTimeTimer(timer) - - // clean up all states - timerState.clear() - flagState.clear() - } -} -{% endhighlight %} ---- -{% unless site.is_stable %} --{% highlight bash %} -$ mvn archetype:generate \ - -DarchetypeGroupId=org.apache.flink \ - -DarchetypeArtifactId=flink-walkthrough-table-java \{% unless site.is_stable %} - -DarchetypeCatalog=https://repository.apache.org/content/repositories/snapshots/ \{% endunless %} - -DarchetypeVersion={{ site.version }} \ - -DgroupId=spend-report \ - -DartifactId=spend-report \ - -Dversion=0.1 \ - -Dpackage=spendreport \ - -DinteractiveMode=false -{% endhighlight %} ---{% highlight bash %} -$ mvn archetype:generate \ - -DarchetypeGroupId=org.apache.flink \ - -DarchetypeArtifactId=flink-walkthrough-table-scala \{% unless site.is_stable %} - -DarchetypeCatalog=https://repository.apache.org/content/repositories/snapshots/ \{% endunless %} - -DarchetypeVersion={{ site.version }} \ - -DgroupId=spend-report \ - -DartifactId=spend-report \ - -Dversion=0.1 \ - -Dpackage=spendreport \ - -DinteractiveMode=false -{% endhighlight %} --- Note: For Maven 3.0 or higher, it is no longer possible to specify the repository (-DarchetypeCatalog) via the command line. For details about this change, please refer to Maven official document - If you wish to use the snapshot repository, you need to add a repository entry to your settings.xml. For example: -{% highlight bash %} -
-{% endunless %} - -You can edit the `groupId`, `artifactId` and `package` if you like. With the above parameters, -Maven will create a project with all the dependencies to complete this tutorial. -After importing the project into your editor, you will see a file with the following code which you can run directly inside your IDE. - -- -{% endhighlight %} -- -apache -- -- -apache -- -- -apache-snapshots -https://repository.apache.org/content/repositories/snapshots/ --- -## Breaking Down The Code - -#### The Execution Environment - -The first two lines set up your `ExecutionEnvironment`. -The execution environment is how you can set properties for your Job, specify whether you are writing a batch or a streaming application, and create your sources. -This walkthrough begins with the batch environment since you are building a periodic batch report. -It is then wrapped in a `BatchTableEnvironment` to have full access to the Table API. - --{% highlight java %} -ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment(); -BatchTableEnvironment tEnv = BatchTableEnvironment.create(env); - -tEnv.registerTableSource("transactions", new BoundedTransactionTableSource()); -tEnv.registerTableSink("spend_report", new SpendReportTableSink()); -tEnv.registerFunction("truncateDateToHour", new TruncateDateToHour()); - -tEnv - .scan("transactions") - .insertInto("spend_report"); - -env.execute("Spend Report"); -{% endhighlight %} -- --{% highlight scala %} -val env = ExecutionEnvironment.getExecutionEnvironment -val tEnv = BatchTableEnvironment.create(env) - -tEnv.registerTableSource("transactions", new BoundedTransactionTableSource) -tEnv.registerTableSink("spend_report", new SpendReportTableSink) - -val truncateDateToHour = new TruncateDateToHour - -tEnv - .scan("transactions") - .insertInto("spend_report") - -env.execute("Spend Report") -{% endhighlight %} ---- - -#### Registering Tables - -Next, tables are registered in the execution environment that you can use to connect to external systems for reading and writing both batch and streaming data. -A table source provides access to data stored in external systems; such as a database, a key-value store, a message queue, or a file system. -A table sink emits a table to an external storage system. -Depending on the type of source and sink, they support different formats such as CSV, JSON, Avro, or Parquet. - --{% highlight java %} -ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment(); -BatchTableEnvironment tEnv = BatchTableEnvironment.create(env); -{% endhighlight %} -- --{% highlight scala %} -val env = ExecutionEnvironment.getExecutionEnvironment -val tEnv = BatchTableEnvironment.create(env) -{% endhighlight %} ---- -Two tables are registered; a transaction input table, and a spend report output table. -The transactions (`transactions`) table lets us read credit card transactions, which contain account ID's (`accountId`), timestamps (`timestamp`), and US$ amounts (`amount`). -In this tutorial, the table is backed by data generated in memory to avoid any dependencies on external systems. -In practice, the `BoundedTransactionTableSource` may be backed by a filesystem, a database, or any other static source. -The spend report (`spend_report`) table logs each row with log level **INFO**, instead of writing to persistent storage, so you can easily see your results. - -#### Registering A UDF - -Along with the tables, a [user-defined function]({{ site.baseurl }}/dev/table/functions/udfs.html) is registered for working with timestamps. -This function takes a timestamp and rounds it down to the nearest hour. - --{% highlight java %} -tEnv.registerTableSource("transactions", new BoundedTransactionTableSource()); -tEnv.registerTableSink("spend_report", new SpendReportTableSink()); -{% endhighlight %} -- --{% highlight scala %} -tEnv.registerTableSource("transactions", new BoundedTransactionTableSource) -tEnv.registerTableSink("spend_report", new SpendReportTableSink) -{% endhighlight %} ---- -#### The Query - -With the environment configured and tables registered, you are ready to build your first application. -From the `TableEnvironment` you can `scan` an input table to read its rows and then write those results into an output table using `insertInto`. - --{% highlight java %} -tEnv.registerFunction("truncateDateToHour", new TruncateDateToHour()); -{% endhighlight %} -- --{% highlight scala %} -val truncateDateToHour = new TruncateDateToHour -{% endhighlight %} ---- -Initially, the Job reads all transactions and logs them out with log level **INFO**. - -#### Execute - -Flink applications are built lazily and shipped to the cluster for execution only once fully formed. -You call `ExecutionEnvironment#execute` to begin the execution of your Job by giving it a name. - --{% highlight java %} -tEnv - .scan("transactions") - .insertInto("spend_report"); -{% endhighlight %} -- --{% highlight scala %} -tEnv - .scan("transactions") - .insertInto("spend_report") -{% endhighlight %} ---- -## Attempt One - -Now with the skeleton of a Job set-up, you are ready to add some business logic. -The goal is to build a report that shows the total spend for each account across each hour of the day. -Just like a SQL query, Flink can select the required fields and group by your keys. -Because the timestamp field has millisecond granularity, you can use the UDF to round it down to the nearest hour. -Finally, select all the fields, summing the total spend per account-hour pair with the built-in `sum` [aggregate function]({{ site.baseurl }}/dev/table/functions/systemFunctions.html#aggregate-functions). - --{% highlight java %} -env.execute("Spend Report"); -{% endhighlight %} -- --{% highlight scala %} -env.execute("Spend Report") -{% endhighlight %} ---- -This query consumes all records from the `transactions` table, calculates the report, and outputs the results in an efficient, scalable manner. - -{% highlight raw %} -# Query 1 output showing account id, timestamp, and amount - -> 1, 2019-01-01 00:00:00.0, $567.87 -> 2, 2019-01-01 00:00:00.0, $726.23 -> 1, 2019-01-01 01:00:00.0, $686.87 -> 2, 2019-01-01 01:00:00.0, $810.06 -> 1, 2019-01-01 02:00:00.0, $859.35 -> 2, 2019-01-01 02:00:00.0, $458.40 -> 1, 2019-01-01 03:00:00.0, $330.85 -> 2, 2019-01-01 03:00:00.0, $730.02 -> 1, 2019-01-01 04:00:00.0, $585.16 -> 2, 2019-01-01 04:00:00.0, $760.76 -{% endhighlight %} - -## Adding Windows - -Grouping data based on time is a typical operation in data processing, especially when working with infinite streams. -A grouping based on time is called a [window]({{ site.baseurl }}/dev/stream/operators/windows.html) and Flink offers flexible windowing semantics. -The most basic type of window is called a `Tumble` window, which has a fixed size and whose buckets do not overlap. - --{% highlight java %} -tEnv - .scan("transactions") - .select("accountId, timestamp.truncateDateToHour as timestamp, amount") - .groupBy("accountId, timestamp") - .select("accountId, timestamp, amount.sum as total") - .insertInto("spend_report"); -{% endhighlight %} -- --{% highlight scala %} -tEnv - .scan("transactions") - .select('accountId, truncateDateToHour('timestamp) as 'timestamp, 'amount) - .groupBy('accountId, 'timestamp) - .select('accountId, 'timestamp, 'amount.sum as 'total) - .insertInto("spend_report") -{% endhighlight %} ---- -This defines your application as using one hour tumbling windows based on the timestamp column. -So a row with timestamp `2019-06-01 01:23:47` is put in the `2019-06-01 01:00:00` window. - -Aggregations based on time are unique because time, as opposed to other attributes, generally moves forward in a continuous streaming application. -In a batch context, windows offer a convenient API for grouping records by a timestamp attribute. - -Running the updated query will produce identical results as before. - -{% highlight raw %} -# Query 2 output showing account id, timestamp, and amount - -> 1, 2019-01-01 00:00:00.0, $567.87 -> 2, 2019-01-01 00:00:00.0, $726.23 -> 1, 2019-01-01 01:00:00.0, $686.87 -> 2, 2019-01-01 01:00:00.0, $810.06 -> 1, 2019-01-01 02:00:00.0, $859.35 -> 2, 2019-01-01 02:00:00.0, $458.40 -> 1, 2019-01-01 03:00:00.0, $330.85 -> 2, 2019-01-01 03:00:00.0, $730.02 -> 1, 2019-01-01 04:00:00.0, $585.16 -> 2, 2019-01-01 04:00:00.0, $760.76 -{% endhighlight %} - -## Once More, With Streaming! - -Because Flink's Table API offers consistent syntax and semantics for both batch and streaming, migrating from one to the other requires just two steps. - -The first step is to replace the batch `ExecutionEnvironment` with its streaming counterpart, `StreamExecutionEnvironment`, which creates a continuous streaming Job. -It includes stream-specific configurations, such as the time characteristic, which when set to [event time]({{ site.baseurl }}/dev/event_time.html) guarantees consistent results even when faced with out-of-order events or a Job failure. -This is what will be used by your `Tumble` window when grouping records. - --{% highlight java %} -tEnv - .scan("transactions") - .window(Tumble.over("1.hour").on("timestamp").as("w")) - .groupBy("accountId, w") - .select("accountId, w.start as timestamp, amount.sum") - .insertInto("spend_report"); -{% endhighlight %} -- --{% highlight scala %} -tEnv - .scan("transactions") - .window(Tumble over 1.hour on 'timestamp as 'w) - .groupBy('accountId, 'w) - .select('accountId, 'w.start as 'timestamp, 'amount.sum) - .insertInto("spend_report") -{% endhighlight %} ---- -The second step is to migrate from a bounded data source to an infinite data source. -The project comes with an `UnboundedTransactionTableSource` that continuously creates transaction events in real-time. -Similar to the `BoundedTransactionTableSource` this table is backed by data generated in memory to avoid any dependencies on external systems. -In practice, this table might read from a streaming source such as Apache Kafka, AWS Kinesis, or Pravega. - --{% highlight java %} -StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); -env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime); - -StreamTableEnvironment tEnv = StreamTableEnvironment.create(env); -{% endhighlight %} -- --{% highlight scala %} -val env = StreamExecutionEnvironment.getExecutionEnvironment -env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime) - -val tEnv = StreamTableEnvironment.create(env) -{% endhighlight %} ---- -And that's it, a fully functional, stateful, distributed streaming application! -The query continuously consumes the stream of transactions, computes the hourly spendings, and emits results as soon as they are ready. -Since the input is unbounded, the query keeps running until it is manually stopped. -And because the Job uses time window-based aggregations, Flink can perform specific optimizations such as state clean up when the framework knows that no more records will arrive for a particular window. - -{% highlight raw %} -# Query 3 output showing account id, timestamp, and amount - -# These rows are calculated continuously over the hour -# and output immediately at the end of the hour -> 1, 2019-01-01 00:00:00.0, $567.87 -> 2, 2019-01-01 00:00:00.0, $726.23 - -# Flink begins computing these rows as soon as -# as the first record for the window arrives -> 1, 2019-01-01 01:00:00.0, $686.87 -> 2, 2019-01-01 01:00:00.0, $810.06 - -{% endhighlight %} - -## Final Application - --{% highlight java %} -tEnv.registerTableSource("transactions", new UnboundedTransactionTableSource()); -{% endhighlight %} -- --{% highlight scala %} -tEnv.registerTableSource("transactions", new UnboundedTransactionTableSource) -{% endhighlight %} ---- diff --git a/docs/getting-started/walkthroughs/table_api.zh.md b/docs/getting-started/walkthroughs/table_api.zh.md deleted file mode 100644 index f6b23a4f635809..00000000000000 --- a/docs/getting-started/walkthroughs/table_api.zh.md +++ /dev/null @@ -1,514 +0,0 @@ ---- -title: "Table API" -nav-id: tableapiwalkthrough -nav-title: 'Table API' -nav-parent_id: walkthroughs -nav-pos: 2 ---- - - -Apache Filnk 提供 Table API 作为批处理和流处理统一的关系型API， -即查询在无界实时流或有界批数据集上以相同的语义执行，并产生相同的结果。 -Flink 中的 Table API 通常用于简化数据分析，数据流水线和 ETL 应用程序的定义。 -* This will be replaced by the TOC -{:toc} - -## 接下来你会构建什么？ - -在本教程中，你将学习如何构建连续的 ETL 流水线，以便按账户随时跟踪金融交易。 -首先你将报表构建为每晚执行的批处理作业，然后迁移到流式管道。 - -## 先决条件 - -本演练假设你对 Java 和 Scala 有一定的了解，但即便你使用其他编程语言，相信也可以学会。 -它还假定你熟悉基本的关系概念比如 `SELECT` 和 `GROUP BY` 子句。 - -## 救命，我被困住了！ - -如果你被难题困住了，可以在[社区](https://flink.apache.org/community.html)寻求帮助。 -值得一提的是，Apache Flink 的[用户邮件列表](https://flink.apache.org/community.html#mailing-lists)一直是最活跃的 Apache 项目之一，也是一个快速获得帮助的好途径。 - -## 如何跟进 - -如果想要继续，你的电脑需要安装： -* Java 8 or 11 -* Maven - -现成的 Flink Maven Archetype 可以快速创建一个具有所有必要依赖的框架项目： - --{% highlight java %} -package spendreport; - -import org.apache.flink.walkthrough.common.table.SpendReportTableSink; -import org.apache.flink.walkthrough.common.table.TransactionTableSource; -import org.apache.flink.streaming.api.TimeCharacteristic; -import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; -import org.apache.flink.table.api.Tumble; -import org.apache.flink.table.api.java.StreamTableEnvironment; - -public class SpendReport { - - public static void main(String[] args) throws Exception { - StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); - env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime); - - StreamTableEnvironment tEnv = StreamTableEnvironment.create(env); - - tEnv.registerTableSource("transactions", new UnboundedTransactionTableSource()); - tEnv.registerTableSink("spend_report", new SpendReportTableSink()); - - tEnv - .scan("transactions") - .window(Tumble.over("1.hour").on("timestamp").as("w")) - .groupBy("accountId, w") - .select("accountId, w.start as timestamp, amount.sum") - .insertInto("spend_report"); - - env.execute("Spend Report"); - } -} -{% endhighlight %} -- --{% highlight scala %} -package spendreport - -import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment -import org.apache.flink.streaming.api.TimeCharacteristic -import org.apache.flink.table.api.Tumble -import org.apache.flink.table.api.scala._ -import org.apache.flink.walkthrough.common.table._ - -object SpendReport { - - def main(args: Array[String]): Unit = { - val env = StreamExecutionEnvironment.getExecutionEnvironment - env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime) - - val tEnv = StreamTableEnvironment.create(env) - - tEnv.registerTableSource("transactions", new UnboundedTransactionTableSource) - tEnv.registerTableSink("spend_report", new SpendReportTableSink) - - tEnv - .scan("transactions") - .window(Tumble over 1.hour on 'timestamp as 'w) - .groupBy('accountId, 'w) - .select('accountId, 'w.start as 'timestamp, 'amount.sum) - .insertInto("spend_report") - - env.execute("Spend Report") - } -} -{% endhighlight %} ---- -{% unless site.is_stable %} --{% highlight bash %} -$ mvn archetype:generate \ - -DarchetypeGroupId=org.apache.flink \ - -DarchetypeArtifactId=flink-walkthrough-table-java \{% unless site.is_stable %} - -DarchetypeCatalog=https://repository.apache.org/content/repositories/snapshots/ \{% endunless %} - -DarchetypeVersion={{ site.version }} \ - -DgroupId=spend-report \ - -DartifactId=spend-report \ - -Dversion=0.1 \ - -Dpackage=spendreport \ - -DinteractiveMode=false -{% endhighlight %} ---{% highlight bash %} -$ mvn archetype:generate \ - -DarchetypeGroupId=org.apache.flink \ - -DarchetypeArtifactId=flink-walkthrough-table-scala \{% unless site.is_stable %} - -DarchetypeCatalog=https://repository.apache.org/content/repositories/snapshots/ \{% endunless %} - -DarchetypeVersion={{ site.version }} \ - -DgroupId=spend-report \ - -DartifactId=spend-report \ - -Dversion=0.1 \ - -Dpackage=spendreport \ - -DinteractiveMode=false -{% endhighlight %} --- 注意：Maven 3.0 及更高版本，不再支持通过命令行指定仓库（-DarchetypeCatalog）。有关这个改动的详细信息， - 请参阅 Maven 官方文档 - 如果你希望使用快照仓库，则需要在 settings.xml 文件中添加一个仓库条目。例如： -{% highlight bash %} -
-{% endunless %} - -你可以根据自己的意愿修改 `groupId`、`artifactId` 和 `package` 参数。通过使用以上参数， -Maven 将创建一个拥有全部所需依赖的项目来完成本教程。 -把项目导入编辑器后，你会看到一个包含以下代码的文件，你可以在 IDE 中直接运行它。 - -- -{% endhighlight %} -- -apache -- -- -apache -- -- -apache-snapshots -https://repository.apache.org/content/repositories/snapshots/ --- -## 代码详解 - -#### 运行环境 - -前两行设置了你的 `ExecutionEnvironment`。 -运行环境用来设置作业的属性、指定应用是批处理还是流处理，以及创建数据源。 -由于你正在建立一个定时的批处理报告，本教程以批处理环境作为开始。 -然后将其包装进 `BatchTableEnvironment` 中从而能够使用所有的 Tabel API。 - --{% highlight java %} -ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment(); -BatchTableEnvironment tEnv = BatchTableEnvironment.create(env); - -tEnv.registerTableSource("transactions", new BoundedTransactionTableSource()); -tEnv.registerTableSink("spend_report", new SpendReportTableSink()); -tEnv.registerFunction("truncateDateToHour", new TruncateDateToHour()); - -tEnv - .scan("transactions") - .insertInto("spend_report"); - -env.execute("Spend Report"); -{% endhighlight %} -- --{% highlight scala %} -val env = ExecutionEnvironment.getExecutionEnvironment -val tEnv = BatchTableEnvironment.create(env) - -tEnv.registerTableSource("transactions", new BoundedTransactionTableSource) -tEnv.registerTableSink("spend_report", new SpendReportTableSink) - -val truncateDateToHour = new TruncateDateToHour - -tEnv - .scan("transactions") - .insertInto("spend_report") - -env.execute("Spend Report") -{% endhighlight %} ---- - -#### 注册表 - -接下来，表将会被注册到运行环境之中，这样你就可以用它们连接外部系统以读取或写入批数据或流数据。 -source 提供对存储在外部系统中的数据的访问；例如数据库、键-值存储、消息队列或文件系统。 -sink 则将表中的数据发送到外部存储系统。 -根据 source 或 sink 的类型，它们支持不同的格式，如 CSV、JSON、Avro 或 Parquet。 - --{% highlight java %} -ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment(); -BatchTableEnvironment tEnv = BatchTableEnvironment.create(env); -{% endhighlight %} -- --{% highlight scala %} -val env = ExecutionEnvironment.getExecutionEnvironment -val tEnv = BatchTableEnvironment.create(env) -{% endhighlight %} ---- -上例代码注册了两张表。交易表作为输入表，支出报告表作为输出表。 -我们可以从交易（`transactions`）表中读取信用卡的交易记录，其中包含了账户 ID（`accountId`）字段、时间戳（`timestamp`）字段和交易金额（`amount`）字段。 -本教程中，该表使用内存中的数据，以避免对外部系统的任何依赖。 -而在实际情况下，`BoundedTransactionTableSource` 可能来源于文件系统、数据库或任何静态数据源。 -支出报告表 `spend_report` 用 **INFO** 日志级别将表的每一行数据记录到日志，而不是写入持久化存储，所以你可以很容易地查看结果。 - -#### 注册 UDF - -一个用来处理时间戳的[自定义函数]({{ site.baseurl }}/zh/dev/table/functions/udfs.html)随表一起被注册到tEnv中。 -此函数将时间戳向下舍入到最接近的小时。 - --{% highlight java %} -tEnv.registerTableSource("transactions", new BoundedTransactionTableSource()); -tEnv.registerTableSink("spend_report", new SpendReportTableSink()); -{% endhighlight %} -- --{% highlight scala %} -tEnv.registerTableSource("transactions", new BoundedTransactionTableSource) -tEnv.registerTableSink("spend_report", new SpendReportTableSink) -{% endhighlight %} ---- -#### 查询 - -完成配置环境和注册表后，你已准备好构建第一个应用程序。 -从 `TableEnvironment` 中，你可以 `scan` 一个输入表读取其中的行，然后用 `insertInto` 把这些数据写到输出表中。 - --{% highlight java %} -tEnv.registerFunction("truncateDateToHour", new TruncateDateToHour()); -{% endhighlight %} -- --{% highlight scala %} -val truncateDateToHour = new TruncateDateToHour -{% endhighlight %} ---- -最初，作业读取所有的交易记录并用 **INFO** 日志级别将其记录下来。 - -#### 运行 - -Flink 应用是延迟构建的，只有完全定义好之后才交付集群运行。 -你可以调用 `ExecutionEnvironment#execute` 来开始作业的执行并给它取一个名字。 - - --{% highlight java %} -tEnv - .scan("transactions") - .insertInto("spend_report"); -{% endhighlight %} -- --{% highlight scala %} -tEnv - .scan("transactions") - .insertInto("spend_report") -{% endhighlight %} ---- -## 尝试一下 - -现在有了作业设置的框架，你就可以添加一些业务逻辑了。 -目标是建立一个报表来显示每天每小时每个账户的总支出。 -就像一个 SQL 查询一样，Flink 可以选取所需的字段并且按键分组。 -由于时间戳字段具有毫秒的粒度，你可以使用自定义函数将其舍入到最近的小时。 -最后，选取所有的字段，用内建的 `sum` [聚合函数]({{ site.baseurl }}/zh/dev/table/functions/systemFunctions.html#aggregate-functions)函数合计每一个账户每小时的支出。 --{% highlight java %} -env.execute("Spend Report"); -{% endhighlight %} -- --{% highlight scala %} -env.execute("Spend Report") -{% endhighlight %} ---- -这个查询处理了 `transactions` 表中的所有记录，计算报告，并以高效、可扩展的方式输出结果。 -{% highlight raw %} -# 查询 1 的输出显示了账户 id、时间戳和消费总额。 - -> 1, 2019-01-01 00:00:00.0, $567.87 -> 2, 2019-01-01 00:00:00.0, $726.23 -> 1, 2019-01-01 01:00:00.0, $686.87 -> 2, 2019-01-01 01:00:00.0, $810.06 -> 1, 2019-01-01 02:00:00.0, $859.35 -> 2, 2019-01-01 02:00:00.0, $458.40 -> 1, 2019-01-01 03:00:00.0, $330.85 -> 2, 2019-01-01 03:00:00.0, $730.02 -> 1, 2019-01-01 04:00:00.0, $585.16 -> 2, 2019-01-01 04:00:00.0, $760.76 -{% endhighlight %} - -## 添加窗口 - -根据时间进行分组在数据处理中是一种很常见的方式，特别是在处理无限的数据流时。 -基于时间的分组称为[窗口]({{ site.baseurl }}/zh/dev/stream/operators/windows.html) ，Flink 提供了灵活的窗口语义。 -其中最基础的是 `Tumble` window （滚动窗口），它具有固定大小且窗口之间不重叠。 - --{% highlight java %} -tEnv - .scan("transactions") - .select("accountId, timestamp.truncateDateToHour as timestamp, amount") - .groupBy("accountId, timestamp") - .select("accountId, timestamp, amount.sum as total") - .insertInto("spend_report"); -{% endhighlight %} -- --{% highlight scala %} -tEnv - .scan("transactions") - .select('accountId, truncateDateToHour('timestamp) as 'timestamp, 'amount) - .groupBy('accountId, 'timestamp) - .select('accountId, 'timestamp, 'amount.sum as 'total) - .insertInto("spend_report") -{% endhighlight %} ---- -你的应用将会使用基于时间戳字段的一小时的滚动窗口。 -因此时间戳是 `2019-06-01 01:23:47` 的行被放入 `2019-06-01 01:00:00` 这个时间窗口之中。 - -在持续的流式应用中，基于时间的聚合结果是唯一的，因为相较于其他属性，时间通常会向前移动。 -在批处理环境中，窗口提供了一个方便的 API，用于按时间戳属性对记录进行分组。 - -运行这个更新过的查询将会得到和之前一样的结果。 - -{% highlight raw %} -# 查询 2 的输出显示了账户 id、时间戳和消费总额 - -> 1, 2019-01-01 00:00:00.0, $567.87 -> 2, 2019-01-01 00:00:00.0, $726.23 -> 1, 2019-01-01 01:00:00.0, $686.87 -> 2, 2019-01-01 01:00:00.0, $810.06 -> 1, 2019-01-01 02:00:00.0, $859.35 -> 2, 2019-01-01 02:00:00.0, $458.40 -> 1, 2019-01-01 03:00:00.0, $330.85 -> 2, 2019-01-01 03:00:00.0, $730.02 -> 1, 2019-01-01 04:00:00.0, $585.16 -> 2, 2019-01-01 04:00:00.0, $760.76 -{% endhighlight %} - -## 通过流处理的方式再来一次！ - -因为 Flink 的 Table API 为批处理和流处理提供了相同的语法和语义，从一种方式迁移到另一种方式只需要两步。 - -第一步是把批处理的 `ExecutionEnvironment` 替换成流处理对应的 `StreamExecutionEnvironment`，后者创建连续的流作业。 -它包含特定于流处理的配置，比如时间特性。当这个属性被设置成 [事件时间]({{ site.baseurl }}/zh/dev/event_time.html)时，它能保证即使遭遇乱序事件或者作业失败的情况也能输出一致的结果。 -滚动窗口在对数据进行分组时就运用了这个特性。 - --{% highlight java %} -tEnv - .scan("transactions") - .window(Tumble.over("1.hour").on("timestamp").as("w")) - .groupBy("accountId, w") - .select("accountId, w.start as timestamp, amount.sum") - .insertInto("spend_report"); -{% endhighlight %} -- --{% highlight scala %} -tEnv - .scan("transactions") - .window(Tumble over 1.hour on 'timestamp as 'w) - .groupBy('accountId, 'w) - .select('accountId, 'w.start as 'timestamp, 'amount.sum) - .insertInto("spend_report") -{% endhighlight %} ---- -第二步就是把有界的数据源替换成无限的数据源。 -这个项目通过 `UnboundedTransactionTableSource` 持续不断地实时生成交易事件。 -与 `BoundedTransactionTableSource` 一样，这个表也是通过在内存中生成数据从而不依赖外部系统。 -在实践中，这个表可能从一个流式数据源中读取数据，比如 Apache Kafka、AWS Kinesis 或者 Pravega。 - --{% highlight java %} -StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); -env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime); - -StreamTableEnvironment tEnv = StreamTableEnvironment.create(env); -{% endhighlight %} -- --{% highlight scala %} -val env = StreamExecutionEnvironment.getExecutionEnvironment -env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime) - -val tEnv = StreamTableEnvironment.create(env) -{% endhighlight %} ---- -这就是一个功能齐全、有状态的分布式流式应用！ -这个查询会持续处理交易流，计算每小时的消费额，然后实时输出结果。 -由于输入是无界的，因此查询将一直运行，直到手动停止为止。 -因为这个作业使用了基于时间窗口的聚合，Flink 可以使用一些特定的优化，比如当系统知道一个特定的窗口不会再有新的数据到来，它就会对状态进行清理。 - -{% highlight raw %} -# 查询 3 的输出显示了账户 id、时间戳消费总额 - -> 1, 2019-01-01 00:00:00.0, $567.87 -> 2, 2019-01-01 00:00:00.0, $726.23 - -# 这些行是在这一小时中连续计算的 -# 并在这一小时结束时立刻输出 - -> 1, 2019-01-01 01:00:00.0, $686.87 -> 2, 2019-01-01 01:00:00.0, $810.06 - -# 当接收到该窗口的第一条数据时 -# Flink 就开始计算了 - -{% endhighlight %} - -## 最终程序 - --{% highlight java %} -tEnv.registerTableSource("transactions", new UnboundedTransactionTableSource()); -{% endhighlight %} -- --{% highlight scala %} -tEnv.registerTableSource("transactions", new UnboundedTransactionTableSource) -{% endhighlight %} ---- diff --git a/docs/index.md b/docs/index.md index 9f0acae12d89f9..ae73a9c7ae13f2 100644 --- a/docs/index.md +++ b/docs/index.md @@ -23,51 +23,72 @@ specific language governing permissions and limitations under the License. --> -This documentation is for Apache Flink version {{ site.version_title }}. These pages were built at: {% build_time %}. +-{% highlight java %} -package spendreport; - -import org.apache.flink.walkthrough.common.table.SpendReportTableSink; -import org.apache.flink.walkthrough.common.table.TransactionTableSource; -import org.apache.flink.streaming.api.TimeCharacteristic; -import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; -import org.apache.flink.table.api.Tumble; -import org.apache.flink.table.api.java.StreamTableEnvironment; - -public class SpendReport { - - public static void main(String[] args) throws Exception { - StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); - env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime); - - StreamTableEnvironment tEnv = StreamTableEnvironment.create(env); - - tEnv.registerTableSource("transactions", new UnboundedTransactionTableSource()); - tEnv.registerTableSink("spend_report", new SpendReportTableSink()); - - tEnv - .scan("transactions") - .window(Tumble.over("1.hour").on("timestamp").as("w")) - .groupBy("accountId, w") - .select("accountId, w.start as timestamp, amount.sum") - .insertInto("spend_report"); - - env.execute("Spend Report"); - } -} -{% endhighlight %} -- --{% highlight scala %} -package spendreport - -import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment -import org.apache.flink.streaming.api.TimeCharacteristic -import org.apache.flink.table.api.Tumble -import org.apache.flink.table.api.scala._ -import org.apache.flink.walkthrough.common.table._ - -object SpendReport { - - def main(args: Array[String]): Unit = { - val env = StreamExecutionEnvironment.getExecutionEnvironment - env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime) - - val tEnv = StreamTableEnvironment.create(env) - - tEnv.registerTableSource("transactions", new UnboundedTransactionTableSource) - tEnv.registerTableSink("spend_report", new SpendReportTableSink) - - tEnv - .scan("transactions") - .window(Tumble over 1.hour on 'timestamp as 'w) - .groupBy('accountId, 'w) - .select('accountId, 'w.start as 'timestamp, 'amount.sum) - .insertInto("spend_report") - - env.execute("Spend Report") - } -} -{% endhighlight %} --+Apache Flink is a framework and distributed processing engine for stateful computations over _unbounded and bounded_ data streams. Flink has been designed to run in _all common cluster environments_, perform computations at _in-memory speed_ and at _any scale_. +
+ +++ ++ +### Try Flink + +If you’re interested in playing around with Flink, try one of our tutorials: + +* [Local Installation]({% link try-flink/local_installation.md %}) +* [Fraud Detection with the DataStream API]({% link try-flink/datastream_api.md %}) +* [Real Time Reporting with the Table API]({% link try-flink/table_api.md %}) +* [Intro to the Python Table API]({% link try-flink/python_table_api.md %}) +* [Flink Operations Playground]({% link try-flink/flink-operations-playground.md %}) -Apache Flink is an open source platform for distributed stream and batch data processing. Flink’s core is a streaming dataflow engine that provides data distribution, communication, and fault tolerance for distributed computations over data streams. Flink builds batch processing on top of the streaming engine, overlaying native iteration support, managed memory, and program optimization. +### Learn Flink -## First Steps +* To dive in deeper, the [Hands-on Training]({% link learn-flink/index.md %}) includes a set of lessons and exercises that provide a step-by-step introduction to Flink. -* **Code Walkthroughs**: Follow step-by-step guides and implement a simple application or query in one of Flink's APIs. - * [Implement a DataStream application](./getting-started/walkthroughs/datastream_api.html) - * [Write a Table API query](./getting-started/walkthroughs/table_api.html) +* The [Concepts]({% link concepts/index.md %}) section explains what you need to know about Flink before exploring the reference documentation. -* **Docker Playgrounds**: Set up a sandboxed Flink environment in just a few minutes to explore and play with Flink. - * [Run and manage Flink streaming applications](./getting-started/docker-playgrounds/flink-operations-playground.html) +### Get Help with Flink -* **Concepts**: Learn about Flink's basic concepts to better understand the documentation. - * [Dataflow Programming Model](concepts/programming-model.html) - * [Distributed Runtime](concepts/runtime.html) - * [Glossary](concepts/glossary.html) +If you get stuck, check out our [community support resources](https://flink.apache.org/community.html). In particular, Apache Flink’s user mailing list is consistently ranked as one of the most active of any Apache project, and is a great way to get help quickly. -## API References ++-The API references list and explain all features of Flink's APIs. +### Explore Flink -* [DataStream API](dev/datastream_api.html) -* [DataSet API](dev/batch/index.html) -* [Table API & SQL](dev/table/index.html) +The reference documentation covers all the details. Some starting points: -## Deployment +++-## External Resources +### Deploy Flink -- **Flink Forward**: Talks from past conferences are available at the [Flink Forward](http://flink-forward.org/) website and on [YouTube](https://www.youtube.com/channel/UCY8_lgiZLZErZPF47a2hXMA). [Robust Stream Processing with Apache Flink](http://2016.flink-forward.org/kb_sessions/robust-stream-processing-with-apache-flink/) is a good place to start. +Before putting your Flink job into production, read the [Production Readiness Checklist]({% link ops/production_ready.md %}). For an overview of possible deployment targets, see [Clusters and Deployments]({% link ops/deployment/index.md %}). -- **Training**: The [training materials](https://training.ververica.com/) from Ververica include slides, exercises, and sample solutions. +### Upgrade Flink -- **Blogs**: The [Apache Flink](https://flink.apache.org/blog/) and [Ververica](https://www.ververica.com/blog) blogs publish frequent, in-depth technical articles about Flink. +Release notes cover important changes between Flink versions. Please read them carefully if you plan to upgrade your Flink setup. + +See the release notes for [Flink 1.11]({% link release-notes/flink-1.11.md %}), [Flink 1.10]({% link release-notes/flink-1.10.md %}), [Flink 1.9]({% link release-notes/flink-1.9.md %}), [Flink 1.8]({% link release-notes/flink-1.8.md %}), or [Flink 1.7]({% link release-notes/flink-1.7.md %}). + +-Before putting your Flink job into production, read the [Production Readiness Checklist](ops/production_ready.html). +* [DataStream API]({% link dev/datastream_api.md %}) +* [Table API & SQL]({% link dev/table/index.md %}) +* [Stateful Functions]({% if site.is_stable %} {{ site.statefundocs_stable_baseurl }} {% else %} {{ site.statefundocs_baseurl }} {% endif %}) -## Release Notes ++-Release notes cover important changes between Flink versions. Please carefully read these notes if you plan to upgrade your Flink setup to a later version. +* [Configuration]({% link ops/config.md %}) +* [Rest API]({% link monitoring/rest_api.md %}) +* [CLI]({% link ops/cli.md %}) -* [Release notes for Flink 1.10](release-notes/flink-1.10.html). -* [Release notes for Flink 1.9](release-notes/flink-1.9.html). -* [Release notes for Flink 1.8](release-notes/flink-1.8.html). -* [Release notes for Flink 1.7](release-notes/flink-1.7.html). -* [Release notes for Flink 1.6](release-notes/flink-1.6.html). -* [Release notes for Flink 1.5](release-notes/flink-1.5.html). +++diff --git a/docs/index.zh.md b/docs/index.zh.md index 2315024829eabe..f9a796657c83fe 100644 --- a/docs/index.zh.md +++ b/docs/index.zh.md @@ -23,53 +23,72 @@ specific language governing permissions and limitations under the License. --> ++This documentation is for Apache Flink version {{ site.version_title }}. These pages were built at: {% build_time %}. +
++Apache Flink 是一个在无界和有界数据流上进行状态计算的框架和分布式处理引擎。 Flink 已经可以在所有常见的集群环境中运行，并以 in-memory 的速度和任意的规模进行计算。 +
-本文档适用于 Apache Flink {{ site.version_title}} 版本。本页面最近更新于 {% build_time %}. +++ +-Apache Flink 是一个分布式流批一体化的开源平台。Flink 的核心是一个提供数据分发、通信以及自动容错的流计算引擎。Flink 在流计算之上构建批处理，并且原生的支持迭代计算，内存管理以及程序优化。 +### 试用 Flink -## 初步印象 +如果您有兴趣使用 Flink, 可以试试我们的教程: -* **代码练习**: 跟随分步指南通过 Flink API 实现简单应用或查询。 - * [实现 DataStream 应用](./getting-started/walkthroughs/datastream_api.html) - * [书写 Table API 查询](./getting-started/walkthroughs/table_api.html) +* [Local Installation]({% link try-flink/local_installation.zh.md %}) +* [DataStream API 进行欺诈检测]({% link try-flink/datastream_api.zh.md %}) +* [Table API 构建实时报表]({% link try-flink/table_api.zh.md %}) +* [Python API 教程]({% link try-flink/python_table_api.zh.md %}) +* [Flink 游乐场]({% link try-flink/flink-operations-playground.zh.md %}) -* **Docker 游乐场**: 你只需花几分钟搭建 Flink 沙盒环境，就可以探索和使用 Flink 了。 - * [运行与管理 Flink 流处理应用](./getting-started/docker-playgrounds/flink-operations-playground.html) +### 学习 Flink -* **概念**: 学习 Flink 的基本概念能更好地理解文档。 - * [数据流编程模型](concepts/programming-model.html) - * [分布式执行](concepts/runtime.html) - * [术语表](concepts/glossary.html) +* [操作培训]({% link learn-flink/index.zh.md %}) 包含了一系列的课程和练习，提供了对 Flink 的逐一介绍。 -## API 参考 +* [概念透析]({% link concepts/index.zh.md %}) 介绍了在浏览参考文档之前你需要了解的 Flink 知识。 -API 参考列举并解释了 Flink API 的所有功能。 +### 获取 Flink 帮助 -* [基本 API 概念](dev/api_concepts.html) -* [DataStream API](dev/datastream_api.html) -* [DataSet API](dev/batch/index.html) -* [Table API & SQL](dev/table/index.html) +如果你遇到困难了, 可以在 [社区](https://flink.apache.org/zh/community.html)寻求帮助。值得一提的是，Apache Flink 的用户邮件列表一直是 Apache 项目里面最活跃的之一，也是一个快速获得帮助的好途径。 -## 部署 ++-在线上环境运行你的 Flink 作业之前，请阅读 [生产环境注意事项检查清单](ops/production_ready.html)。 +### 探索 Flink -## 发布日志 +参考文档包含了 Flink 所有内容。 你可以从以下几点开始学习: -发布日志包含了 Flink 版本之间的重大更新。请在你升级 Flink 之前仔细阅读相应的发布日志。 +++-- **博客**: [Apache Flink](https://flink.apache.org/blog/) 以及 [Ververica](https://www.ververica.com/blog) 的博客会经常更新一些有关 Flink 的技术文章。 +### 部署 Flink + +在线上环境运行你的 Flink 作业之前，请阅读 [生产环境注意事项检查清单]({% link ops/production_ready.zh.md %}). 各种部署环境一览，详见 [集群与部署]({% link ops/deployment/index.zh.md %}). + +### 升级 Flink + +release notes 包含了 Flink 版本之间的重大更新。请在你升级 Flink 之前仔细阅读相应的 release notes。 + +请阅读 release notes [Flink 1.11]({% link release-notes/flink-1.11.zh.md %}), [Flink 1.10]({% link release-notes/flink-1.10.zh.md %}), [Flink 1.9]({% link release-notes/flink-1.9.zh.md %}), [Flink 1.8]({% link release-notes/flink-1.8.zh.md %}), or [Flink 1.7]({% link release-notes/flink-1.7.zh.md %}). + +-* [Flink 1.10 的发布日志](release-notes/flink-1.10.html). -* [Flink 1.9 的发布日志](release-notes/flink-1.9.html)。 -* [Flink 1.8 的发布日志](release-notes/flink-1.8.html)。 -* [Flink 1.7 的发布日志](release-notes/flink-1.7.html)。 -* [Flink 1.6 的发布日志](release-notes/flink-1.6.html)。 -* [Flink 1.5 的发布日志](release-notes/flink-1.5.html)。 +* [DataStream API]({% link dev/datastream_api.zh.md %}) +* [Table API & SQL]({% link dev/table/index.zh.md %}) +* [状态方法]({% if site.is_stable %} {{ site.statefundocs_stable_baseurl }} {% else %} {{ site.statefundocs_baseurl }} {% endif %}) -## 外部资源 ++-- **Flink Forward**: 已举办的所有大会演讲均可在 [Flink Forward](http://flink-forward.org/) 官网以及 [YouTube](https://www.youtube.com/channel/UCY8_lgiZLZErZPF47a2hXMA)找到。[使用 Apache Flink 进行高可靠的流处理](http://2016.flink-forward.org/kb_sessions/robust-stream-processing-with-apache-flink/) 可以作为你第一个学习的资源。 +* [配置参数]({% link ops/config.zh.md %}) +* [Rest API]({% link monitoring/rest_api.zh.md %}) +* [CLI]({% link ops/cli.zh.md %}) -- **培训**: [培训资料](https://training.ververica.com/) 包含讲义，练习以及示例程序。 +++diff --git a/docs/internals/task_lifecycle.md b/docs/internals/task_lifecycle.md index 44f847f0cba6d5..ee81780c3c3ebe 100644 --- a/docs/internals/task_lifecycle.md +++ b/docs/internals/task_lifecycle.md @@ -92,7 +92,7 @@ operator is opened and before it is closed. The responsibility of this method is to the specified [state backend]({{ site.baseurl }}/ops/state/state_backends.html) from where it will be retrieved when the job resumes execution after a failure. Below we include a brief description of Flink's checkpointing mechanism, and for a more detailed discussion on the principles around checkpointing in Flink please read the corresponding documentation: -[Data Streaming Fault Tolerance]({{ site.baseurl }}/internals/stream_checkpointing.html). +[Data Streaming Fault Tolerance]({{ site.baseurl }}/learn-flink/fault_tolerance.html). ## Task Lifecycle diff --git a/docs/internals/task_lifecycle.zh.md b/docs/internals/task_lifecycle.zh.md index 7de935b4ed88f8..a06ec4ecebd685 100644 --- a/docs/internals/task_lifecycle.zh.md +++ b/docs/internals/task_lifecycle.zh.md @@ -92,7 +92,7 @@ operator is opened and before it is closed. The responsibility of this method is to the specified [state backend]({{ site.baseurl }}/ops/state/state_backends.html) from where it will be retrieved when the job resumes execution after a failure. Below we include a brief description of Flink's checkpointing mechanism, and for a more detailed discussion on the principles around checkpointing in Flink please read the corresponding documentation: -[Data Streaming Fault Tolerance]({{ site.baseurl }}/internals/stream_checkpointing.html). +[Data Streaming Fault Tolerance]({{ site.baseurl }}/learn-flink/fault_tolerance.html). ## Task Lifecycle diff --git a/docs/learn-flink/datastream_api.md b/docs/learn-flink/datastream_api.md new file mode 100644 index 00000000000000..23f1f802f145a9 --- /dev/null +++ b/docs/learn-flink/datastream_api.md @@ -0,0 +1,236 @@ +--- +title: Intro to the DataStream API +nav-id: datastream-api +nav-pos: 2 +nav-title: Intro to the DataStream API +nav-parent_id: learn-flink +--- + + +The focus of this training is to broadly cover the DataStream API well enough that you will be able +to get started writing streaming applications. + +* This will be replaced by the TOC +{:toc} + +## What can be Streamed? + +Flink's DataStream APIs for Java and Scala will let you stream anything they can serialize. Flink's +own serializer is used for + +- basic types, i.e., String, Long, Integer, Boolean, Array +- composite types: Tuples, POJOs, and Scala case classes + +and Flink falls back to Kryo for other types. It is also possible to use other serializers with +Flink. Avro, in particular, is well supported. + +### Java tuples and POJOs + +Flink's native serializer can operate efficiently on tuples and POJOs. + +#### Tuples + +For Java, Flink defines its own `Tuple0` thru `Tuple25` types. + +{% highlight java %} +Tuple2+本文档适用于 Apache Flink {{ site.version_title }} 版本。本页面最近更新于: {% build_time %}. +
+flintstones = env.fromElements( + new Person("Fred", 35), + new Person("Wilma", 35), + new Person("Pebbles", 2)); + + DataStream adults = flintstones.filter(new FilterFunction () { + @Override + public boolean filter(Person person) throws Exception { + return person.age >= 18; + } + }); + + adults.print(); + + env.execute(); + } + + public static class Person { + public String name; + public Integer age; + public Person() {}; + + public Person(String name, Integer age) { + this.name = name; + this.age = age; + }; + + public String toString() { + return this.name.toString() + ": age " + this.age.toString(); + }; + } +} +{% endhighlight %} + +### Stream execution environment + +Every Flink application needs an execution environment, `env` in this example. Streaming +applications need to use a `StreamExecutionEnvironment`. + +The DataStream API calls made in your application build a job graph that is attached to the +`StreamExecutionEnvironment`. When `env.execute()` is called this graph is packaged up and sent to +the JobManager, which parallelizes the job and distributes slices of it to the Task Managers for +execution. Each parallel slice of your job will be executed in a *task slot*. + +Note that if you don't call execute(), your application won't be run. + + 
+
+This distributed runtime depends on your application being serializable. It also requires that all
+dependencies are available to each node in the cluster.
+
+### Basic stream sources
+
+The example above constructs a `DataStream` using `env.fromElements(...)`. This is a +convenient way to throw together a simple stream for use in a prototype or test. There is also a +`fromCollection(Collection)` method on `StreamExecutionEnvironment`. So instead, you could do this: + +{% highlight java %} +List people = new ArrayList (); + +people.add(new Person("Fred", 35)); +people.add(new Person("Wilma", 35)); +people.add(new Person("Pebbles", 2)); + +DataStream flintstones = env.fromCollection(people); +{% endhighlight %} + +Another convenient way to get some data into a stream while prototyping is to use a socket + +{% highlight java %} +DataStream lines = env.socketTextStream("localhost", 9999) +{% endhighlight %} + +or a file + +{% highlight java %} +DataStream lines = env.readTextFile("file:///path"); +{% endhighlight %} + +In real applications the most commonly used data sources are those that support low-latency, high +throughput parallel reads in combination with rewind and replay -- the prerequisites for high +performance and fault tolerance -- such as Apache Kafka, Kinesis, and various filesystems. REST APIs +and databases are also frequently used for stream enrichment. + +### Basic stream sinks + +The example above uses `adults.print()` to print its results to the task manager logs (which will +appear in your IDE's console, when running in an IDE). This will call `toString()` on each element +of the stream. + +The output looks something like this + + 1> Fred: age 35 + 2> Wilma: age 35 + +where 1> and 2> indicate which sub-task (i.e., thread) produced the output. + +In production, commonly used sinks include the StreamingFileSink, various databases, +and several pub-sub systems. + +### Debugging + +In production, your application will run in a remote cluster or set of containers. And if it fails, +it will fail remotely. The JobManager and TaskManager logs can be very helpful in debugging such +failures, but it is much easier to do local debugging inside an IDE, which is something that Flink +supports. You can set breakpoints, examine local variables, and step through your code. You can also +step into Flink's code, which can be a great way to learn more about its internals if you are +curious to see how Flink works. + +{% top %} + +## Hands-on + +At this point you know enough to get started coding and running a simple DataStream application. +Clone the [flink-training repo](https://github.com/apache/flink-training/tree/{% if site.is_stable %}release-{{ site.version_title }}{% else %}master{% endif %}), and after following the +instructions in the README, do the first exercise: +[Filtering a Stream (Ride Cleansing)](https://github.com/apache/flink-training/tree/{% if site.is_stable %}release-{{ site.version_title }}{% else %}master{% endif %}/ride-cleansing). + +{% top %} + +## Further Reading + +- [Flink Serialization Tuning Vol. 1: Choosing your Serializer — if you can](https://flink.apache.org/news/2020/04/15/flink-serialization-tuning-vol-1.html) +- [Anatomy of a Flink Program]({% link dev/datastream_api.md %}#anatomy-of-a-flink-program) +- [Data Sources]({% link dev/datastream_api.md %}#data-sources) +- [Data Sinks]({% link dev/datastream_api.md %}#data-sinks) +- [DataStream Connectors]({% link dev/connectors/index.md %}) + +{% top %} diff --git a/docs/learn-flink/datastream_api.zh.md b/docs/learn-flink/datastream_api.zh.md new file mode 100644 index 00000000000000..92dfbdaeee5158 --- /dev/null +++ b/docs/learn-flink/datastream_api.zh.md @@ -0,0 +1,213 @@ +--- +title: DataStream API 简介 +nav-id: datastream-api +nav-pos: 2 +nav-title: DataStream API 简介 +nav-parent_id: learn-flink +--- + + +该练习的重点是充分全面地了解 DataStream API，以便于编写流式应用入门。 + +* This will be replaced by the TOC +{:toc} + +## 什么能被转化成流？ + +Flink 的 Java 和 Scala DataStream API 可以将任何可序列化的对象转化为流。Flink 自带的序列化器有 + +- 基本类型，即 String、Long、Integer、Boolean、Array +- 复合类型：Tuples、POJOs 和 Scala case classes + +而且 Flink 会交给 Kryo 序列化其他类型。也可以将其他序列化器和 Flink 一起使用。特别是有良好支持的 Avro。 + +### Java tuples 和 POJOs + +Flink 的原生序列化器可以高效地操作 tuples 和 POJOs + +#### Tuples + +对于 Java，Flink 自带有 `Tuple0` 到 `Tuple25` 类型。 + +{% highlight java %} +Tuple2 flintstones = env.fromElements( + new Person("Fred", 35), + new Person("Wilma", 35), + new Person("Pebbles", 2)); + + DataStream adults = flintstones.filter(new FilterFunction () { + @Override + public boolean filter(Person person) throws Exception { + return person.age >= 18; + } + }); + + adults.print(); + + env.execute(); + } + + public static class Person { + public String name; + public Integer age; + public Person() {}; + + public Person(String name, Integer age) { + this.name = name; + this.age = age; + }; + + public String toString() { + return this.name.toString() + ": age " + this.age.toString(); + }; + } +} +{% endhighlight %} + +### Stream 执行环境 + +每个 Flink 应用都需要有执行环境，在该示例中为 `env`。流式应用需要用到 `StreamExecutionEnvironment`。 + +DataStream API 将你的应用构建为一个 job graph，并附加到 `StreamExecutionEnvironment` 。当调用 `env.execute()` 时此 graph 就被打包并发送到 JobManager 上，后者对作业并行处理并将其子任务分发给 Task Manager 来执行。每个作业的并行子任务将在 *task slot* 中执行。 + +注意，如果没有调用 execute()，应用就不会运行。 + +
+
+此分布式运行时取决于你的应用是否是可序列化的。它还要求所有依赖对集群中的每个节点均可用。
+
+### 基本的 stream source
+
+上述示例用 `env.fromElements(...)` 方法构造 `DataStream` 。这样将简单的流放在一起是为了方便用于原型或测试。`StreamExecutionEnvironment` 上还有一个 `fromCollection(Collection)` 方法。因此，你可以这样做： + +{% highlight java %} +List people = new ArrayList (); + +people.add(new Person("Fred", 35)); +people.add(new Person("Wilma", 35)); +people.add(new Person("Pebbles", 2)); + +DataStream flintstones = env.fromCollection(people); +{% endhighlight %} + +另一个获取数据到流中的便捷方法是用 socket + +{% highlight java %} +DataStream lines = env.socketTextStream("localhost", 9999) +{% endhighlight %} + +或读取文件 + +{% highlight java %} +DataStream lines = env.readTextFile("file:///path"); +{% endhighlight %} + +在真实的应用中，最常用的数据源是那些支持低延迟，高吞吐并行读取以及重复（高性能和容错能力为先决条件）的数据源，例如 Apache Kafka，Kinesis 和各种文件系统。REST API 和数据库也经常用于增强流处理的能力（stream enrichment）。 + +### 基本的 stream sink + +上述示例用 `adults.print()` 打印其结果到 task manager 的日志中（如果运行在 IDE 中时，将追加到你的 IDE 控制台）。它会对流中的每个元素都调用 `toString()` 方法。 + +输出看起来类似于 + + 1> Fred: age 35 + 2> Wilma: age 35 + +1> 和 2> 指出输出来自哪个 sub-task（即 thread） + +In production, commonly used sinks include the StreamingFileSink, various databases, +and several pub-sub systems. + +### 调试 + +在生产中，应用程序将在远程集群或一组容器中运行。如果集群或容器挂了，这就属于远程失败。JobManager 和 TaskManager 日志对于调试此类故障非常有用，但是更简单的是 Flink 支持在 IDE 内部进行本地调试。你可以设置断点，检查局部变量，并逐行执行代码。如果想了解 Flink 的工作原理和内部细节，查看 Flink 源码也是非常好的方法。 + +{% top %} + +## 动手实践 + +至此，你已经可以开始编写并运行一个简单的 DataStream 应用了。 +克隆 [flink-training repo](https://github.com/apache/flink-training/tree/{% if site.is_stable %}release-{{ site.version_title }}{% else %}master{% endif %}) 并在阅读完 README 中的指示后，开始尝试第一个练习吧： +[Filtering a Stream (Ride Cleansing)](https://github.com/apache/flink-training/tree/{% if site.is_stable %}release-{{ site.version_title }}{% else %}master{% endif %}/ride-cleansing)。 + +{% top %} + +## 更多阅读 + +- [Flink Serialization Tuning Vol. 1: Choosing your Serializer — if you can](https://flink.apache.org/news/2020/04/15/flink-serialization-tuning-vol-1.html) +- [Anatomy of a Flink Program]({% link dev/datastream_api.zh.md %}#anatomy-of-a-flink-program) +- [Data Sources]({% link dev/datastream_api.zh.md %}#data-sources) +- [Data Sinks]({% link dev/datastream_api.zh.md %}#data-sinks) +- [DataStream Connectors]({% link dev/connectors/index.zh.md %}) + +{% top %} diff --git a/docs/training/etl.md b/docs/learn-flink/etl.md similarity index 99% rename from docs/training/etl.md rename to docs/learn-flink/etl.md index 9337be315447b7..29cbdbfd269014 100644 --- a/docs/training/etl.md +++ b/docs/learn-flink/etl.md @@ -3,7 +3,7 @@ title: Data Pipelines & ETL nav-id: etl nav-pos: 3 nav-title: Data Pipelines & ETL -nav-parent_id: training +nav-parent_id: learn-flink --- + +* This will be replaced by the TOC +{:toc} + +## State Backends + +由 Flink 管理的 keyed state 是一种分片的键/值存储，每个 keyed state 的工作副本都保存在负责该键的 taskmanager 本地中。另外，Operator state 也保存在机器节点本地。Flink 定期获取所有状态的快照，并将这些快照复制到持久化的位置，例如分布式文件系统。 + +如果发生故障，Flink 可以恢复应用程序的完整状态并继续处理，就如同没有出现过异常。 + +Flink 管理的状态存储在 _state backend_ 中。Flink 有两种 state backend 的实现 -- 一种基于 RocksDB 内嵌 key/value 存储将其工作状态保存在磁盘上的，另一种基于堆的 state backend，将其工作状态保存在 Java 的堆内存中。这种基于堆的 state backend 有两种类型：FsStateBackend，将其状态快照持久化到分布式文件系统；MemoryStateBackend，它使用 JobManager 的堆保存状态快照。 + + + +
+ +当使用基于堆的 state backend 保存状态时，访问和更新涉及在堆上读写对象。但是对于保存在 `RocksDBStateBackend` 中的对象，访问和更新涉及序列化和反序列化，所以会有更大的开销。但 RocksDB 的状态量仅受本地磁盘大小的限制。还要注意，只有 `RocksDBStateBackend` 能够进行增量快照，这对于具有大量变化缓慢状态的应用程序来说是大有裨益的。 + +所有这些 state backends 都能够异步执行快照，这意味着它们可以在不妨碍正在进行的流处理的情况下执行快照。 + +{% top %} + +## 状态快照 + +### 定义 + +* _快照_ -- 是 Flink 作业状态全局一致镜像的通用术语。快照包括指向每个数据源的指针（例如，到文件或 Kafka 分区的偏移量）以及每个作业的有状态运算符的状态副本，该状态副本是处理了 sources 偏移位置之前所有的事件后而生成的状态。 + +* _Checkpoint_ -- 一种由 Flink 自动执行的快照，其目的是能够从故障中恢复。Checkpoints 可以是增量的，并为快速恢复进行了优化。 + +* _外部化的 Checkpoint_ -- 通常 checkpoints 不会被用户操纵。Flink 只保留作业运行时的最近的 _n_ 个 checkpoints（_n_ 可配置），并在作业取消时删除它们。但你可以将它们配置为保留，在这种情况下，你可以手动从中恢复。 + +* _Savepoint_ -- 用户出于某种操作目的（例如有状态的重新部署/升级/缩放操作）手动（或 API 调用）触发的快照。Savepoints 始终是完整的，并且已针对操作灵活性进行了优化。 + +### 状态快照如何工作？ + +Flink 使用 [Chandy-Lamport algorithm](https://en.wikipedia.org/wiki/Chandy-Lamport_algorithm) 算法的一种变体，称为异步 barrier 快照（_asynchronous barrier snapshotting_）。 + +当 checkpoint coordinator（job manager 的一部分）指示 task manager 开始 checkpoint 时，它会让所有 sources 记录它们的偏移量，并将编号的 _checkpoint barriers_ 插入到它们的流中。这些 barriers 流经 job graph，标注每个 checkpoint 前后的流部分。 + ++ + + +名称 +Working State +状态备份 +快照 ++ +RocksDBStateBackend +本地磁盘（tmp dir） +分布式文件系统 +全量 / 增量 ++ ++ +-
+
- 支持大于内存大小的状态 +
- 经验法则：比基于堆的后端慢10倍 +
+ +FsStateBackend +JVM Heap +分布式文件系统 +全量 ++ ++ +-
+
- 快速，需要大的堆内存 +
- 受限制于 GC +
+ +MemoryStateBackend +JVM Heap +JobManager JVM Heap +全量 ++ + ++ +-
+
- 适用于小状态（本地）的测试和实验 +
+
+Checkpoint _n_ 将包含每个 operator 的 state，这些 state 是对应的 operator 消费了**严格在 checkpoint barrier _n_ 之前的所有事件，并且不包含在此（checkpoint barrier _n_）后的任何事件**后而生成的状态。
+
+当 job graph 中的每个 operator 接收到 barriers 时，它就会记录下其状态。拥有两个输入流的 Operators（例如 `CoProcessFunction`）会执行 _barrier 对齐（barrier alignment）_ 以便当前快照能够包含消费两个输入流 barrier 之前（但不超过）的所有 events 而产生的状态。
+
+
+
+Flink 的 state backends 利用写时复制（copy-on-write）机制允许当异步生成旧版本的状态快照时，能够不受影响地继续流处理。只有当快照被持久保存后，这些旧版本的状态才会被当做垃圾回收。
+
+### 确保精确一次（exactly once）
+
+当流处理应用程序发生错误的时候，结果可能会产生丢失或者重复。Flink 根据你为应用程序和集群的配置，可以产生以下结果：
+
+- Flink 不会从快照中进行恢复（_at most once_）
+- 没有任何丢失，但是你可能会得到重复冗余的结果（_at least once_）
+- 没有丢失或冗余重复（_exactly once_）
+
+Flink 通过回退和重新发送 source 数据流从故障中恢复，当理想情况被描述为**精确一次**时，这并*不*意味着每个事件都将被精确一次处理。相反，这意味着 _每一个事件都会影响 Flink 管理的状态精确一次_。
+
+Barrier 只有在需要提供精确一次的语义保证时需要进行对齐（Barrier alignment）。如果不需要这种语义，可以通过配置 `CheckpointingMode.AT_LEAST_ONCE` 关闭 Barrier 对齐来提高性能。
+
+### 端到端精确一次
+
+为了实现端到端的精确一次，以便 sources 中的每个事件都仅精确一次对 sinks 生效，必须满足以下条件：
+
+1. 你的 sources 必须是可重放的，并且
+2. 你的 sinks 必须是事务性的（或幂等的）
+
+{% top %}
+
+## 实践练习
+
+[Flink Operations Playground]({% link try-flink/flink-operations-playground.zh.md %}) 包括有关 [Observing Failure & Recovery]({% link try-flink/flink-operations-playground.zh.md %}#observing-failure--recovery) 的部分。
+
+{% top %}
+
+## 延伸阅读
+
+- [Stateful Stream Processing]({% link concepts/stateful-stream-processing.zh.md %})
+- [State Backends]({% link ops/state/state_backends.zh.md %})
+- [Data Sources 和 Sinks 的容错保证]({% link dev/connectors/guarantees.zh.md %})
+- [开启和配置 Checkpointing]({% link dev/stream/state/checkpointing.zh.md %})
+- [Checkpoints]({% link ops/state/checkpoints.zh.md %})
+- [Savepoints]({% link ops/state/savepoints.zh.md %})
+- [大状态与 Checkpoint 调优]({% link ops/state/large_state_tuning.zh.md %})
+- [监控 Checkpoint]({% link monitoring/checkpoint_monitoring.zh.md %})
+- [Task 故障恢复]({% link dev/task_failure_recovery.zh.md %})
+
+{% top %}
diff --git a/docs/learn-flink/index.md b/docs/learn-flink/index.md
new file mode 100644
index 00000000000000..873c0204abfba4
--- /dev/null
+++ b/docs/learn-flink/index.md
@@ -0,0 +1,185 @@
+---
+title: "Learn Flink: Hands-on Training"
+nav-id: learn-flink
+nav-pos: 2
+nav-title: ' Learn Flink'
+nav-parent_id: root
+nav-show_overview: true
+---
+
+
+* This will be replaced by the TOC
+{:toc}
+
+## Goals and Scope of this Training
+
+This training presents an introduction to Apache Flink that includes just enough to get you started
+writing scalable streaming ETL, analytics, and event-driven applications, while leaving out a lot of
+(ultimately important) details. The focus is on providing straightforward introductions to Flink's
+APIs for managing state and time, with the expectation that having mastered these fundamentals,
+you'll be much better equipped to pick up the rest of what you need to know from the more detailed
+reference documentation. The links at the end of each section will lead you to where you
+can learn more.
+
+Specifically, you will learn:
+
+- how to implement streaming data processing pipelines
+- how and why Flink manages state
+- how to use event time to consistently compute accurate analytics
+- how to build event-driven applications on continuous streams
+- how Flink is able to provide fault-tolerant, stateful stream processing with exactly-once semantics
+
+This training focuses on four critical concepts: continuous processing of streaming data, event
+time, stateful stream processing, and state snapshots. This page introduces these concepts.
+
+{% info Note %} Accompanying this training is a set of hands-on exercises that will
+guide you through learning how to work with the concepts being presented. A link to the relevant
+exercise is provided at the end of each section.
+
+{% top %}
+
+## Stream Processing
+
+Streams are data's natural habitat. Whether it is events from web servers, trades from a stock
+exchange, or sensor readings from a machine on a factory floor, data is created as part of a stream.
+But when you analyze data, you can either organize your processing around _bounded_ or _unbounded_
+streams, and which of these paradigms you choose has profound consequences.
+
+
+
+**Batch processing** is the paradigm at work when you process a bounded data stream. In this mode of
+operation you can choose to ingest the entire dataset before producing any results, which means that
+it is possible, for example, to sort the data, compute global statistics, or produce a final report
+that summarizes all of the input.
+
+**Stream processing**, on the other hand, involves unbounded data streams. Conceptually, at least,
+the input may never end, and so you are forced to continuously process the data as it arrives.
+
+In Flink, applications are composed of **streaming dataflows** that may be transformed by
+user-defined **operators**. These dataflows form directed graphs that start with one or more
+**sources**, and end in one or more **sinks**.
+
+
+
+Often there is a one-to-one correspondence between the transformations in the program and the
+operators in the dataflow. Sometimes, however, one transformation may consist of multiple operators.
+
+An application may consume real-time data from streaming sources such as message queues or
+distributed logs, like Apache Kafka or Kinesis. But flink can also consume bounded, historic data
+from a variety of data sources. Similarly, the streams of results being produced by a Flink
+application can be sent to a wide variety of systems that can be connected as sinks.
+
+
+
+### Parallel Dataflows
+
+Programs in Flink are inherently parallel and distributed. During execution, a
+*stream* has one or more **stream partitions**, and each *operator* has one or
+more **operator subtasks**. The operator subtasks are independent of one
+another, and execute in different threads and possibly on different machines or
+containers.
+
+The number of operator subtasks is the **parallelism** of that particular
+operator.
+Different operators of the same program may have different levels of
+parallelism.
+
+
+
+Streams can transport data between two operators in a *one-to-one* (or
+*forwarding*) pattern, or in a *redistributing* pattern:
+
+ - **One-to-one** streams (for example between the *Source* and the *map()*
+ operators in the figure above) preserve the partitioning and ordering of
+ the elements. That means that subtask[1] of the *map()* operator will see
+ the same elements in the same order as they were produced by subtask[1] of
+ the *Source* operator.
+
+ - **Redistributing** streams (as between *map()* and *keyBy/window* above, as
+ well as between *keyBy/window* and *Sink*) change the partitioning of
+ streams. Each *operator subtask* sends data to different target subtasks,
+ depending on the selected transformation. Examples are *keyBy()* (which
+ re-partitions by hashing the key), *broadcast()*, or *rebalance()* (which
+ re-partitions randomly). In a *redistributing* exchange the ordering among
+ the elements is only preserved within each pair of sending and receiving
+ subtasks (for example, subtask[1] of *map()* and subtask[2] of
+ *keyBy/window*). So, for example, the redistribution between the keyBy/window and
+ the Sink operators shown above introduces non-determinism regarding the
+ order in which the aggregated results for different keys arrive at the Sink.
+
+{% top %}
+
+## Timely Stream Processing
+
+For most streaming applications it is very valuable to be able re-process historic data with the
+same code that is used to process live data -- and to produce deterministic, consistent results,
+regardless.
+
+It can also be crucial to pay attention to the order in which events occurred, rather than the order
+in which they are delivered for processing, and to be able to reason about when a set of events is
+(or should be) complete. For example, consider the set of events involved in an e-commerce
+transaction, or financial trade.
+
+These requirements for timely stream processing can be met by using event time timestamps that are
+recorded in the data stream, rather than using the clocks of the machines processing the data.
+
+{% top %}
+
+## Stateful Stream Processing
+
+Flink's operations can be stateful. This means that how one event is handled can depend on the
+accumulated effect of all the events that came before it. State may be used for something simple,
+such as counting events per minute to display on a dashboard, or for something more complex, such as
+computing features for a fraud detection model.
+
+A Flink application is run in parallel on a distributed cluster. The various parallel instances of a
+given operator will execute independently, in separate threads, and in general will be running on
+different machines.
+
+The set of parallel instances of a stateful operator is effectively a sharded key-value store. Each
+parallel instance is responsible for handling events for a specific group of keys, and the state for
+those keys is kept locally.
+
+The diagram below shows a job running with a parallelism of two across the first three operators in
+the job graph, terminating in a sink that has a parallelism of one. The third operator is stateful,
+and you can see that a fully-connected network shuffle is occurring between the second and third
+operators. This is being done to partition the stream by some key, so that all of the events that
+need to be processed together, will be.
+
+
+
+State is always accessed locally, which helps Flink applications achieve high throughput and
+low-latency. You can choose to keep state on the JVM heap, or if it is too large, in efficiently
+organized on-disk data structures.
+
+
+
+{% top %}
+
+## Fault Tolerance via State Snapshots
+
+Flink is able to provide fault-tolerant, exactly-once semantics through a combination of state
+snapshots and stream replay. These snapshots capture the entire state of the distributed pipeline,
+recording offsets into the input queues as well as the state throughout the job graph that has
+resulted from having ingested the data up to that point. When a failure occurs, the sources are
+rewound, the state is restored, and processing is resumed. As depicted above, these state snapshots
+are captured asynchronously, without impeding the ongoing processing.
+
+{% top %}
diff --git a/docs/learn-flink/index.zh.md b/docs/learn-flink/index.zh.md
new file mode 100644
index 00000000000000..75be8be376e6ff
--- /dev/null
+++ b/docs/learn-flink/index.zh.md
@@ -0,0 +1,117 @@
+---
+title: 实践练习
+nav-id: learn-flink
+nav-pos: 2
+nav-title: ' 实践练习'
+nav-parent_id: root
+nav-show_overview: true
+---
+
+
+* This will be replaced by the TOC
+{:toc}
+
+## 本章教程的目标及涵盖范围
+
+本章教程对 Apache Flink 的基本概念进行了介绍，虽然省略了许多重要细节，但是如果你掌握了本章内容，就足以实现可扩展并行度的 ETL、数据分析以及事件驱动的流式应用程序。本章重点对 Flink API 中的状态管理和时间进行了介绍，掌握了这些基础知识后，你将能更好地从其他详细参考文档中获取和掌握你所需要的知识。每小节结尾都有链接去引导你了解更多内容。
+
+具体来说，你将在本章学习到以下内容：
+
+- 如何实现流数据处理管道（pipelines）
+- Flink 如何管理状态以及为何需要管理状态
+- 如何使用事件时间（event time）来一致并准确地进行计算分析
+- 如何在源源不断的数据流上构建事件驱动的应用程序
+- Flink 如何提供具有精确一次（exactly-once）计算语义的可容错、有状态流处理
+
+本章教程着重介绍四个概念：源源不断的流式数据处理、事件时间、有状态流处理和状态快照。基本概念介绍如下。
+
+{% info Note %} 每小节教程都有实践练习引导你如何在程序中使用其所述的概念，并在小节结尾都提供了相关实践练习的代码链接。
+
+{% top %}
+
+## 流处理
+
+在自然环境中，数据的产生原本就是流式的。无论是来自 Web 服务器的事件数据，证券交易所的交易数据，还是来自工厂车间机器上的传感器数据，其数据都是流式的。但是当你分析数据时，可以围绕 _有界流_（_bounded_）或 _无界流_（_unbounded_）两种模型来组织处理数据，当然，选择不同的模型，程序的执行和处理方式也都会不同。
+
+
+
+**批处理**是有界数据流处理的范例。在这种模式下，你可以选择在计算结果输出之前输入整个数据集，这也就意味着你可以对整个数据集的数据进行排序、统计或汇总计算后再输出结果。
+
+**流处理**正相反，其涉及无界数据流。至少理论上来说，它的数据输入永远不会结束，因此程序必须持续不断地对到达的数据进行处理。
+
+在 Flink 中，应用程序由用户自定义**算子**转换而来的**流式 dataflows** 所组成。这些流式 dataflows 形成了有向图，以一个或多个**源**（source）开始，并以一个或多个**汇**（sink）结束。
+
+
+
+通常，程序代码中的 transformation 和 dataflow 中的算子（operator）之间是一一对应的。但有时也会出现一个 transformation 包含多个算子的情况，如上图所示。
+
+Flink 应用程序可以消费来自消息队列或分布式日志这类流式数据源（例如 Apache Kafka 或 Kinesis）的实时数据，也可以从各种的数据源中消费有界的历史数据。同样，Flink 应用程序生成的结果流也可以发送到各种数据汇中。
+
+
+
+### 并行 Dataflows
+
+Flink 程序本质上是分布式并行程序。在程序执行期间，一个流有一个或多个**流分区**（Stream Partition），每个算子有一个或多个**算子子任务**（Operator Subtask）。每个子任务彼此独立，并在不同的线程中运行，或在不同的计算机或容器中运行。
+
+算子子任务数就是其对应算子的**并行度**。在同一程序中，不同算子也可能具有不同的并行度。
+
+
+
+Flink 算子之间可以通过*一对一*（*直传*）模式或*重新分发*模式传输数据：
+
+ - **一对一**模式（例如上图中的 *Source* 和 *map()* 算子之间）可以保留元素的分区和顺序信息。这意味着 *map()* 算子的 subtask[1] 输入的数据以及其顺序与 *Source* 算子的 subtask[1] 输出的数据和顺序完全相同，即同一分区的数据只会进入到下游算子的同一分区。
+
+ - **重新分发**模式（例如上图中的 *map()* 和 *keyBy/window* 之间，以及 *keyBy/window* 和 *Sink* 之间）则会更改数据所在的流分区。当你在程序中选择使用不同的 *transformation*，每个*算子子任务*也会根据不同的 transformation 将数据发送到不同的目标子任务。例如以下这几种 transformation 和其对应分发数据的模式：*keyBy()*（通过散列键重新分区）、*broadcast()*（广播）或 *rebalance()*（随机重新分发）。在*重新分发*数据的过程中，元素只有在每对输出和输入子任务之间才能保留其之间的顺序信息（例如，*keyBy/window* 的 subtask[2] 接收到的 *map()* 的 subtask[1] 中的元素都是有序的）。因此，上图所示的 *keyBy/window* 和 *Sink* 算子之间数据的重新分发时，不同键（key）的聚合结果到达 Sink 的顺序是不确定的。
+
+{% top %}
+
+## 自定义时间流处理
+
+对于大多数流数据处理应用程序而言，能够使用处理实时数据的代码重新处理历史数据并产生确定并一致的结果非常有价值。
+
+在处理流式数据时，我们通常更需要关注事件本身发生的顺序而不是事件被传输以及处理的顺序，因为这能够帮助我们推理出一组事件（事件集合）是何时发生以及结束的。例如电子商务交易或金融交易中涉及到的事件集合。
+
+为了满足上述这类的实时流处理场景，我们通常会使用记录在数据流中的事件时间的时间戳，而不是处理数据的机器时钟的时间戳。
+
+{% top %}
+
+## 有状态流处理
+
+Flink 中的算子可以是有状态的。这意味着如何处理一个事件可能取决于该事件之前所有事件数据的累积结果。Flink 中的状态不仅可以用于简单的场景（例如统计仪表板上每分钟显示的数据），也可以用于复杂的场景（例如训练作弊检测模型）。
+
+Flink 应用程序可以在分布式群集上并行运行，其中每个算子的各个并行实例会在单独的线程中独立运行，并且通常情况下是会在不同的机器上运行。
+
+有状态算子的并行实例组在存储其对应状态时通常是按照键（key）进行分片存储的。每个并行实例算子负责处理一组特定键的事件数据，并且这组键对应的状态会保存在本地。
+
+如下图的 Flink 作业，其前三个算子的并行度为 2，最后一个 sink 算子的并行度为 1，其中第三个算子是有状态的，并且你可以看到第二个算子和第三个算子之间是全互联的（fully-connected），它们之间通过网络进行数据分发。通常情况下，实现这种类型的 Flink 程序是为了通过某些键对数据流进行分区，以便将需要一起处理的事件进行汇合，然后做统一计算处理。
+
+
+
+Flink 应用程序的状态访问都在本地进行，因为这有助于其提高吞吐量和降低延迟。通常情况下 Flink 应用程序都是将状态存储在 JVM 堆上，但如果状态太大，我们也可以选择将其以结构化数据格式存储在高速磁盘中。
+
+
+
+{% top %}
+
+## 通过状态快照实现的容错
+
+通过状态快照和流重放两种方式的组合，Flink 能够提供可容错的，精确一次计算的语义。这些状态快照在执行时会获取并存储分布式 pipeline 中整体的状态，它会将数据源中消费数据的偏移量记录下来，并将整个 job graph 中算子获取到该数据（记录的偏移量对应的数据）时的状态记录并存储下来。当发生故障时，Flink 作业会恢复上次存储的状态，重置数据源从状态中记录的上次消费的偏移量开始重新进行消费处理。而且状态快照在执行时会异步获取状态并存储，并不会阻塞正在进行的数据处理逻辑。
+
+{% top %}
diff --git a/docs/training/streaming_analytics.md b/docs/learn-flink/streaming_analytics.md
similarity index 95%
rename from docs/training/streaming_analytics.md
rename to docs/learn-flink/streaming_analytics.md
index 7dc0503ac48cde..c826ffbee9509f 100644
--- a/docs/training/streaming_analytics.md
+++ b/docs/learn-flink/streaming_analytics.md
@@ -3,7 +3,7 @@ title: Streaming Analytics
nav-id: analytics
nav-pos: 4
nav-title: Streaming Analytics
-nav-parent_id: training
+nav-parent_id: learn-flink
---
+
+
+ + + +``` + +Running `mvn package` in the same directory will create a `jars/` folder containing all the jar files, +which you can add to the desired folder, Docker image etc. diff --git a/docs/ops/deployment/index.zh.md b/docs/ops/deployment/index.zh.md index 4499abb5c086c8..f318ff4a763440 100644 --- a/docs/ops/deployment/index.zh.md +++ b/docs/ops/deployment/index.zh.md @@ -29,6 +29,84 @@ When deciding how and where to run Flink, there's a wide range of options availa * This will be replaced by the TOC {:toc} +## Deployment Modes + +Flink can execute applications in one of three ways: + - in Session Mode, + - in a Per-Job Mode, or + - in Application Mode. + + The above modes differ in: + - the cluster lifecycle and resource isolation guarantees + - whether the application's `main()` method is executed on the client or on the cluster. + +#### Session Mode + +*Session mode* assumes an already running cluster and uses the resources of that cluster to execute any +submitted application. Applications executed in the same (session) cluster use, and consequently compete +for, the same resources. This has the advantage that you do not pay the resource overhead of spinning up +a full cluster for every submitted job. But, if one of the jobs misbehaves or brings down a Task Manager, +then all jobs running on that Task Manager will be affected by the failure. This, apart from a negative +impact on the job that caused the failure, implies a potential massive recovery process with all the +restarting jobs accessing the filesystem concurrently and making it unavailable to other services. +Additionally, having a single cluster running multiple jobs implies more load for the JobManager, who +is responsible for the book-keeping of all the jobs in the cluster. + +#### Per-Job Mode + +Aiming at providing better resource isolation guarantees, the *Per-Job* mode uses the available cluster manager +framework (e.g. YARN, Kubernetes) to spin up a cluster for each submitted job. This cluster is available to +that job only. When the job finishes, the cluster is torn down and any lingering resources (files, etc) are +cleared up. This provides better resource isolation, as a misbehaving job can only bring down its own +Task Managers. In addition, it spreads the load of book-keeping across multiple JobManagers, as there is +one per job. For these reasons, the *Per-Job* resource allocation model is the preferred mode by many +production reasons. + +#### Application Mode + +In all the above modes, the application's `main()` method is executed on the client side. This process +includes downloading the application's dependencies locally, executing the `main()` to extract a representation +of the application that Flink's runtime can understand (i.e. the `JobGraph`) and ship the dependencies and +the `JobGraph(s)` to the cluster. This makes the Client a heavy resource consumer as it may need substantial +network bandwidth to download dependencies and ship binaries to the cluster, and CPU cycles to execute the +`main()`. This problem can be more pronounced when the Client is shared across users. + +Building on this observation, the *Application Mode* creates a cluster per submitted application, but this time, +the `main()` method of the application is executed on the JobManager. Creating a cluster per application can be +seen as creating a session cluster shared only among the jobs of a particular application, and torn down when +the application finishes. With this architecture, the *Application Mode* provides the same resource isolation +and load balancing guarantees as the *Per-Job* mode, but at the granularity of a whole application. Executing +the `main()` on the JobManager allows for saving the CPU cycles required, but also save the bandwidth required +for downloading the dependencies locally. Furthermore, it allows for more even spread of the network load of +downloading the dependencies of the applications in the cluster, as there is one JobManager per application. + ++ ++ +org.apache.maven.plugins +maven-dependency-plugin +3.1.2 ++ ++ +copy-dependencies +package +copy-dependencies jars + Note: In the Application Mode, the `main()` is executed on the cluster and not on the client, + as in the other modes. This may have implications for your code as, for example, any paths you register in + your environment using the `registerCachedFile()` must be accessible by the JobManager of your application. ++ +Compared to the *Per-Job* mode, the *Application Mode* allows the submission of applications consisting of +multiple jobs. The order of job execution is not affected by the deployment mode but by the call used +to launch the job. Using `execute()`, which is blocking, establishes an order and it will lead to the +execution of the "next" job being postponed until "this" job finishes. Using `executeAsync()`, which is +non-blocking, will lead to the "next" job starting before "this" job finishes. + ++ Attention: The Application Mode allows for multi-`execute()` applications but + High-Availability is not supported in these cases. High-Availability in Application Mode is only + supported for single-`execute()` applications. ++ +#### Summary + +In *Session Mode*, the cluster lifecycle is independent of that of any job running on the cluster +and the resources are shared across all jobs. The *Per-Job* mode pays the price of spinning up a cluster +for every submitted job, but this comes with better isolation guarantees as the resources are not shared +across jobs. In this case, the lifecycle of the cluster is bound to that of the job. Finally, the +*Application Mode* creates a session cluster per application and executes the application's `main()` +method on the cluster. + ## Deployment Targets Apache Flink ships with first class support for a number of common deployment targets. @@ -41,7 +119,7 @@ Apache Flink ships with first class support for a number of common deployment ta- Deploy Flink on-top Apache Hadoop's resource manager -
Learn more + Deploy Flink on-top of Apache Hadoop's resource manager +
Learn more
When creating a Flink Kubernetes session cluster, the Flink client will first connect to the Kubernetes ApiServer to submit the cluster description, including ConfigMap spec, Job Manager Service spec, Job Manager Deployment spec and Owner Reference.
-Kubernetes will then create the Flink master deployment, during which time the Kubelet will pull the image, prepare and mount the volume, and then execute the start command.
-After the master pod has launched, the Dispatcher and KubernetesResourceManager are available and the cluster is ready to accept one or more jobs.
+Kubernetes will then create the JobManager deployment, during which time the Kubelet will pull the image, prepare and mount the volume, and then execute the start command.
+After the JobManager pod has launched, the Dispatcher and KubernetesResourceManager are available and the cluster is ready to accept one or more jobs.
When users submit jobs through the Flink client, the job graph will be generated by the client and uploaded along with users jars to the Dispatcher.
-A JobMaster for that Job will be then be spawned.
-The JobMaster requests resources, known as slots, from the KubernetesResourceManager.
+The JobManager requests resources, known as slots, from the KubernetesResourceManager.
If no slots are available, the resource manager will bring up TaskManager pods and registering them with the cluster.
{% top %}
diff --git a/docs/ops/deployment/native_kubernetes.zh.md b/docs/ops/deployment/native_kubernetes.zh.md
index e341154af2d8da..6d1c0ae0112e12 100644
--- a/docs/ops/deployment/native_kubernetes.zh.md
+++ b/docs/ops/deployment/native_kubernetes.zh.md
@@ -24,46 +24,44 @@ specific language governing permissions and limitations
under the License.
-->
-This page describes how to deploy a Flink session cluster natively on [Kubernetes](https://kubernetes.io).
+本页面描述了如何在 [Kubernetes](https://kubernetes.io) 原生地部署 Flink session 集群。
* This will be replaced by the TOC
{:toc}
-When creating a Flink Kubernetes session cluster, the Flink client will first connect to the Kubernetes ApiServer to submit the cluster description, including ConfigMap spec, Job Manager Service spec, Job Manager Deployment spec and Owner Reference.
-Kubernetes will then create the Flink master deployment, during which time the Kubelet will pull the image, prepare and mount the volume, and then execute the start command.
-After the master pod has launched, the Dispatcher and KubernetesResourceManager are available and the cluster is ready to accept one or more jobs.
+创建 Flink Kubernetes session 集群时，Flink 客户端首先将连接到 Kubernetes ApiServer 提交集群描述信息，包括 ConfigMap 描述信息、Job Manager Service 描述信息、Job Manager Deployment 描述信息和 Owner Reference。
+Kubernetes 将创建 JobManager 的 deployment，在此期间 Kubelet 将拉取镜像，准备并挂载卷，然后执行 start 命令。
+JobManager pod 启动后，Dispatcher 和 KubernetesResourceManager 服务会相继启动，然后集群准备完成，并等待提交作业。
-When users submit jobs through the Flink client, the job graph will be generated by the client and uploaded along with users jars to the Dispatcher.
-A JobMaster for that Job will be then be spawned.
+当用户通过 Flink 客户端提交作业时，将通过客户端生成 jobGraph 并将其与用户 jar 一起上传到 Dispatcher。
+然后 Dispatcher 会为每个 job 启动一个单独的 JobMaster。
-The JobMaster requests resources, known as slots, from the KubernetesResourceManager.
-If no slots are available, the resource manager will bring up TaskManager pods and registering them with the cluster.
+JobManager 向 KubernetesResourceManager 请求被称为 slots 的资源。
+如果没有可用的 slots，KubernetesResourceManager 将拉起 TaskManager pod 并且把它们注册到集群中。
{% top %}
diff --git a/docs/ops/deployment/yarn_setup.md b/docs/ops/deployment/yarn_setup.md
index 4dcd2903140798..27a29d63205dea 100644
--- a/docs/ops/deployment/yarn_setup.md
+++ b/docs/ops/deployment/yarn_setup.md
@@ -190,7 +190,7 @@ Action "run" compiles and runs a program.
method or "getPlan()" method. Only needed
if the JAR file does not specify the class
in its manifest.
- -m,--jobmanager taskmanager.memory.fraction
[`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size) | [`jobmanager.memory.heap.size`](../config.html#jobmanager-memory-heap-size) | +{:.table-bordered} +
+ +Flink 自带的[默认 flink-conf.yaml](#default-configuration-in-flink-confyaml) 文件指定了 [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)（*>= 1.10*）和 [`jobmanager.memory.process.size`](../config.html#jobmanager-memory-process-size) (*>= 1.11*)，以便与此前的行为保持一致。 可以使用这张[电子表格](https://docs.google.com/spreadsheets/d/1mJaMkMPfDJJ-w6nMXALYmTc4XxiV30P5U7DzgwLkSoE)来估算和比较原本的和新的内存配置下的计算结果。 -## 配置参数变化 + + +## 升级 TaskManager 内存配置 + + + +### 配置参数变化 本节简要列出了 *Flink 1.10* 引入的配置参数变化，并援引其他章节中关于如何升级到新配置参数的相关描述。 @@ -64,16 +77,16 @@ Flink 自带的[默认 flink-conf.yaml](#flink-confyaml-中的默认配置) 文
taskmanager.memory.off-heap
taskmanager.memory.preallocate
| Deprecated option | -Interpreted as | +弃用的配置参数 | +对应的新配置参数 | |||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
taskmanager.memory.size |
- taskmanager.memory.managed.size。请参考如何升级托管内存。 | +taskmanager.memory.managed.size。请参考如何升级托管内存。 | ||||||||||||||||||||||||||||
taskmanager.network.memory.min |
@@ -119,11 +132,14 @@ Flink 自带的[默认 flink-conf.yaml](#flink-confyaml-中的默认配置) 文
尽管网络内存的配置参数没有发生太多变化，我们仍建议您检查其配置结果。
网络内存的大小可能会受到其他内存部分大小变化的影响，例如总内存变化时，根据占比计算出的网络内存也可能发生变化。
-请参考[内存模型详解](mem_detail.html)。
+请参考[内存模型详解](mem_setup_tm.html#detailed-memory-model)。
+
+容器切除（Cut-Off）内存相关的配置参数（`containerized.heap-cutoff-ratio` 和 `containerized.heap-cutoff-min`）将不再对 TaskManager 进程生效。
+请参考[如何升级容器切除内存](#container-cut-off-memory)。
-容器切除（Cut-Off）内存相关的配置参数（`containerized.heap-cutoff-ratio` 和 `containerized.heap-cutoff-min`）将不再对进程生效。
+
-## 总内存（原堆内存）
+### 总内存（原堆内存）
在原本的内存配置方法中，用于指定用于 Flink 的总内存的配置参数是 `taskmanager.heap.size` 或 `taskmanager.heap.mb`。
尽管这两个参数以“堆（Heap）”命名，实际上它们指定的内存既包含了 JVM 堆内存，也包含了其他堆外内存部分。
@@ -137,81 +153,152 @@ Flink 在 Mesos 上还有另一个具有同样语义的配置参数 `mesos.resou
建议您尽早使用新的配置参数取代启用的配置参数，它们在今后的版本中可能会被彻底移除。
-请参考[如何配置总内存](mem_setup.html#配置总内存).
+请参考[如何配置总内存](mem_setup.html#configure-total-memory).
-## JVM 堆内存
+
+
+### JVM 堆内存
此前，JVM 堆空间由托管内存（仅在配置为堆上时）及 Flink 用到的所有其他堆内存组成。
这里的其他堆内存是由总内存减去所有其他非堆内存得到的。
-请参考[如何升级托管内存](#托管内存)。
+请参考[如何升级托管内存](#managed-memory)。
现在，如果仅配置了*Flink总内存*或*进程总内存*，JVM 的堆空间依然是根据总内存减去所有其他非堆内存得到的。
-请参考[如何配置总内存](mem_setup.html#配置总内存)。
+请参考[如何配置总内存](mem_setup.html#configure-total-memory)。
-此外，你现在可以更直接地控制用于任务和算子的 JVM 的堆内存（[`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size)），详见[任务堆内存](mem_setup.html#任务算子堆内存)。
+此外，你现在可以更直接地控制用于任务和算子的 JVM 的堆内存（[`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size)），详见[任务堆内存](mem_setup_tm.html#task-operator-heap-memory)。
如果流处理作业选择使用 Heap State Backend（[MemoryStateBackend](../state/state_backends.html#memorystatebackend)
或 [FsStateBackend](../state/state_backends.html#fsstatebackend)），那么它同样需要使用 JVM 堆内存。
Flink 现在总是会预留一部分 JVM 堆内存供框架使用（[`taskmanager.memory.framework.heap.size`](../config.html#taskmanager-memory-framework-heap-size)）。
-请参考[框架内存](mem_detail.html#框架内存)。
+请参考[框架内存](mem_setup_tm.html#framework-memory)。
+
+
-## 托管内存
+### 托管内存
-请参考[如何配置托管内存](mem_setup.html#托管内存)。
+请参考[如何配置托管内存](mem_setup_tm.html#managed-memory)。
-### 明确的大小
+
+
+#### 明确的大小
原本用于指定明确的托管内存大小的配置参数（`taskmanager.memory.size`）已被弃用，与它具有相同语义的新配置参数为 [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)。
建议使用新的配置参数，原本的配置参数在今后的版本中可能会被彻底移除。
-### 占比
+
+
+#### 占比
此前，如果不指定明确的大小，也可以将托管内存配置为占用总内存减去网络内存和容器切除内存（仅在 [Yarn](../deployment/yarn_setup.html) 和
[Mesos](../deployment/mesos.html) 上）之后剩余部分的固定比例（`taskmanager.memory.fraction`）。
该配置参数已经被彻底移除，配置它不会产生任何效果。
请使用新的配置参数 [`taskmanager.memory.managed.fraction`](../config.html#taskmanager-memory-managed-fraction)。
-在未通过 [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size) 指定明确大小的情况下，新的配置参数将指定[托管内存](mem_setup.html#托管内存)在 [Flink 总内存](mem_setup.html#配置总内存)中的所占比例。
+在未通过 [`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size) 指定明确大小的情况下，新的配置参数将指定[托管内存](mem_setup_tm.html#managed-memory)在 [Flink 总内存](mem_setup.html#configure-total-memory)中的所占比例。
-### RocksDB State Backend
+
-流处理作业如果选择使用 [RocksDBStateBackend](../state/state_backends.html#rocksdbstatebackend)，它使用的本地内存现在也被归为[托管内存](mem_setup.html#托管内存)。
-默认情况下，RocksDB 将限制其内存用量不超过[托管内存](mem_setup.html#托管内存)大小，以避免在 [Yarn](../deployment/yarn_setup.html) 或 [Mesos](../deployment/mesos.html) 上容器被杀。你也可以通过设置 [state.backend.rocksdb.memory.managed](../config.html#state-backend-rocksdb-memory-managed) 来关闭 RocksDB 的内存控制。
-请参考[如何升级容器切除内存](#容器切除cut-off内存)。
+#### RocksDB State Backend
-### 其他变化
+流处理作业如果选择使用 [RocksDBStateBackend](../state/state_backends.html#rocksdbstatebackend)，它使用的本地内存现在也被归为[托管内存](mem_setup_tm.html#managed-memory)。
+默认情况下，RocksDB 将限制其内存用量不超过[托管内存](mem_setup_tm.html#managed-memory)大小，以避免在 [Yarn](../deployment/yarn_setup.html) 或 [Mesos](../deployment/mesos.html) 上容器被杀。你也可以通过设置 [state.backend.rocksdb.memory.managed](../config.html#state-backend-rocksdb-memory-managed) 来关闭 RocksDB 的内存控制。
+请参考[如何升级容器切除内存](#container-cut-off-memory)。
+
+
+
+#### 其他变化
此外，Flink 1.10 对托管内存还引入了下列变化：
-* [托管内存](mem_setup.html#托管内存)现在总是在堆外。配置参数 `taskmanager.memory.off-heap` 已被彻底移除，配置它不会产生任何效果。
-* [托管内存](mem_setup.html#托管内存)现在使用本地内存而非直接内存。这意味着托管内存将不在 JVM 直接内存限制的范围内。
-* [托管内存](mem_setup.html#托管内存)现在总是惰性分配的。配置参数 `taskmanager.memory.preallocate` 已被彻底移除，配置它不会产生任何效果。
+* [托管内存](mem_setup_tm.html#managed-memory)现在总是在堆外。配置参数 `taskmanager.memory.off-heap` 已被彻底移除，配置它不会产生任何效果。
+* [托管内存](mem_setup_tm.html#managed-memory)现在使用本地内存而非直接内存。这意味着托管内存将不在 JVM 直接内存限制的范围内。
+* [托管内存](mem_setup_tm.html#managed-memory)现在总是惰性分配的。配置参数 `taskmanager.memory.preallocate` 已被彻底移除，配置它不会产生任何效果。
+
+
+
+## 升级 JobManager 内存配置
+
+在原本的内存配置方法中，用于指定 *JVM 堆内存* 的配置参数是:
+* `jobmanager.heap.size`
+* `jobmanager.heap.mb`
+
+尽管这两个参数以“堆（Heap）”命名，在此之前它们实际上只有在[独立部署模式](../deployment/cluster_setup.html)才完全对应于 *JVM 堆内存*。
+在容器化部署模式下（[Kubernetes](../deployment/kubernetes.html) 和 [Yarn](../deployment/yarn_setup.html)），它们指定的内存还包含了其他堆外内存部分。
+*JVM 堆空间*的实际大小，是参数指定的大小减去容器切除（Cut-Off）内存后剩余的部分。
+容器切除内存在 *1.11* 及以上版本中已被彻底移除。
+
+上述两个参数此前对 [Mesos](../deployment/mesos.html) 部署模式并不生效。
+Flink 在 Mesos 上启动 JobManager 进程时并未设置任何 JVM 内存参数。
+从 *1.11* 版本开始，Flink 将采用与[独立部署模式](../deployment/cluster_setup.html)相同的方式设置这些参数。
+
+这两个配置参数目前已被弃用。
+如果配置了上述弃用的参数，同时又没有配置与之对应的新配置参数，那它们将按如下规则对应到新的配置参数。
+* 独立部署模式（Standalone Deployment）、Mesos 部署模式下：JVM 堆内存（[`jobmanager.memory.heap.size`](../config.html#jobmanager-memory-heap-size)）
+* 容器化部署模式（Containerized Deployement）下（Kubernetes、Yarn）：进程总内存（[`jobmanager.memory.process.size`](../config.html#jobmanager-memory-process-size)）
+
+建议您尽早使用新的配置参数取代启用的配置参数，它们在今后的版本中可能会被彻底移除。
+
+如果仅配置了 *Flink 总内存*或*进程总内存*，那么 [JVM 堆内存](mem_setup_jobmanager.html#configure-jvm-heap)将是总内存减去其他内存部分后剩余的部分。
+请参考[如何配置总内存](mem_setup.html#configure-total-memory)。
+此外，也可以通过配置 [`jobmanager.memory.heap.size`](../config.html#jobmanager-memory-heap-size) 的方式直接指定 [JVM 堆内存](mem_setup_jobmanager.html#configure-jvm-heap)。
+
+
+
+## Flink JVM 进程内存限制
+
+从 *1.10* 版本开始，Flink 通过设置相应的 JVM 参数，对 TaskManager 进程使用的 *JVM Metaspace* 和 *JVM 直接内存*进行限制。
+从 *1.11* 版本开始，Flink 同样对 JobManager 进程使用的 *JVM Metaspace* 进行限制。
+此外，还可以通过设置 [`jobmanager.memory.enable-jvm-direct-memory-limit`](../config.html#jobmanager-memory-enable-jvm-direct-memory-limit) 对 JobManager 进程的 *JVM 直接内存*进行限制。
+请参考 [JVM 参数](mem_setup.html#jvm-parameters)。
+
+Flink 通过设置上述 JVM 内存限制降低内存泄漏问题的排查难度，以避免出现[容器内存溢出](mem_trouble.html#container-memory-exceeded)等问题。
+请参考常见问题中关于 [JVM Metaspace](mem_trouble.html#outofmemoryerror-metaspace) 和 [JVM 直接内存](mem_trouble.html#outofmemoryerror-direct-buffer-memory) *OutOfMemoryError* 异常的描述。
+
+
## 容器切除（Cut-Off）内存
在容器化部署模式（Containerized Deployment）下，此前你可以指定切除内存。
这部分内存将预留给所有未被 Flink 计算在内的内存开销。
其主要来源是不受 Flink 直接管理的依赖使用的内存，例如 RocksDB、JVM 内部开销等。
-相应的配置参数（`containerized.heap-cutoff-ratio` 和 `containerized.heap-cutoff-min`）现在不再对 TaskExecutor 生效。
+相应的配置参数（`containerized.heap-cutoff-ratio` 和 `containerized.heap-cutoff-min`）不再生效。
新的内存配置方法引入了新的内存组成部分来具体描述这些内存用量。
-流处理作业如果使用了 [RocksDBStateBackend](../state/state_backends.html#the-rocksdbstatebackend)，RocksDB 使用的本地内存现在将被归为[托管内存](mem_setup.html#托管内存)。
-默认情况下，RocksDB 将限制其内存用量不超过[托管内存](mem_setup.html#托管内存)大小。
-请同时参考[如何升级托管内存](#托管内存)以及[如何配置托管内存](mem_setup.html#托管内存)。
+
+
+### TaskManager
-其他直接内存或堆外内存开销，现在可以通过下列配置参数进行设置：
+流处理作业如果使用了 [RocksDBStateBackend](../state/state_backends.html#the-rocksdbstatebackend)，RocksDB 使用的本地内存现在将被归为[托管内存](mem_setup_tm.html#managed-memory)。
+默认情况下，RocksDB 将限制其内存用量不超过[托管内存](mem_setup_tm.html#managed-memory)大小。
+请同时参考[如何升级托管内存](#managed-memory)以及[如何配置托管内存](mem_setup_tm.html#managed-memory)。
+
+其他堆外（直接或本地）内存开销，现在可以通过下列配置参数进行设置：
* 任务堆外内存（[`taskmanager.memory.task.off-heap.size`](../config.html#taskmanager-memory-task-off-heap-size)）
* 框架堆外内存（[`taskmanager.memory.framework.off-heap.size`](../config.html#taskmanager-memory-framework-off-heap-size)）
* JVM Metaspace（[`taskmanager.memory.jvm-metaspace.size`](../config.html#taskmanager-memory-jvm-metaspace-size)）
-* JVM 开销（请参考[内存模型详解](mem_detail.html)）
+* [JVM 开销](mem_setup_tm.html#detailed-memory-model)
+
+
-提示 JobManager 进程仍保留了容器切除内存，相关配置项和此前一样仍对 JobManager 生效。
+### JobManager
+
+可以通过下列配置参数设置堆外（直接或本地）内存开销：
+* 堆外内存 ([`jobmanager.memory.off-heap.size`](../config.html#jobmanager-memory-off-heap-size))
+* JVM Metaspace ([`jobmanager.memory.jvm-metaspace.size`](../config.html#jobmanager-memory-jvm-metaspace-size))
+* [JVM 开销](mem_setup_jobmanager.html#detailed-configuration)
+
+
## flink-conf.yaml 中的默认配置
本节描述 Flink 自带的默认 `flink-conf.yaml` 文件中的变化。
-在默认 `flink-conf.yaml` 文件中，原本的总内存（`taskmanager.heap.size`）被新的配置项 [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size) 所取代。
-默认值从 1024Mb 增加到了 1568Mb。
-请参考[如何配置总内存](mem_setup.html#配置总内存)。
+原本的 TaskManager 总内存（`taskmanager.heap.size`）被新的配置项 [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size) 所取代。
+默认值从 1024Mb 增加到了 1728Mb。
+
+原本的 JobManager 总内存（`jobmanager.heap.size`）被新的配置项 [`jobmanager.memory.process.size`](../config.html#taskmanager-memory-process-size) 所取代。
+默认值从 1024Mb 增加到了 1600Mb。
+
+请参考[如何配置总内存](mem_setup.html#configure-total-memory)。
||||||||||||||||||||||||||||||
+| t |
-
-This distributed runtime depends on your application being serializable. It also requires that all
-dependencies are available to each node in the cluster.
-
-### Basic stream sources
-
-The example above constructs a `DataStream
-
-此分布式运行时取决于你的应用是否是可序列化的。它还要求所有依赖对集群中的每个节点均可用。
-
-### 基本的 stream source
-
-上述示例用 `env.fromElements(...)` 方法构造 `DataStream| Name | -Working State | -State Backup | -Snapshotting | -
|---|---|---|---|
| RocksDBStateBackend | -Local disk (tmp dir) | -Distributed file system | -Full / Incremental | -
-
|
- |||
| FsStateBackend | -JVM Heap | -Distributed file system | -Full | -
-
|
- |||
| MemoryStateBackend | -JVM Heap | -JobManager JVM Heap | -Full | -
-
|
- |||
-
-Checkpoint _n_ will contain the state of each operator that resulted from having consumed **every
-event before checkpoint barrier _n_, and none of the events after it**.
-
-As each operator in the job graph receives one of these barriers, it records its state. Operators
-with two input streams (such as a `CoProcessFunction`) perform _barrier alignment_ so that the
-snapshot will reflect the state resulting from consuming events from both input streams up to (but
-not past) both barriers.
-
-
-
-Flink's state backends use a copy-on-write mechanism to allow stream processing to continue
-unimpeded while older versions of the state are being asynchronously snapshotted. Only when the
-snapshots have been durably persisted will these older versions of the state be garbage collected.
-
-### Exactly Once Guarantees
-
-When things go wrong in a stream processing application, it is possible to have either lost, or
-duplicated results. With Flink, depending on the choices you make for your application and the
-cluster you run it on, any of these outcomes is possible:
-
-- Flink makes no effort to recover from failures (_at most once_)
-- Nothing is lost, but you may experience duplicated results (_at least once_)
-- Nothing is lost or duplicated (_exactly once_)
-
-Given that Flink recovers from faults by rewinding and replaying the source data streams, when the
-ideal situation is described as **exactly once** this does *not* mean that every event will be
-processed exactly once. Instead, it means that _every event will affect the state being managed by
-Flink exactly once_.
-
-Barrier alignment is only needed for providing exactly once guarantees. If you don't need this, you
-can gain some performance by configuring Flink to use `CheckpointingMode.AT_LEAST_ONCE`, which has
-the effect of disabling barrier alignment.
-
-### Exactly Once End-to-end
-
-To achieve exactly once end-to-end, so that every event from the sources affects the sinks exactly
-once, the following must be true:
-
-1. your sources must be replayable, and
-2. your sinks must be transactional (or idempotent)
-
-{% top %}
-
-## Hands-on
-
-The [Flink Operations Playground]({% link
-getting-started/docker-playgrounds/flink-operations-playground.zh.md %}) includes a section on
-[Observing Failure & Recovery]({% link
-getting-started/docker-playgrounds/flink-operations-playground.zh.md %}#observing-failure--recovery).
-
-{% top %}
-
-## Further Reading
-
-- [Stateful Stream Processing]({% link concepts/stateful-stream-processing.zh.md %})
-- [State Backends]({% link ops/state/state_backends.zh.md %})
-- [Fault Tolerance Guarantees of Data Sources and Sinks]({% link dev/connectors/guarantees.zh.md %})
-- [Enabling and Configuring Checkpointing]({% link dev/stream/state/checkpointing.zh.md %})
-- [Checkpoints]({% link ops/state/checkpoints.zh.md %})
-- [Savepoints]({% link ops/state/savepoints.zh.md %})
-- [Tuning Checkpoints and Large State]({% link ops/state/large_state_tuning.zh.md %})
-- [Monitoring Checkpointing]({% link monitoring/checkpoint_monitoring.zh.md %})
-- [Task Failure Recovery]({% link dev/task_failure_recovery.zh.md %})
-
-{% top %}
diff --git a/docs/training/index.md b/docs/training/index.md
deleted file mode 100644
index 8f00f9079d6f17..00000000000000
--- a/docs/training/index.md
+++ /dev/null
@@ -1,186 +0,0 @@
----
-title: Hands-on Training
-nav-id: training
-nav-pos: 2
-nav-title: ' Hands-on Training'
-nav-parent_id: root
-nav-show_overview: true
-always-expand: true
----
-
-
-* This will be replaced by the TOC
-{:toc}
-
-## Goals and Scope of this Training
-
-This training presents an introduction to Apache Flink that includes just enough to get you started
-writing scalable streaming ETL, analytics, and event-driven applications, while leaving out a lot of
-(ultimately important) details. The focus is on providing straightforward introductions to Flink's
-APIs for managing state and time, with the expectation that having mastered these fundamentals,
-you'll be much better equipped to pick up the rest of what you need to know from the more detailed
-reference documentation. The links at the end of each section will lead you to where you
-can learn more.
-
-Specifically, you will learn:
-
-- how to implement streaming data processing pipelines
-- how and why Flink manages state
-- how to use event time to consistently compute accurate analytics
-- how to build event-driven applications on continuous streams
-- how Flink is able to provide fault-tolerant, stateful stream processing with exactly-once semantics
-
-This training focuses on four critical concepts: continuous processing of streaming data, event
-time, stateful stream processing, and state snapshots. This page introduces these concepts.
-
-{% info Note %} Accompanying this training is a set of hands-on exercises that will
-guide you through learning how to work with the concepts being presented. A link to the relevant
-exercise is provided at the end of each section.
-
-{% top %}
-
-## Stream Processing
-
-Streams are data's natural habitat. Whether it is events from web servers, trades from a stock
-exchange, or sensor readings from a machine on a factory floor, data is created as part of a stream.
-But when you analyze data, you can either organize your processing around _bounded_ or _unbounded_
-streams, and which of these paradigms you choose has profound consequences.
-
-
-
-**Batch processing** is the paradigm at work when you process a bounded data stream. In this mode of
-operation you can choose to ingest the entire dataset before producing any results, which means that
-it is possible, for example, to sort the data, compute global statistics, or produce a final report
-that summarizes all of the input.
-
-**Stream processing**, on the other hand, involves unbounded data streams. Conceptually, at least,
-the input may never end, and so you are forced to continuously process the data as it arrives.
-
-In Flink, applications are composed of **streaming dataflows** that may be transformed by
-user-defined **operators**. These dataflows form directed graphs that start with one or more
-**sources**, and end in one or more **sinks**.
-
-
-
-Often there is a one-to-one correspondence between the transformations in the program and the
-operators in the dataflow. Sometimes, however, one transformation may consist of multiple operators.
-
-An application may consume real-time data from streaming sources such as message queues or
-distributed logs, like Apache Kafka or Kinesis. But flink can also consume bounded, historic data
-from a variety of data sources. Similarly, the streams of results being produced by a Flink
-application can be sent to a wide variety of systems that can be connected as sinks.
-
-
-
-### Parallel Dataflows
-
-Programs in Flink are inherently parallel and distributed. During execution, a
-*stream* has one or more **stream partitions**, and each *operator* has one or
-more **operator subtasks**. The operator subtasks are independent of one
-another, and execute in different threads and possibly on different machines or
-containers.
-
-The number of operator subtasks is the **parallelism** of that particular
-operator.
-Different operators of the same program may have different levels of
-parallelism.
-
-
-
-Streams can transport data between two operators in a *one-to-one* (or
-*forwarding*) pattern, or in a *redistributing* pattern:
-
- - **One-to-one** streams (for example between the *Source* and the *map()*
- operators in the figure above) preserve the partitioning and ordering of
- the elements. That means that subtask[1] of the *map()* operator will see
- the same elements in the same order as they were produced by subtask[1] of
- the *Source* operator.
-
- - **Redistributing** streams (as between *map()* and *keyBy/window* above, as
- well as between *keyBy/window* and *Sink*) change the partitioning of
- streams. Each *operator subtask* sends data to different target subtasks,
- depending on the selected transformation. Examples are *keyBy()* (which
- re-partitions by hashing the key), *broadcast()*, or *rebalance()* (which
- re-partitions randomly). In a *redistributing* exchange the ordering among
- the elements is only preserved within each pair of sending and receiving
- subtasks (for example, subtask[1] of *map()* and subtask[2] of
- *keyBy/window*). So, for example, the redistribution between the keyBy/window and
- the Sink operators shown above introduces non-determinism regarding the
- order in which the aggregated results for different keys arrive at the Sink.
-
-{% top %}
-
-## Timely Stream Processing
-
-For most streaming applications it is very valuable to be able re-process historic data with the
-same code that is used to process live data -- and to produce deterministic, consistent results,
-regardless.
-
-It can also be crucial to pay attention to the order in which events occurred, rather than the order
-in which they are delivered for processing, and to be able to reason about when a set of events is
-(or should be) complete. For example, consider the set of events involved in an e-commerce
-transaction, or financial trade.
-
-These requirements for timely stream processing can be met by using event time timestamps that are
-recorded in the data stream, rather than using the clocks of the machines processing the data.
-
-{% top %}
-
-## Stateful Stream Processing
-
-Flink's operations can be stateful. This means that how one event is handled can depend on the
-accumulated effect of all the events that came before it. State may be used for something simple,
-such as counting events per minute to display on a dashboard, or for something more complex, such as
-computing features for a fraud detection model.
-
-A Flink application is run in parallel on a distributed cluster. The various parallel instances of a
-given operator will execute independently, in separate threads, and in general will be running on
-different machines.
-
-The set of parallel instances of a stateful operator is effectively a sharded key-value store. Each
-parallel instance is responsible for handling events for a specific group of keys, and the state for
-those keys is kept locally.
-
-The diagram below shows a job running with a parallelism of two across the first three operators in
-the job graph, terminating in a sink that has a parallelism of one. The third operator is stateful,
-and you can see that a fully-connected network shuffle is occurring between the second and third
-operators. This is being done to partition the stream by some key, so that all of the events that
-need to be processed together, will be.
-
-
-
-State is always accessed locally, which helps Flink applications achieve high throughput and
-low-latency. You can choose to keep state on the JVM heap, or if it is too large, in efficiently
-organized on-disk data structures.
-
-
-
-{% top %}
-
-## Fault Tolerance via State Snapshots
-
-Flink is able to provide fault-tolerant, exactly-once semantics through a combination of state
-snapshots and stream replay. These snapshots capture the entire state of the distributed pipeline,
-recording offsets into the input queues as well as the state throughout the job graph that has
-resulted from having ingested the data up to that point. When a failure occurs, the sources are
-rewound, the state is restored, and processing is resumed. As depicted above, these state snapshots
-are captured asynchronously, without impeding the ongoing processing.
-
-{% top %}
diff --git a/docs/training/index.zh.md b/docs/training/index.zh.md
deleted file mode 100644
index 8f00f9079d6f17..00000000000000
--- a/docs/training/index.zh.md
+++ /dev/null
@@ -1,186 +0,0 @@
----
-title: Hands-on Training
-nav-id: training
-nav-pos: 2
-nav-title: ' Hands-on Training'
-nav-parent_id: root
-nav-show_overview: true
-always-expand: true
----
-
-
-* This will be replaced by the TOC
-{:toc}
-
-## Goals and Scope of this Training
-
-This training presents an introduction to Apache Flink that includes just enough to get you started
-writing scalable streaming ETL, analytics, and event-driven applications, while leaving out a lot of
-(ultimately important) details. The focus is on providing straightforward introductions to Flink's
-APIs for managing state and time, with the expectation that having mastered these fundamentals,
-you'll be much better equipped to pick up the rest of what you need to know from the more detailed
-reference documentation. The links at the end of each section will lead you to where you
-can learn more.
-
-Specifically, you will learn:
-
-- how to implement streaming data processing pipelines
-- how and why Flink manages state
-- how to use event time to consistently compute accurate analytics
-- how to build event-driven applications on continuous streams
-- how Flink is able to provide fault-tolerant, stateful stream processing with exactly-once semantics
-
-This training focuses on four critical concepts: continuous processing of streaming data, event
-time, stateful stream processing, and state snapshots. This page introduces these concepts.
-
-{% info Note %} Accompanying this training is a set of hands-on exercises that will
-guide you through learning how to work with the concepts being presented. A link to the relevant
-exercise is provided at the end of each section.
-
-{% top %}
-
-## Stream Processing
-
-Streams are data's natural habitat. Whether it is events from web servers, trades from a stock
-exchange, or sensor readings from a machine on a factory floor, data is created as part of a stream.
-But when you analyze data, you can either organize your processing around _bounded_ or _unbounded_
-streams, and which of these paradigms you choose has profound consequences.
-
-
-
-**Batch processing** is the paradigm at work when you process a bounded data stream. In this mode of
-operation you can choose to ingest the entire dataset before producing any results, which means that
-it is possible, for example, to sort the data, compute global statistics, or produce a final report
-that summarizes all of the input.
-
-**Stream processing**, on the other hand, involves unbounded data streams. Conceptually, at least,
-the input may never end, and so you are forced to continuously process the data as it arrives.
-
-In Flink, applications are composed of **streaming dataflows** that may be transformed by
-user-defined **operators**. These dataflows form directed graphs that start with one or more
-**sources**, and end in one or more **sinks**.
-
-
-
-Often there is a one-to-one correspondence between the transformations in the program and the
-operators in the dataflow. Sometimes, however, one transformation may consist of multiple operators.
-
-An application may consume real-time data from streaming sources such as message queues or
-distributed logs, like Apache Kafka or Kinesis. But flink can also consume bounded, historic data
-from a variety of data sources. Similarly, the streams of results being produced by a Flink
-application can be sent to a wide variety of systems that can be connected as sinks.
-
-
-
-### Parallel Dataflows
-
-Programs in Flink are inherently parallel and distributed. During execution, a
-*stream* has one or more **stream partitions**, and each *operator* has one or
-more **operator subtasks**. The operator subtasks are independent of one
-another, and execute in different threads and possibly on different machines or
-containers.
-
-The number of operator subtasks is the **parallelism** of that particular
-operator.
-Different operators of the same program may have different levels of
-parallelism.
-
-
-
-Streams can transport data between two operators in a *one-to-one* (or
-*forwarding*) pattern, or in a *redistributing* pattern:
-
- - **One-to-one** streams (for example between the *Source* and the *map()*
- operators in the figure above) preserve the partitioning and ordering of
- the elements. That means that subtask[1] of the *map()* operator will see
- the same elements in the same order as they were produced by subtask[1] of
- the *Source* operator.
-
- - **Redistributing** streams (as between *map()* and *keyBy/window* above, as
- well as between *keyBy/window* and *Sink*) change the partitioning of
- streams. Each *operator subtask* sends data to different target subtasks,
- depending on the selected transformation. Examples are *keyBy()* (which
- re-partitions by hashing the key), *broadcast()*, or *rebalance()* (which
- re-partitions randomly). In a *redistributing* exchange the ordering among
- the elements is only preserved within each pair of sending and receiving
- subtasks (for example, subtask[1] of *map()* and subtask[2] of
- *keyBy/window*). So, for example, the redistribution between the keyBy/window and
- the Sink operators shown above introduces non-determinism regarding the
- order in which the aggregated results for different keys arrive at the Sink.
-
-{% top %}
-
-## Timely Stream Processing
-
-For most streaming applications it is very valuable to be able re-process historic data with the
-same code that is used to process live data -- and to produce deterministic, consistent results,
-regardless.
-
-It can also be crucial to pay attention to the order in which events occurred, rather than the order
-in which they are delivered for processing, and to be able to reason about when a set of events is
-(or should be) complete. For example, consider the set of events involved in an e-commerce
-transaction, or financial trade.
-
-These requirements for timely stream processing can be met by using event time timestamps that are
-recorded in the data stream, rather than using the clocks of the machines processing the data.
-
-{% top %}
-
-## Stateful Stream Processing
-
-Flink's operations can be stateful. This means that how one event is handled can depend on the
-accumulated effect of all the events that came before it. State may be used for something simple,
-such as counting events per minute to display on a dashboard, or for something more complex, such as
-computing features for a fraud detection model.
-
-A Flink application is run in parallel on a distributed cluster. The various parallel instances of a
-given operator will execute independently, in separate threads, and in general will be running on
-different machines.
-
-The set of parallel instances of a stateful operator is effectively a sharded key-value store. Each
-parallel instance is responsible for handling events for a specific group of keys, and the state for
-those keys is kept locally.
-
-The diagram below shows a job running with a parallelism of two across the first three operators in
-the job graph, terminating in a sink that has a parallelism of one. The third operator is stateful,
-and you can see that a fully-connected network shuffle is occurring between the second and third
-operators. This is being done to partition the stream by some key, so that all of the events that
-need to be processed together, will be.
-
-
-
-State is always accessed locally, which helps Flink applications achieve high throughput and
-low-latency. You can choose to keep state on the JVM heap, or if it is too large, in efficiently
-organized on-disk data structures.
-
-
-
-{% top %}
-
-## Fault Tolerance via State Snapshots
-
-Flink is able to provide fault-tolerant, exactly-once semantics through a combination of state
-snapshots and stream replay. These snapshots capture the entire state of the distributed pipeline,
-recording offsets into the input queues as well as the state throughout the job graph that has
-resulted from having ingested the data up to that point. When a failure occurs, the sources are
-rewound, the state is restored, and processing is resumed. As depicted above, these state snapshots
-are captured asynchronously, without impeding the ongoing processing.
-
-{% top %}
diff --git a/docs/try-flink/datastream_api.md b/docs/try-flink/datastream_api.md
new file mode 100644
index 00000000000000..3d2e32366aae35
--- /dev/null
+++ b/docs/try-flink/datastream_api.md
@@ -0,0 +1,943 @@
+---
+title: "Fraud Detection with the DataStream API"
+nav-title: 'Fraud Detection with the DataStream API'
+nav-parent_id: try-flink
+nav-pos: 2
+---
+
+
+Apache Flink offers a DataStream API for building robust, stateful streaming applications.
+It provides fine-grained control over state and time, which allows for the implementation of advanced event-driven systems.
+In this step-by-step guide you'll learn how to build a stateful streaming application with Flink's DataStream API.
+
+* This will be replaced by the TOC
+{:toc}
+
+## What Are You Building?
+
+Credit card fraud is a growing concern in the digital age.
+Criminals steal credit card numbers by running scams or hacking into insecure systems.
+Stolen numbers are tested by making one or more small purchases, often for a dollar or less.
+If that works, they then make more significant purchases to get items they can sell or keep for themselves.
+
+In this tutorial, you will build a fraud detection system for alerting on suspicious credit card transactions.
+Using a simple set of rules, you will see how Flink allows us to implement advanced business logic and act in real-time.
+
+## Prerequisites
+
+This walkthrough assumes that you have some familiarity with Java or Scala, but you should be able to follow along even if you are coming from a different programming language.
+
+## Help, I’m Stuck!
+
+If you get stuck, check out the [community support resources](https://flink.apache.org/gettinghelp.html).
+In particular, Apache Flink's [user mailing list](https://flink.apache.org/community.html#mailing-lists) is consistently ranked as one of the most active of any Apache project and a great way to get help quickly.
+
+## How to Follow Along
+
+If you want to follow along, you will require a computer with:
+
+* Java 8 or 11
+* Maven
+
+A provided Flink Maven Archetype will create a skeleton project with all the necessary dependencies quickly, so you only need to focus on filling out the business logic.
+These dependencies include `flink-streaming-java` which is the core dependency for all Flink streaming applications and `flink-walkthrough-common` that has data generators and other classes specific to this walkthrough.
+
+{% panel **Note:** Each code block within this walkthrough may not contain the full surrounding class for brevity. The full code is available [at the bottom of the page](#final-application). %}
+
+
+ Note: For Maven 3.0 or higher, it is no longer possible to specify the repository (-DarchetypeCatalog) via the command line. For details about this change, please refer to Maven official document
+ If you wish to use the snapshot repository, you need to add a repository entry to your settings.xml. For example:
+{% highlight bash %}
+
+
+
+ 注意：Maven 3.0 及更高版本，不再支持通过命令行指定仓库（-DarchetypeCatalog）。有关这个改动的详细信息，
+ 请参阅 Maven 官方文档
+ 如果你希望使用快照仓库，则需要在 settings.xml 文件中添加一个仓库条目。例如：
+{% highlight bash %}
+
+
+
+
+ NOTE: The Apache Flink Docker images used for this playground are only available for
+ released versions of Apache Flink.
+
+ Since you are currently looking at the latest SNAPSHOT
+ version of the documentation, all version references below will not work.
+ Please switch the documentation to the latest released version via the release picker which you
+ find on the left side below the menu.
+
+
+When the playground is started a Flink Job called *Flink Event Count* will be submitted to the
+JobManager. Additionally, two Kafka Topics *input* and *output* are created.
+
+
+
+The Job consumes `ClickEvent`s from the *input* topic, each with a `timestamp` and a `page`. The
+events are then keyed by `page` and counted in 15 second
+[windows]({{ site.baseurl }}/dev/stream/operators/windows.html). The results are written to the
+*output* topic.
+
+There are six different pages and we generate 1000 click events per page and 15 seconds. Hence, the
+output of the Flink job should show 1000 views per page and window.
+
+{% top %}
+
+## Starting the Playground
+
+The playground environment is set up in just a few steps. We will walk you through the necessary
+commands and show how to validate that everything is running correctly.
+
+We assume that you have [Docker](https://docs.docker.com/) (1.12+) and
+[docker-compose](https://docs.docker.com/compose/) (2.1+) installed on your machine.
+
+The required configuration files are available in the
+[flink-playgrounds](https://github.com/apache/flink-playgrounds) repository. Check it out and spin
+up the environment:
+
+{% highlight bash %}
+git clone --branch release-{{ site.version_title }} https://github.com/apache/flink-playgrounds.git
+cd flink-playgrounds/operations-playground
+docker-compose build
+docker-compose up -d
+{% endhighlight %}
+
+Afterwards, you can inspect the running Docker containers with the following command:
+
+{% highlight bash %}
+docker-compose ps
+
+ Name Command State Ports
+-----------------------------------------------------------------------------------------------------------------------------
+operations-playground_clickevent-generator_1 /docker-entrypoint.sh java ... Up 6123/tcp, 8081/tcp
+operations-playground_client_1 /docker-entrypoint.sh flin ... Exit 0
+operations-playground_jobmanager_1 /docker-entrypoint.sh jobm ... Up 6123/tcp, 0.0.0.0:8081->8081/tcp
+operations-playground_kafka_1 start-kafka.sh Up 0.0.0.0:9094->9094/tcp
+operations-playground_taskmanager_1 /docker-entrypoint.sh task ... Up 6123/tcp, 8081/tcp
+operations-playground_zookeeper_1 /bin/sh -c /usr/sbin/sshd ... Up 2181/tcp, 22/tcp, 2888/tcp, 3888/tcp
+{% endhighlight %}
+
+This indicates that the client container has successfully submitted the Flink Job (`Exit 0`) and all
+cluster components as well as the data generator are running (`Up`).
+
+You can stop the playground environment by calling:
+
+{% highlight bash %}
+docker-compose down -v
+{% endhighlight %}
+
+## Entering the Playground
+
+There are many things you can try and check out in this playground. In the following two sections we
+will show you how to interact with the Flink Cluster and demonstrate some of Flink's key features.
+
+### Flink WebUI
+
+The most natural starting point to observe your Flink Cluster is the WebUI exposed under
+[http://localhost:8081](http://localhost:8081). If everything went well, you'll see that the cluster initially consists of
+one TaskManager and executes a Job called *Click Event Count*.
+
+
+
+The Flink WebUI contains a lot of useful and interesting information about your Flink Cluster and
+its Jobs (JobGraph, Metrics, Checkpointing Statistics, TaskManager Status,...).
+
+### Logs
+
+**JobManager**
+
+The JobManager logs can be tailed via `docker-compose`.
+
+{% highlight bash %}
+docker-compose logs -f jobmanager
+{% endhighlight %}
+
+After the initial startup you should mainly see log messages for every checkpoint completion.
+
+**TaskManager**
+
+The TaskManager log can be tailed in the same way.
+{% highlight bash %}
+docker-compose logs -f taskmanager
+{% endhighlight %}
+
+After the initial startup you should mainly see log messages for every checkpoint completion.
+
+### Flink CLI
+
+The [Flink CLI]({{ site.baseurl }}/ops/cli.html) can be used from within the client container. For
+example, to print the `help` message of the Flink CLI you can run
+{% highlight bash%}
+docker-compose run --no-deps client flink --help
+{% endhighlight %}
+
+### Flink REST API
+
+The [Flink REST API]({{ site.baseurl }}/monitoring/rest_api.html#api) is exposed via
+`localhost:8081` on the host or via `jobmanager:8081` from the client container, e.g. to list all
+currently running jobs, you can run:
+{% highlight bash%}
+curl localhost:8081/jobs
+{% endhighlight %}
+
+{% if site.version contains "SNAPSHOT" %}
++ Note: If the curl command is not available on your machine, you can run it from the client + container (similar to the Flink CLI): +{% highlight bash%} +docker-compose run --no-deps client curl jobmanager:8081/jobs +{% endhighlight %} +
+{% endif %} + +### Kafka Topics + +You can look at the records that are written to the Kafka Topics by running +{% highlight bash%} +//input topic (1000 records/s) +docker-compose exec kafka kafka-console-consumer.sh \ + --bootstrap-server localhost:9092 --topic input + +//output topic (24 records/min) +docker-compose exec kafka kafka-console-consumer.sh \ + --bootstrap-server localhost:9092 --topic output +{% endhighlight %} + +{% top %} + +## Time to Play! + +Now that you learned how to interact with Flink and the Docker containers, let's have a look at +some common operational tasks that you can try out on our playground. +All of these tasks are independent of each other, i.e. you can perform them in any order. +Most tasks can be executed via the [CLI](#flink-cli) and the [REST API](#flink-rest-api). + +### Listing Running Jobs + +
+
++ Note: Even though the tasks of the job are in SCHEDULED state and not RUNNING yet, the overall + status of a Job is shown as RUNNING. +
+ +At this point, the tasks of the Job cannot move from the `SCHEDULED` state to `RUNNING` because there +are no resources (TaskSlots provided by TaskManagers) to the run the tasks. +Until a new TaskManager becomes available, the Job will go through a cycle of cancellations and resubmissions. + +In the meantime, the data generator keeps pushing `ClickEvent`s into the *input* topic. This is +similar to a real production setup where data is produced while the Job to process it is down. + +#### Step 3: Recovery + +Once you restart the TaskManager, it reconnects to the JobManager. + +{% highlight bash%} +docker-compose up -d taskmanager +{% endhighlight %} + +When the JobManager is notified about the new TaskManager, it schedules the tasks of the +recovering Job to the newly available TaskSlots. Upon restart, the tasks recover their state from +the last successful [checkpoint]({{ site.baseurl }}/learn-flink/fault_tolerance.html) that was taken +before the failure and switch to the `RUNNING` state. + +The Job will quickly process the full backlog of input events (accumulated during the outage) +from Kafka and produce output at a much higher rate (> 24 records/minute) until it reaches +the head of the stream. In the *output* you will see that all keys (`page`s) are present for all time +windows and that every count is exactly one thousand. Since we are using the +[FlinkKafkaProducer]({{ site.baseurl }}/dev/connectors/kafka.html#kafka-producers-and-fault-tolerance) +in its "at-least-once" mode, there is a chance that you will see some duplicate output records. + ++ Note: Most production setups rely on a resource manager (Kubernetes, Yarn, Mesos) to + automatically restart failed processes. +
+ +### Upgrading & Rescaling a Job + +Upgrading a Flink Job always involves two steps: First, the Flink Job is gracefully stopped with a +[Savepoint]({{ site.baseurl }}/ops/state/savepoints.html). A Savepoint is a consistent snapshot of +the complete application state at a well-defined, globally consistent point in time (similar to a +checkpoint). Second, the upgraded Flink Job is started from the Savepoint. In this context "upgrade" +can mean different things including the following: + +* An upgrade to the configuration (incl. the parallelism of the Job) +* An upgrade to the topology of the Job (added/removed Operators) +* An upgrade to the user-defined functions of the Job + +Before starting with the upgrade you might want to start tailing the *output* topic, in order to +observe that no data is lost or corrupted in the course the upgrade. + +{% highlight bash%} +docker-compose exec kafka kafka-console-consumer.sh \ + --bootstrap-server localhost:9092 --topic output +{% endhighlight %} + +#### Step 1: Stopping the Job + +To gracefully stop the Job, you need to use the "stop" command of either the CLI or the REST API. +For this you will need the JobID of the Job, which you can obtain by +[listing all running Jobs](#listing-running-jobs) or from the WebUI. With the JobID you can proceed +to stopping the Job: + +
+
+ NOTE: The Apache Flink community only publishes official builds for
+ released versions of Apache Flink.
+
+ Since you are currently looking at the latest SNAPSHOT
+ version of the documentation, all version references below will not work.
+ Please switch the documentation to the latest released version via the release picker which you
+ find on the left side below the menu.
+
+
+ NOTE: The Apache Flink community only publishes official builds for
+ released versions of Apache Flink.
+
+ Since you are currently looking at the latest SNAPSHOT
+ version of the documentation, all version references below will not work.
+ Please switch the documentation to the latest released version via the release picker which you
+ find on the left side below the menu.
+
+
+ NOTE: The Apache Flink Docker images used for this playground are only available for
+ released versions of Apache Flink.
+
+ Since you are currently looking at the latest SNAPSHOT
+ version of the documentation, all version references below will not work.
+ Please switch the documentation to the latest released version via the release picker which you
+ find on the left side below the menu.
+
+
+ NOTE: The Apache Flink Docker images used for this playground are only available for
+ released versions of Apache Flink.
+
+ Since you are currently looking at the latest SNAPSHOT
+ version of the documentation, all version references below will not work.
+ Please switch the documentation to the latest released version via the release picker which you
+ find on the left side below the menu.
+
2mb.
*/
public Builder setBufferFlushMaxSizeInBytes(long bufferFlushMaxSizeInBytes) {
- checkArgument(
- bufferFlushMaxSizeInBytes > 0,
- "Max byte size of buffered rows must be larger than 0.");
this.bufferFlushMaxSizeInBytes = bufferFlushMaxSizeInBytes;
return this;
}
@@ -122,9 +116,6 @@ public Builder setBufferFlushMaxSizeInBytes(long bufferFlushMaxSizeInBytes) {
* Defaults to not set, i.e. won't flush based on the number of buffered rows.
*/
public Builder setBufferFlushMaxRows(long bufferFlushMaxRows) {
- checkArgument(
- bufferFlushMaxRows > 0,
- "Max number of buffered rows must be larger than 0.");
this.bufferFlushMaxRows = bufferFlushMaxRows;
return this;
}
@@ -134,9 +125,6 @@ public Builder setBufferFlushMaxRows(long bufferFlushMaxRows) {
* Defaults to not set, i.e. won't flush based on flush interval.
*/
public Builder setBufferFlushIntervalMillis(long bufferFlushIntervalMillis) {
- checkArgument(
- bufferFlushIntervalMillis > 0,
- "Interval (in milliseconds) between each flush must be larger than 0.");
this.bufferFlushIntervalMillis = bufferFlushIntervalMillis;
return this;
}
diff --git a/flink-connectors/flink-connector-hbase/src/main/java/org/apache/flink/connector/hbase/sink/HBaseSinkFunction.java b/flink-connectors/flink-connector-hbase/src/main/java/org/apache/flink/connector/hbase/sink/HBaseSinkFunction.java
index 3d96e5d0916310..3ba533ae47909b 100644
--- a/flink-connectors/flink-connector-hbase/src/main/java/org/apache/flink/connector/hbase/sink/HBaseSinkFunction.java
+++ b/flink-connectors/flink-connector-hbase/src/main/java/org/apache/flink/connector/hbase/sink/HBaseSinkFunction.java
@@ -118,8 +118,10 @@ public void open(Configuration parameters) throws Exception {
}
// create a parameter instance, set the table name and custom listener reference.
BufferedMutatorParams params = new BufferedMutatorParams(TableName.valueOf(hTableName))
- .listener(this)
- .writeBufferSize(bufferFlushMaxSizeInBytes);
+ .listener(this);
+ if (bufferFlushMaxSizeInBytes > 0) {
+ params.writeBufferSize(bufferFlushMaxSizeInBytes);
+ }
this.mutator = connection.getBufferedMutator(params);
if (bufferFlushIntervalMillis > 0) {
diff --git a/flink-connectors/flink-connector-hbase/src/main/java/org/apache/flink/connector/hbase/source/AbstractTableInputFormat.java b/flink-connectors/flink-connector-hbase/src/main/java/org/apache/flink/connector/hbase/source/AbstractTableInputFormat.java
index 684afd2220240e..94d36d3c06bcca 100644
--- a/flink-connectors/flink-connector-hbase/src/main/java/org/apache/flink/connector/hbase/source/AbstractTableInputFormat.java
+++ b/flink-connectors/flink-connector-hbase/src/main/java/org/apache/flink/connector/hbase/source/AbstractTableInputFormat.java
@@ -24,8 +24,10 @@
import org.apache.flink.api.common.io.RichInputFormat;
import org.apache.flink.api.common.io.statistics.BaseStatistics;
import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.hbase.util.HBaseConfigurationUtil;
import org.apache.flink.core.io.InputSplitAssigner;
+import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.ResultScanner;
@@ -60,6 +62,13 @@ abstract class AbstractTableInputFormat