Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
81f86ae
Update for 2.2.0
Nov 10, 2025
b408850
[FLINK-38640][table-planner] Fix NPE in DeltaJoinUtil#isFilterOnOneSe…
xuyangzhong Nov 10, 2025
6e89776
[hotfix] [docs] Ververica URL and Product Name has been updated
nacisimsek Nov 12, 2025
f9ad1b1
[hotfix][examples] Add Python examples on how to read binary data fro…
dianfu Nov 12, 2025
1bca9b9
[FLINK-38436][doc] Add vector search doc (#27216)
fsk119 Nov 11, 2025
d421622
[FLINK-38436][doc] Add Chinese version of vector search doc (#27232)
lihaosky Nov 13, 2025
7e2e854
[FLINK-38622][runtime] Enhance the requests and slots balanced alloca…
RocMarshal Nov 13, 2025
71fb419
[FLINK-38611][doc] Add doc for delta join (#27225)
xuyangzhong Nov 14, 2025
30b5b97
[FLINK-33392][docs] Add the documentation pages for balanced tasks sc…
RocMarshal Nov 14, 2025
ca9f7b7
[hotfix][docs] Add `SHOW MATERIALIZED TABLES`, `SHOW CREATE MATERIALI…
snuyanzin Nov 14, 2025
f621ab6
[FLINK-38682][table-planner] Support unknown -> RAW cast during type …
ferenc-csaky Nov 17, 2025
434c800
[FLINK-38686][doc] Add model table api documentation (#27243) (#27246)
lihaosky Nov 18, 2025
ef31ad4
[FLINK-38695][table-planner] Fix wrong metric about left cache reques…
xuyangzhong Nov 20, 2025
4fd5bea
[FLINK-36746][core] Fix the deadlock bug in SerializedThrowable
RocMarshal Nov 3, 2025
e666847
[FLINK-38709][table][python] Fix ScalarFunctionSplitter to allow Pyth…
dianfu Nov 20, 2025
bf9cc5d
[FLINK-38700][python] Fix kubernetes pyflink application test
snuyanzin Nov 25, 2025
2910d72
[BP-2.2][FLINK-38711][build] Set maven-shade-plugin logs to DEBUG
mateczagany Nov 25, 2025
fefb741
[hotfix] Fix flink-model-openai NOTICE for jtokkit
mateczagany Nov 26, 2025
e4c197d
[FLINK-38576][table] Align commonJoinKey in MultiJoin for logical and…
gustavodemorais Nov 26, 2025
ad47717
[FLINK-38750][table] Validation of queries with functions erroneously…
snuyanzin Dec 2, 2025
3f79fac
[FLINK-38084][doc] Add download doc for model providers (#27296)
lihaosky Dec 3, 2025
3546641
[FLINK-38767][table] Fix vector search execnode transform name (#27304)
lihaosky Dec 4, 2025
8de272c
[FLINK-38773][table] Fix batch vector search excnode context (#27311)
lihaosky Dec 4, 2025
eb65369
[FLINK-38797][python] Fixed CsvSchemaBuilder.set_null_value to return…
wchan87 Dec 10, 2025
932aa43
[FLINK-38824][table] Fix incorrect default values for primitive types
dylanhz Dec 25, 2025
0f27e47
[FLINK-38703][runtime] Update slot manager metrics in thread-safety m…
ztison Nov 20, 2025
8647457
[FLINK-38914][docs] Preserve page path when linking to stable version
MartijnVisser Dec 29, 2025
28bda0b
[FLINK-38914][docs] Add canonical tags pointing to stable docs
MartijnVisser Jan 14, 2026
0a83669
[FLINK-38925][docs] Update Matomo URL to the right domain
MartijnVisser Jan 15, 2026
00c6e94
[FLINK-38924][docs] Redirect users to documentation home page when en…
MartijnVisser Jan 15, 2026
0280f6b
Update datagen.md
JavaLeb Jan 20, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/docs.sh
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ mvn javadoc:aggregate -B \
-Dcheckstyle.skip=true \
-Dspotless.check.skip=true \
-Denforcer.skip=true \
-Dheader="<a href=\"http://flink.apache.org/\" target=\"_top\"><h1>Back to Flink Website</h1></a> <script>var _paq=window._paq=window._paq||[];_paq.push([\"disableCookies\"]),_paq.push([\"setDomains\",[\"*.flink.apache.org\",\"*.nightlies.apache.org/flink\"]]),_paq.push([\"trackPageView\"]),_paq.push([\"enableLinkTracking\"]),function(){var u=\"//matomo.privacy.apache.org/\";_paq.push([\"setTrackerUrl\",u+\"matomo.php\"]),_paq.push([\"setSiteId\",\"1\"]);var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0];g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s)}();</script>"
-Dheader="<a href=\"http://flink.apache.org/\" target=\"_top\"><h1>Back to Flink Website</h1></a> <script>var _paq=window._paq=window._paq||[];_paq.push([\"disableCookies\"]),_paq.push([\"setDomains\",[\"*.flink.apache.org\",\"*.nightlies.apache.org/flink\"]]),_paq.push([\"trackPageView\"]),_paq.push([\"enableLinkTracking\"]),function(){var u=\"//analytics.apache.org/\";_paq.push([\"setTrackerUrl\",u+\"matomo.php\"]),_paq.push([\"setSiteId\",\"1\"]);var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0];g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s)}();</script>"
mv target/site/apidocs docs/target/api/java

# build python docs
Expand Down
19 changes: 10 additions & 9 deletions docs/config.toml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.

baseURL = '//nightlies.apache.org/flink/flink-docs-master'
baseURL = '//nightlies.apache.org/flink/flink-docs-release-2.2'
languageCode = "en-us"
title = "Apache Flink"
enableGitInfo = false
Expand All @@ -24,7 +24,7 @@ pygmentsUseClasses = true
[params]
# Flag whether this is a stable version or not.
# Used for the quickstart page.
IsStable = false
IsStable = true

# Flag to indicate whether an outdated warning should be shown.
ShowOutDatedWarning = false
Expand All @@ -34,14 +34,14 @@ pygmentsUseClasses = true
# we change the version for the complete docs when forking of a release branch
# etc.
# The full version string as referenced in Maven (e.g. 1.2.1)
Version = "2.2-SNAPSHOT"
Version = "2.2.0"

# For stable releases, leave the bugfix version out (e.g. 1.2). For snapshot
# release this should be the same as the regular version
VersionTitle = "2.2-SNAPSHOT"
VersionTitle = "2.2"

# The branch for this version of Apache Flink
Branch = "master"
Branch = "release-2.2"

# The github repository for Apache Flink
Repo = "//github.com/apache/flink"
Expand All @@ -60,19 +60,20 @@ pygmentsUseClasses = true

ZhDownloadPage = "//flink.apache.org/zh/downloads.html"

JavaDocs = "//nightlies.apache.org/flink/flink-docs-master/api/java/"
JavaDocs = "//nightlies.apache.org/flink/flink-docs-release-2.2/api/java/"

PyDocs = "//nightlies.apache.org/flink/flink-docs-master/api/python/"
PyDocs = "//nightlies.apache.org/flink/flink-docs-release-2.2/api/python/"

# External links at the bottom
# of the menu
MenuLinks = [
["Project Homepage", "//flink.apache.org"],
["JavaDocs", "//nightlies.apache.org/flink/flink-docs-master/api/java/"],
["PyDocs", "//nightlies.apache.org/flink/flink-docs-master/api/python/"]
["JavaDocs", "//nightlies.apache.org/flink/flink-docs-release-2.2/api/java/"],
["PyDocs", "//nightlies.apache.org/flink/flink-docs-release-2.2/api/python/"]
]

PreviousDocs = [
["2.2", "http://nightlies.apache.org/flink/flink-docs-release-2.2"],
["2.1", "http://nightlies.apache.org/flink/flink-docs-release-2.1"],
["2.0", "http://nightlies.apache.org/flink/flink-docs-release-2.0"],
["1.20", "http://nightlies.apache.org/flink/flink-docs-release-1.20"],
Expand Down
41 changes: 41 additions & 0 deletions docs/content.zh/docs/connectors/models/downloads.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
---
title: 下载页面
weight: 100
type: docs
bookToc: false
aliases:
- /zh/dev/table/connectors/downloads.html
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

# SQL Models 下载页面

{{< unstable >}}
{{< hint info >}}
Download links are available only for stable releases.
{{< /hint >}}
{{< /unstable >}}

The page contains links to optional sql-client models that are not part of the binary distribution.

# 可选的 SQL Models
-------------------

{{< sql_optional_models >}}
2 changes: 1 addition & 1 deletion docs/content.zh/docs/connectors/table/datagen.md
Original file line number Diff line number Diff line change
Expand Up @@ -353,7 +353,7 @@ CREATE TABLE Orders (
<td><h5>fields.#.null-rate</h5></td>
<td>optional</td>
<td style="word-wrap: break-word;">0</td>
<td>(Type of field)</td>
<td>Float</td>
<td>空值比例。</td>
</tr>
</tbody>
Expand Down
4 changes: 2 additions & 2 deletions docs/content.zh/docs/deployment/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -302,9 +302,9 @@ Supported Environments:
Supported Environment:
{{< label Huawei Cloud >}}

#### Ververica Platform
#### Ververica's Unified Streaming Data Platform (Managed Service / BYOC / Self-Managed)

[Website](https://www.ververica.com/platform-overview)
[Website](https://www.ververica.com/product)

Supported Environments:
{{< label AliCloud >}}
Expand Down
23 changes: 23 additions & 0 deletions docs/content.zh/docs/deployment/tasks-scheduling/_index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
---
title: Tasks Scheduling
bookCollapseSection: true
weight: 9
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
---
title: Balanced Tasks Scheduling
weight: 5
type: docs

---
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

# Balanced Tasks Scheduling

This page describes the background and principle of balanced tasks scheduling,
how to use it when running streaming jobs.

## Background

When the parallelism of all vertices within a Flink streaming job is inconsistent,
the [default strategy]({{< ref "docs/deployment/config" >}}#taskmanager-load-balance-mode)
of Flink to deploy tasks sometimes leads some `TaskManagers` have more tasks while others have fewer tasks,
resulting in excessive resource utilization at some `TaskManagers`
that contain more tasks and becoming a bottleneck for the entire job processing.

{{< img src="/fig/deployments/tasks-scheduling/tasks_scheduling_skew_case.svg" alt="The Skew Case of Tasks Scheduling" class="offset" width="50%" >}}

As shown in figure (a), given a Flink job comprising two vertices, `JobVertex-A (JV-A)` and `JobVertex-B (JV-B)`,
with parallelism degrees of `6` and `3` respectively,
and both vertices sharing the same slot sharing group.
Under the default tasks scheduling strategy, as illustrated in figure (b),
the distribution of tasks across `TaskManagers` may result in significant disparities in task load.
Specifically, the `TaskManager`s with the highest number of tasks may host `4` tasks,
while the one with the lowest load may have only `2` tasks.
Consequently, the `TaskManager`s bearing 4 tasks is prone to become a performance bottleneck for the entire job.

Therefore, Flink provides the task-quantity-based balanced tasks scheduling capability.
Within the job's resource view, it aims to ensure that the number of tasks
scheduled to each `TaskManager` as close as possible to,
thereby improving the resource usage skew among `TaskManagers`.

<span class="label label-info">Note</span> The presence of inconsistent parallelism does not imply that this strategy must be used, as this is not always the case in practice.

## Principle

The task-quantity-based load balancing tasks scheduling strategy completes the assignment of tasks to `TaskManagers` in two phases:
- The tasks-to-slots assignment phase
- The slots-to-TaskManagers assignment phase

This section will use two examples to illustrate the simplified process and principle of
how the task-quantity-based tasks scheduling strategy handles the assignments in these two phases.

### The tasks-to-slots assignment phase

Taking the job shown in figure (c) as an example, it contains five job vertices with parallelism degrees of `1`, `4`, `4`, `2`, and `3`, respectively.
All five job vertices belong to the default slot sharing group.

{{< img src="/fig/deployments/tasks-scheduling/tasks_to_slots_allocation_principle.svg" alt="The Tasks To Slots Allocation Principle Demo" class="offset" width="65%" >}}

During the tasks-to-slots assignment phase, this tasks scheduling strategy:
- First directly assigns the tasks of the vertices with the highest parallelism to the `i-th` slot.

That is, task `JV-Bi` is assigned directly to `sloti`, and task `JV-Ci` is assigned directly to `sloti`.

- Next, for tasks belonging to job vertices with sub-maximal parallelism, they are assigned in a round-robin fashion across the slots within the current
slot sharing group until all tasks are allocated.

As shown in figure (e), under the task-quantity-based assignment strategy, the range (max-min difference) of the number of tasks per slot is `1`,
which is better than the range of `3` under the default strategy shown in figure (d).

Thus, this ensures a more balanced distribution of the number of tasks across slots.

### The slots-to-TaskManagers assignment phase

As shown in figure (f), given a Flink job comprising two vertices, `JV-A` and `JV-B`, with parallelism of `6` and `3` respectively,
and both vertices sharing the same slot sharing group.

{{< img src="/fig/deployments/tasks-scheduling/slots_to_taskmanagers_allocation_principle.svg" alt="The Slots to TaskManagers Allocation Principle Demo" class="offset" width="75%" >}}

The assignment result after the first phase is shown in figure (g),
where `Slot0`, `Slot1`, and `Slot2` each contain `2` tasks, while the remaining slots contain `1` task each.

Subsequently:
- The strategy submits all slot requests and waits until all slot resources required for the current job are ready.

Once the slot resources are ready:
- The strategy then sorts all slot requests in descending order based on the number of tasks contained in each request.
Afterward, it sequentially assigns each slot request to the `TaskManager` with the smallest current tasks loading.
This process continues until all slot requests have been allocated.

The final assignment result is shown in figure (i), where each `TaskManager` ends up with exactly `3` tasks,
resulting in a task count difference of `0` between `TaskManagers`. In contrast, the scheduling result under the default strategy,
shown in figure (h), has a task count difference of `2` between `TaskManagers`.

Therefore, if you are seeing performance bottlenecks of the sort described above,
then using this load balancing tasks scheduling strategy can improve performance.
Be aware that you should not use this strategy, if you are not seeing these bottlenecks,
as you may experience performance degradation.

## Usage

You can enable balanced tasks scheduling through the following configuration item:

- `taskmanager.load-balance.mode`: `tasks`

## More details

See the <a href="https://cwiki.apache.org/confluence/x/U56zDw">FLIP-370</a> for more details.

{{< top >}}
10 changes: 10 additions & 0 deletions docs/content.zh/docs/dev/table/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,30 +116,40 @@ Flink SQL> SET 'table.exec.mini-batch.size' = '5000';
{{< /tab >}}
{{< /tabs >}}

<a name="execution-options" />

### 执行配置

以下选项可用于优化查询执行的性能。

{{< generated/execution_config_configuration >}}

<a name="optimizer-options" />

### 优化器配置

以下配置可以用于调整查询优化器的行为以获得更好的执行计划。

{{< generated/optimizer_config_configuration >}}

<a name="table-options" />

### Planner 配置

以下配置可以用于调整 planner 的行为。

{{< generated/table_config_configuration >}}

<a name="materialized-table-options" />

### Materialized Table 配置

以下配置可以用于调整 Materialized Table 的行为。

{{< generated/materialized_table_config_configuration >}}

<a name="sql-client-options" />

### SQL Client 配置

以下配置可以用于调整 sql client 的行为。
Expand Down
17 changes: 16 additions & 1 deletion docs/content.zh/docs/dev/table/sourcesSinks.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,8 +149,9 @@ Flink 会对工厂类逐个进行检查,确保其“标识符”是全局唯
在读取动态表时,表中数据可以是以下情况之一:
- changelog 流(支持有界或无界),在 changelog 流结束前,所有的改变都会被源源不断地消费,由 `ScanTableSource` 接口表示。
- 处于一直变换或数据量很大的外部表,其中的数据一般不会被全量读取,除非是在查询某个值时,由 `LookupTableSource` 接口表示。
- 外部表支持向量搜索,由 `VectorSearchTableSource` 接口表示。

一个类可以同时实现这两个接口,Planner 会根据查询的 Query 选择相应接口中的方法。
一个类可以同时实现这三个接口,Planner 会根据查询的 Query 选择相应接口中的方法。

<a name= "scan-table-source"></a>

Expand Down Expand Up @@ -188,6 +189,20 @@ Flink 会对工厂类逐个进行检查,确保其“标识符”是全局唯

`LookupTableSource` 的实现方法可以是 `TableFunction` 或者 `AsyncTableFunction`,Flink运行时会根据要查询的 key 值,调用这个实现方法进行查询。

#### Vector Search Table Source

在运行期间, `VectorSearchTableSource` 会使用一个输入向量来搜索外部存储系统,并返回最相似的 Top-K 行。用户可以决定使用何种算法来计算输入数据与外部系统中存储的数据之间的相似度。总的来说,大多数向量数据库支持使用欧几里得距离(Euclidean distance)或余弦距离(Cosine distance)来计算相似度。

与 `ScanTableSource` 相比,该源无需读取整个表,并可以在需要时从一个(可能在持续变化的)外部表中惰性获取(lazily fetch)单个值。

与 `ScanTableSource` 相比,`VectorSearchTableSource` 目前仅支持 insert-only 数据流。

与 `LookupTableSource` 相比,`VectorSearchTableSource` 不会使用等值(equality)来判断行是否匹配。

目前不支持其他更进一步的功能。更多信息请参阅 `org.apache.flink.table.connector.source.VectorSearchTableSource` 的文档。

`VectorSearchTableSource` 的运行时实现是一个 `TableFunction` 或 `AsyncTableFunction`。在运行时,算子会根据给定的向量值调用该函数。

<a name="source-abilities"></a>

#### source 端的功能接口
Expand Down
Loading