[DOC] Add docs of Runtime Filter #6154
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hf200012
reviewed
Jul 5, 2021
Comment on lines
190
to
267
| +-------------------------------------------------------------------+ | ||
| -- 上面`runtime filters`的行显示了`PLAN FRAGMENT 1`的`2:HASH JOIN`生成了ID为RF000的IN predicate， | ||
| -- 其中`test2`.`t2`的key values仅在运行时可知， | ||
| -- 在`0:OlapScanNode`使用了该IN predicate用于在读取`test`.`t1`时过滤不必要的数据。 | ||
|
|
||
| SELECT t1 FROM test JOIN test2 where test.t1 = test2.t2; | ||
| -- 返回2行结果[3, 4]; | ||
|
|
||
| -- 通过query的profile（set is_report_success=true;）可以查看查询内部工作的详细信息， | ||
| -- 包括每个Runtime Filter是否下推、等待耗时、以及OLAP_SCAN_NODE从prepare到接收到Runtime Filter的总时长。 | ||
| RuntimeFilter:in: | ||
| - HasPushDownToEngine: true | ||
| - AWaitTimeCost: 0ns | ||
| - EffectTimeCost: 2.76ms | ||
|
|
||
| -- 此外，在profile的OLAP_SCAN_NODE中还可以查看Runtime Filter下推后的过滤效果和耗时。 | ||
| - RowsVectorPredFiltered: 9.320008M (9320008) | ||
| - VectorPredEvalTime: 364.39ms | ||
| ``` | ||
|
|
xinyiZzz
force-pushed
the
runtime_filter_docs
branch
from
cab6d8f
to
baceee7
Compare
morningman
added
area/runtime-filter
kind/docs
morningman
reviewed
Jul 9, 2021
|
|
||
| ## 使用方式 | ||
|
|
||
| ### 设置Session变量 |
There was a problem hiding this comment.
建议在这里先把所有runtime filter相关参数罗列一些，然后说明，在绝大多数情况下，只需要修改 runtime_filter_type，其他参数保持默认即可。在某些特定场景下，才需要进一步的调整其他参数已达到最优效果
pengxiangyu
reviewed
Jul 9, 2021
| Runtime Filter is generated during query planning, constructed in HashJoinNode, and applied in ScanNode. | ||
|
|
||
| For example, there is currently a Join query between the T1 table and the T2 table. Its Join mode is HashJoin. T1 is a fact table with 100,000 rows of data. T2 is a dimension table with 100 rows of data. Doris join The actual situation is: | ||
| ``` |
There was a problem hiding this comment.
这里写100 rows，下面图里为什么是2000?
pengxiangyu
reviewed
Jul 9, 2021
| ## 原理 | ||
| Runtime Filter在查询规划时生成，在HashJoinNode中构建，在ScanNode中应用。 | ||
|
|
||
| 举个例子，当前存在T1表与T2表的Join查询，它的Join方式为HashJoin，T1是一张事实表，数据行数为100000，T2是一张维度表，数据行数为100，Doris join的实际情况是: |
pengxiangyu
reviewed
Jul 9, 2021
| set runtime_filter_type="BLOOM_FILTER,IN,MIN_MAX"; | ||
| ``` | ||
| 默认会保守的只使用IN predicate，部分情况下同时使用Bloom Filter、MinMax Filter、IN predicate时性能更高。 | ||
|
|
There was a problem hiding this comment.
分别解释一下Bloom Filter MinMax Filter IN predicate的具体算法吧。
There was a problem hiding this comment.
Bloom Filter如何过滤的，没有说明白。
There was a problem hiding this comment.
Bloom Filter的概念和原理感觉没必要在这里解释了，解释起来繁琐，这应该算基础知识…
pengxiangyu
reviewed
Jul 9, 2021
| - **MinMax Filter**: 包含最大值和最小值，从而过滤小于最小值和大于最大值的数据，MinMax Filter的过滤效果与join on clause中Key列的类型和左右表数据分布有关。 | ||
| - 当join on clause中Key列的类型为int/bigint/double等时，极端情况下，如果左右表的最大最小值相同则没有效果，反之右表最大值小于左表最小值，或右表最小值大于左表最大值，则效果最好。 | ||
| - 当join on clause中Key列的类型为varchar等时，应用MinMax Filter往往会导致性能降低。 | ||
| - **IN predicate**: 根据join on clause中Key列在右表上的所有值构建IN predicate，使用构建的IN predicate在左表上过滤，相比loom Filter构建和应用的开销更低，在右表数据量较少时往往性能更高。 |
pengxiangyu
reviewed
Jul 9, 2021
| 在不同Fragment上构建和应用Runtime Filter时，需要合并Runtime Filter的原因和策略可参阅 [ISSUE 6116](https://github.com/apache/incubator-doris/issues/6116) | ||
|
|
||
| #### 3.runtime_filter_wait_time_ms | ||
| 在开启Runtime Filter后，join左表的ScanNode会为每一个分配给自己的Runtime Filter等待一段时间再扫描数据，默认等待1s（1000ms，以毫秒为单位），即如果ScanNode被分配了3个Runtime Filter，那么它最多会等待3s。 |
There was a problem hiding this comment.
不是，目前是在一个循环里面，一个一个等待的
xinyiZzz
force-pushed
the
runtime_filter_docs
branch
2 times, most recently
from
e3e6a6d
to
a893a64
Compare
pengxiangyu
reviewed
Jul 9, 2021
| 如果Runtime Filter在ScanNode开始扫描之后到达，则ScanNode不会将该Runtime Filter下推到存储引擎，而是对已经从存储引擎扫描上来的数据，在ScanNode上基于该Runtime Filter使用表达式过滤，之前已经扫描的数据则不会应用该Runtime Filter，这样得到的中间数据规模会大于最优解，但可以避免严重的裂化。 | ||
|
|
||
| 如果集群比较繁忙，并且集群上有许多资源密集型或长耗时的查询，可以考虑增加等待时间，以避免复杂查询错过优化机会。如果集群负载较轻，并且集群上有许多只需要几秒的小查询，可以考虑减少等待时间，以避免每个查询增加1s的延迟。 | ||
|
|
There was a problem hiding this comment.
错过优化机会是什么意思？如何错过的？如何才能避免错过？
There was a problem hiding this comment.
错过：如果没有在规定时间内等到Runtime filter到达，会不使用Runtime FIlter直接开始扫描，即没用上runtime filter
避免：可以考虑增加等待时间
xinyiZzz
force-pushed
the
runtime_filter_docs
branch
from
a893a64
to
c50a1a0
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
Add docs of Runtime Filter #6116
Types of changes
What types of changes does your code introduce to Doris?
Put an
xin the boxes that applyChecklist
Put an
xin the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.