Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

change LIMIT,OFFSET's data type from int64 to uint64 #16697

Merged
merged 13 commits into from
Jun 7, 2024

Conversation

daviszhen
Copy link
Contributor

@daviszhen daviszhen commented Jun 5, 2024

User description

What type of PR is this?

  • API-change
  • BUG
  • Improvement
  • Documentation
  • Feature
  • Test and CI
  • Code Refactoring

Which issue(s) this PR fixes:

issue ##16624

What this PR does / why we need it:

limit和offset 的数据类型改为uint64。


PR Type

Bug fix, Tests


Description

  • Changed the data type of limit and offset from int64 to uint64 across multiple files.
  • Updated related functions and logic to handle uint64 limits and offsets.
  • Added test cases and results for uint64 limits and offsets.
  • Implemented GetSnapshot function in compiler_context.go.

Changes walkthrough 📝

Relevant files
Bug fix
13 files
limit.go
Change `limit` data type from `int64` to `uint64`               

pkg/sql/colexec/limit/limit.go

  • Changed the data type of limit from int64 to uint64.
+1/-1     
limit.go
Change `limit` data type from `int64` to `uint64`               

pkg/sql/colexec/mergelimit/limit.go

  • Changed the data type of limit from int64 to uint64.
+1/-1     
offset.go
Change `offset` data type from `int64` to `uint64`             

pkg/sql/colexec/mergeoffset/offset.go

  • Changed the data type of offset from int64 to uint64.
+1/-1     
top.go
Change `limit` data type and related functions to `uint64`

pkg/sql/colexec/mergetop/top.go

  • Changed the data type of limit from int64 to uint64.
  • Updated related functions to handle uint64 limits.
  • +7/-7     
    types.go
    Change `limit` data type from `int64` to `uint64`               

    pkg/sql/colexec/mergetop/types.go

    • Changed the data type of limit from int64 to uint64.
    +1/-1     
    offset.go
    Change `offset` data type from `int64` to `uint64`             

    pkg/sql/colexec/offset/offset.go

    • Changed the data type of offset from int64 to uint64.
    +1/-1     
    top.go
    Change `limit` data type and related functions to `uint64`

    pkg/sql/colexec/top/top.go

  • Changed the data type of limit from int64 to uint64.
  • Updated related functions to handle uint64 limits.
  • +8/-8     
    types.go
    Change `limit` data type from `int64` to `uint64`               

    pkg/sql/colexec/top/types.go

    • Changed the data type of limit from int64 to uint64.
    +1/-1     
    compile.go
    Change limit and offset data types to uint64 and add overflow check

    pkg/sql/compile/compile.go

  • Changed the data type of limit and offset from int64 to uint64.
  • Added overflow check for topN calculation.
  • +7/-3     
    apply_indices_vector.go
    Change constants data type from `int64` to `uint64`           

    pkg/sql/plan/apply_indices_vector.go

    • Changed the data type of constants from int64 to uint64.
    +2/-2     
    limit_binder.go
    Change `limit` and `offset` data types to `uint64` and add checks

    pkg/sql/plan/limit_binder.go

  • Changed the data type of limit and offset from int64 to uint64.
  • Added checks for negative values and conversions.
  • +21/-9   
    query_builder.go
    Change `limit` and `offset` data types to `uint64`             

    pkg/sql/plan/query_builder.go

  • Changed the data type of limit and offset from int64 to uint64.
  • Updated related logic to handle uint64.
  • +20/-26 
    stats.go
    Change `limit` data type from `int64` to `uint64`               

    pkg/sql/plan/stats.go

    • Changed the data type of limit from int64 to uint64.
    +4/-4     
    Tests
    10 files
    limit_test.go
    Update test cases for `limit` to use `uint64`                       

    pkg/sql/colexec/limit/limit_test.go

  • Updated test cases to use uint64 instead of int64 for LimitExpr.
  • +5/-4     
    limit_test.go
    Update test cases for `mergelimit` to use `uint64`             

    pkg/sql/colexec/mergelimit/limit_test.go

    • Updated test cases to use uint64 instead of int64 for Limit.
    +3/-2     
    offset_test.go
    Update test cases for `mergeoffset` to use `uint64`           

    pkg/sql/colexec/mergeoffset/offset_test.go

  • Updated test cases to use uint64 instead of int64 for OffsetExpr.
  • +3/-2     
    top_test.go
    Update test cases for `mergetop` to use `uint64`                 

    pkg/sql/colexec/mergetop/top_test.go

    • Updated test cases to use uint64 instead of int64 for Limit.
    +3/-2     
    offset_test.go
    Update test cases for `offset` to use `uint64`                     

    pkg/sql/colexec/offset/offset_test.go

  • Updated test cases to use uint64 instead of int64 for OffsetExpr.
  • +5/-4     
    top_test.go
    Update test cases for `top` to use `uint64`                           

    pkg/sql/colexec/top/top_test.go

    • Updated test cases to use uint64 instead of int64 for Limit.
    +3/-2     
    compile_test.go
    Update compile tests and add assertions                                   

    pkg/sql/compile/compile_test.go

  • Added require import for assertions.
  • Commented out a failing memory check assertion.
  • +9/-3     
    build_test.go
    Add test cases for `uint64` limits                                             

    pkg/sql/plan/build_test.go

    • Added test cases for uint64 limits.
    +20/-1   
    limit.result
    Add test results for `uint64` limits and offsets                 

    test/distributed/cases/dml/select/limit.result

    • Added test results for uint64 limits and offsets.
    +54/-0   
    limit.sql
    Add test cases for `uint64` limits and offsets                     

    test/distributed/cases/dml/select/limit.sql

    • Added test cases for uint64 limits and offsets.
    +19/-0   
    Enhancement
    2 files
    make.go
    Add function to create `uint64` constant expressions         

    pkg/sql/plan/make.go

    • Added function to create uint64 constant expressions.
    +1/-0     
    compiler_context.go
    Implement `GetSnapshot` function                                                 

    pkg/vm/engine/memoryengine/compiler_context.go

    • Implemented GetSnapshot function.
    +2/-4     

    💡 PR-Agent usage:
    Comment /help on the PR to get a list of all available PR-Agent tools and their descriptions

    @matrix-meow matrix-meow added the size/M Denotes a PR that changes [100,499] lines label Jun 5, 2024
    Copy link

    PR-Agent was enabled for this repository. To continue using it, please link your git user with your CodiumAI identity here.

    PR Review 🔍

    ⏱️ Estimated effort to review [1-5]

    3, because the PR involves changes across multiple files with modifications in data types and related logic which requires careful review to ensure that all edge cases and interactions are correctly handled.

    🧪 Relevant tests

    Yes

    ⚡ Possible issues

    Possible Bug: The conversion from int64 to uint64 could introduce issues if negative values were previously handled in a specific way. It's important to ensure that all previous usages of these variables are audited for correct behavior with the new unsigned type.

    Data Loss: If any system or logic implicitly relied on the signed nature of int64 (e.g., using negative values for special cases), this change could lead to unexpected behavior or data loss.

    🔒 Security concerns

    No

    Copy link

    PR-Agent was enabled for this repository. To continue using it, please link your git user with your CodiumAI identity here.

    PR Code Suggestions ✨

    CategorySuggestion                                                                                                                                    Score
    Possible bug
    Add a nil check before type assertion to prevent potential runtime panics

    To avoid potential runtime panics, add a check to ensure that cExpr.Lit.Value is not nil
    before asserting its type.

    pkg/sql/plan/stats.go [894]

    -if c, ok := cExpr.Lit.Value.(*plan.Literal_U64Val); ok {
    +if cExpr.Lit.Value != nil {
    +    if c, ok := cExpr.Lit.Value.(*plan.Literal_U64Val); ok {
     
    Suggestion importance[1-10]: 8

    Why: This suggestion addresses a potential runtime panic by adding a nil check before a type assertion, which is a crucial safety improvement.

    8
    Maintainability
    Replace unprofessional comments with a TODO comment to investigate the issue

    The commented-out code with the FIXME and !!!GOD!!! comments should be either removed or
    properly addressed. Leaving such comments in the codebase can be confusing and
    unprofessional.

    pkg/sql/compile/compile_test.go [161-165]

    -//FIXME:
    -//!!!GOD!!!
    -//Sometimes it is 0.
    -//Sometimes it is 24.
    -//require.Equal(t, int64(0), tc.proc.Mp().CurrNB())
    +// TODO: Investigate why `tc.proc.Mp().CurrNB()` sometimes returns 0 and sometimes 24.
    +// require.Equal(t, int64(0), tc.proc.Mp().CurrNB())
     
    Suggestion importance[1-10]: 7

    Why: The suggestion correctly identifies unprofessional comments and recommends replacing them with a more appropriate TODO comment, improving code professionalism and clarity.

    7
    Remove the unused testutil.NewProc() call to clean up the code

    The testutil.NewProc() call seems to be unused in the Test_limitUint64 function. If it is
    not necessary, it should be removed to clean up the code.

    pkg/sql/plan/build_test.go [1186]

    -testutil.NewProc()
     
    +
    Suggestion importance[1-10]: 7

    Why: Removing unused code is beneficial for maintainability and clarity. The suggestion correctly identifies an unnecessary function call.

    7
    Enhancement
    Simplify the overflow check by directly assigning the condition result to the variable

    The overflow variable can be directly assigned the result of the condition, simplifying
    the code.

    pkg/sql/compile/compile.go [766-768]

    -overflow := false
    -if topN < limit || topN < offset {
    -    overflow = true
    -}
    +overflow := topN < limit || topN < offset
     
    Suggestion importance[1-10]: 6

    Why: The suggestion improves code readability and efficiency by simplifying the overflow check, although it's a minor enhancement.

    6

    @mergify mergify bot merged commit a01fba2 into matrixorigin:main Jun 7, 2024
    17 of 18 checks passed
    XuPeng-SH added a commit to XuPeng-SH/matrixone that referenced this pull request Jun 11, 2024
    * fix bvt test (matrixorigin#16605)
    
    fix bvt test
    
    Approved by: @heni02
    
    * remove duplicates in object list for flushing (matrixorigin#16677)
    
    - reduce the size of tombstone files
    
    Approved by: @XuPeng-SH
    
    * fix stats for prefix_eq function (matrixorigin#16666)
    
    由于索引表总是将主键序列化进去,导致ndv很高,索引表的过滤度估计严重错误,会导致优化器错判tp/ap语句
    现在改成利用原始过滤条件的过滤度去计算prefix_eq函数的过滤度
    
    Approved by: @aunjgr
    
    * Fix condition to ignore delete booking if no transfer needed (matrixorigin#16633)
    
    Add a log statement to improve traceability when the transfer is not needed
    
    Approved by: @XuPeng-SH
    
    * make sure all pipeline run in single parallel for tp query (matrixorigin#16685)
    
    make sure all pipeline run in single parallel for tp query
    
    Approved by: @ouyuanning, @aunjgr
    
    * [Cherry-pick] handle null in convertRowsIntoBatch (matrixorigin#16676)
    
    handle null in convertRowsIntoBatch
    
    Approved by: @daviszhen
    
    * Fix enumtype system variable check  (matrixorigin#16691)
    
    Fix enumtype system variable check
    
    Approved by: @daviszhen
    
    * support query replica count of special cn (matrixorigin#16642)
    
    support query replica count of special cn
    
    Approved by: @reusee, @daviszhen
    
    * split build operator into merge and build operators (matrixorigin#16673)
    
    把收发数据的功能从build算子中拆开,拆成merge+build,为后续的重构做准备
    
    Approved by: @m-schen, @ouyuanning, @aunjgr
    
    * fileservice: add caching dns resolver (matrixorigin#16702)
    
    fileservice: longer timeout for http client
    
    Approved by: @zhangxu19830126
    
    * Fix-16620 (matrixorigin#16681)
    
    1.  Reuse latest partition state.
    
    Approved by: @badboynt1, @m-schen, @XuPeng-SH
    
    * rmTag16601_16597 (matrixorigin#16700)
    
    rm  tag 16601 and 16597
    
    Approved by: @heni02
    
    * optimize top operator in pipeline for tp query (matrixorigin#16704)
    
    optimize top operator in pipeline for tp query, don't need mergetop
    
    Approved by: @m-schen
    
    * optimize limit operator in pipeline for tp query (matrixorigin#16705)
    
    optimize limit operator in pipeline for tp query, don't need toplimit
    
    Approved by: @m-schen
    
    * add global system variable and session variable account isolation cases (matrixorigin#16694)
    
    add global system variable and session variable account isolation cases
    
    Approved by: @aressu1985
    
    * Add issue 16613 cases (matrixorigin#16719)
    
    Add issue 16613 cases
    
    Approved by: @aressu1985
    
    * add case for function hex() and unhex() (matrixorigin#16711)
    
    add case for hex() and unhex().
    
    Approved by: @heni02
    
    * optimize group operator in pipeline for tp query (matrixorigin#16717)
    
    optimize group operator in pipeline for tp query, don't need mergegroup
    
    Approved by: @m-schen
    
    * add debug info for panic (matrixorigin#16634)
    
    issue上的问题是事务状态异常。
    
    在出问题的调用栈上,增加事务状态的检测逻辑。
    
    txnIsValid 判断事务状态是否异常。
    
    Approved by: @badboynt1, @m-schen, @ouyuanning, @triump2020, @qingxinhome, @aunjgr
    
    * add optimizer hint exectype to force query to be ap or tp (matrixorigin#16722)
    
    add optimizer hint exectype to force query to be ap or tp
    
    Approved by: @ouyuanning
    
    * update bloom filter for the new prefix bf (matrixorigin#16684)
    
    support prefix bloom filter for object reader and writer
    
    Approved by: @XuPeng-SH
    
    * memorycache: code clean-ups (matrixorigin#16313)
    
    fileservice: remove IOEntry.ReadFromOSFile
    
    memorycache: remove RCBytes
    
    Approved by: @zhangxu19830126
    
    * optimize offset operator in pipeline for tp query (matrixorigin#16706)
    
    optimize limit operator in pipeline for tp query, don't need mergeoffset
    
    Approved by: @m-schen
    
    * add issue 16139 cases (matrixorigin#16733)
    
    add issue 16139 cases
    
    Approved by: @aressu1985
    
    * handle Restore Duplicate Entry (matrixorigin#16567)
    
    SQL执行时将事务WriteOffset与当前语句绑定,解决读数据万圣节问题
    
    MO Checkin Regression test susccess:
    https://github.com/matrixorigin/ci-test/actions/runs/9362961560
    https://github.com/matrixorigin/ci-test/actions/runs/9379340928
    
    Approved by: @daviszhen, @badboynt1, @m-schen, @reusee, @zhangxu19830126, @XuPeng-SH, @aunjgr, @triump2020
    
    * Handle Cancel Restore Statement Fail (matrixorigin#16735)
    
    handle `ctrl+c` failed to cancel during restore data
    
    Approved by: @daviszhen
    
    * fix a bug that cause ap performance regression on multi cn (matrixorigin#16737)
    
    fix a bug that cause ap performance regression on multi cn
    
    Approved by: @ouyuanning
    
    * optimize order operator in pipeline for tp query (matrixorigin#16742)
    
    把mergeorder算子拆成merge+mergeorder,把接收数据的功能独立出来。
    对于tp query,pipeline直接改成scan->order->mergeorder即可,不需要通过connector-merge进行连接
    
    Approved by: @ouyuanning, @m-schen
    
    * dashboard: refactor runtime dashboard (matrixorigin#16746)
    
    refine go runtime metrics dashboard
    
    Approved by: @zhangxu19830126
    
    * malloc: add profiler (matrixorigin#16699)
    
    malloc: refactor config
    
    malloc: add chainDeallocator, FuncDeallocator; optimize metrics allocator
    
    malloc: optimize metrics allocator
    
    malloc: enable metrics default
    
    Approved by: @zhangxu19830126
    
    * skip stats for create view (matrixorigin#16728)
    
    skip stats for create view
    
    Approved by: @daviszhen, @aunjgr
    
    * fileservice: add disk-based object storage (matrixorigin#16610)
    
    add local disk s3 fs object storage for testing purposes
    
    Approved by: @zhangxu19830126
    
    * block reader supports between filter. (matrixorigin#16674)
    
    block reader supports between filters.
    
    Approved by: @XuPeng-SH, @heni02, @aunjgr
    
    * change LIMIT,OFFSET's data type from int64 to uint64 (matrixorigin#16697)
    
    limit和offset 的数据类型改为uint64。
    
    Approved by: @m-schen, @reusee, @ouyuanning, @badboynt1, @aunjgr, @aressu1985
    
    * fix shard service panic (matrixorigin#16759)
    
    fix shard service panic
    
    Approved by: @reusee
    
    * make add txn error trace async (matrixorigin#16757)
    
    Fix hung when add txn error trace
    
    Approved by: @iamlinjunhong
    
    * Revert "resubmit pipeline client max connections is too large (matrixorigin#16209)" (matrixorigin#16754)
    
    Revert "resubmit pipeline client max connections is too large (matrixorigin#16209)"
    
    Approved by: @badboynt1, @daviszhen
    
    * [bug] launch: fix timeout mechanism when TN service startup (matrixorigin#16760)
    
    fix timeout mechanism when TN service startup:
    5m timeout for total wait and 5s timeout for each request.
    
    Approved by: @zhangxu19830126
    
    * add a code owner for vector (matrixorigin#16762)
    
    add XuPeng-SH as vector owner
    
    Approved by: @fengttt
    
    * refactor the block reader filter to support more expressions (matrixorigin#16756)
    
    1.  `>, >=, <, <=`
    2. `between and, prefix in, prefix between, prefix eq`
    3. `in, eq`
    
    Approved by: @XuPeng-SH, @aressu1985
    
    * update readme(1.2.0) (matrixorigin#16243)
    
    * Remove list checkpoint meta file (matrixorigin#16723)
    
    Remove list checkpoint meta file
    
    Approved by: @XuPeng-SH
    
    * fileservice: fix fd leak in LocalFS.read (matrixorigin#16748)
    
    fix fd leak in LocalFS.read
    
    Approved by: @fengttt
    
    * [BugFix]: Remove unnecessary projections in master index (matrixorigin#16766)
    
    Remove unnecessary projects from Master Index.
    
    ```sql
    mysql> explain analyze SELECT tbl.a100  FROM tbl  WHERE tbl.a75 = 'I2nJ0RqIQu';
    +---------------------------------------------------------------------------------------------------------------------------------------------------------+
    | AP QUERY PLAN ON MULTICN(10 core)                                                                                                                       |
    +---------------------------------------------------------------------------------------------------------------------------------------------------------+
    | Project                                                                                                                                                 |
    |   Analyze: timeConsumed=0ms waitTime=6ms inputRows=2 outputRows=1 InputSize=48bytes OutputSize=24bytes MemorySize=0bytes                                |
    |   ->  Join                                                                                                                                              |
    |         Analyze: timeConsumed=2ms waitTime=44ms inputRows=2 outputRows=1 InputSize=48bytes OutputSize=24bytes MemorySize=0bytes                         |
    |         Join Type: INDEX                                                                                                                                |
    |         Join Cond: (tbl.a100 = #[1,0])                                                                                                                  |
    |         Runtime Filter Build: #[-1,0]                                                                                                                   |
    |         ->  Table Scan on a.tbl [ForceOneCN]                                                                                                            |
    |               Analyze: timeConsumed=0ms waitTime=0ms inputBlocks=1 inputRows=1 outputRows=1 InputSize=48bytes OutputSize=24bytes MemorySize=50bytes     |
    |               Filter Cond: (tbl.a75 = 'I2nJ0RqIQu')                                                                                                     |
    |               Block Filter Cond: (tbl.a75 = 'I2nJ0RqIQu')                                                                                               |
    |               Runtime Filter Probe: tbl.a100                                                                                                            |
    |         ->  Table Scan on a.__mo_index_secondary_019003d8-3fd8-7455-b750-bd977ca13178 [ForceOneCN]                                                      |
    |               Analyze: timeConsumed=2ms waitTime=0ms inputBlocks=6 inputRows=49152 outputRows=1 InputSize=3mb OutputSize=24bytes MemorySize=696320bytes |
    |               Filter Cond: prefix_eq(#[0,0], 'F74 FI2nJ0RqIQu ')                                                                                      |
    |               Block Filter Cond: prefix_eq(#[0,0], 'F74 FI2nJ0RqIQu ')                                                                                |
    +---------------------------------------------------------------------------------------------------------------------------------------------------------+
    16 rows in set (0.01 sec)
    
    
    mysql> explain analyze SELECT tbl.a100  FROM tbl  WHERE tbl.a37 = '3Tfm6CEXy5' AND tbl.a94 = '6PRBdXpsVB';
    +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
    | AP QUERY PLAN ON MULTICN(10 core)                                                                                                                                                                         |
    +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
    | Project                                                                                                                                                                                                   |
    |   Analyze: timeConsumed=0ms waitTime=15ms inputRows=3 outputRows=1 InputSize=72bytes OutputSize=24bytes MemorySize=0bytes                                                                                 |
    |   ->  Join                                                                                                                                                                                                |
    |         Analyze: timeConsumed=4ms waitTime=72ms inputRows=2 outputRows=1 InputSize=48bytes OutputSize=24bytes MemorySize=0bytes                                                                           |
    |         Join Type: INDEX                                                                                                                                                                                  |
    |         Join Cond: (tbl.a100 = #[1,0])                                                                                                                                                                    |
    |         Runtime Filter Build: #[-1,0]                                                                                                                                                                     |
    |         ->  Table Scan on a.tbl [ForceOneCN]                                                                                                                                                              |
    |               Analyze: timeConsumed=0ms waitTime=0ms inputBlocks=1 inputRows=1 outputRows=1 InputSize=72bytes OutputSize=24bytes MemorySize=75bytes                                                       |
    |               Filter Cond: (tbl.a94 = '6PRBdXpsVB'), (tbl.a37 = '3Tfm6CEXy5')                                                                                                                             |
    |               Block Filter Cond: (tbl.a94 = '6PRBdXpsVB'), (tbl.a37 = '3Tfm6CEXy5')                                                                                                                       |
    |               Runtime Filter Probe: tbl.a100                                                                                                                                                              |
    |         ->  Join                                                                                                                                                                                          |
    |               Analyze: timeConsumed=4ms probe_time=[total=0ms,min=0ms,max=0ms,dop=10] build_time=[4ms] waitTime=57ms inputRows=2 outputRows=1 InputSize=48bytes OutputSize=24bytes MemorySize=361187bytes |
    |               Join Type: INNER                                                                                                                                                                            |
    |               Join Cond: (#[0,0] = #[1,0])                                                                                                                                                                |
    |               ->  Table Scan on a.__mo_index_secondary_019003d8-3fd8-7455-b750-bd977ca13178 [ForceOneCN]                                                                                                  |
    |                     Analyze: timeConsumed=3ms waitTime=0ms inputBlocks=4 inputRows=32768 outputRows=1 InputSize=2mb OutputSize=24bytes MemorySize=679936bytes                                             |
    |                     Filter Cond: prefix_eq(#[0,0], 'F36 F3Tfm6CEXy5 ')                                                                                                                                  |
    |                     Block Filter Cond: prefix_eq(#[0,0], 'F36 F3Tfm6CEXy5 ')                                                                                                                            |
    |               ->  Table Scan on a.__mo_index_secondary_019003d8-3fd8-7455-b750-bd977ca13178 [ForceOneCN]                                                                                                  |
    |                     Analyze: timeConsumed=3ms waitTime=0ms inputBlocks=6 inputRows=49152 outputRows=1 InputSize=3mb OutputSize=24bytes MemorySize=696320bytes                                             |
    |                     Filter Cond: prefix_eq(#[0,0], 'F93 F6PRBdXpsVB ')                                                                                                                                  |
    |                     Block Filter Cond: prefix_eq(#[0,0], 'F93 F6PRBdXpsVB ')                                                                                                                            |
    +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
    24 rows in set (0.00 sec)
    
    ```
    
    Approved by: @badboynt1
    
    * fix
    
    * fix
    
    * fix
    
    * fix
    
    ---------
    
    Co-authored-by: YANGGMM <www.yangzhao123@gmail.com>
    Co-authored-by: aptend <49832303+aptend@users.noreply.github.com>
    Co-authored-by: nitao <badboynt@126.com>
    Co-authored-by: Wei Ziran <weiziran125@gmail.com>
    Co-authored-by: Kai Cao <ck89119@users.noreply.github.com>
    Co-authored-by: fagongzi <zhangxu19830126@gmail.com>
    Co-authored-by: reusee <reusee@gmail.com>
    Co-authored-by: triump2020 <63033222+triump2020@users.noreply.github.com>
    Co-authored-by: Ariznawlll <ariznawl@163.com>
    Co-authored-by: heni02 <113406637+heni02@users.noreply.github.com>
    Co-authored-by: davis zhen <daviszhen007@gmail.com>
    Co-authored-by: GreatRiver <2552853833@qq.com>
    Co-authored-by: qingxinhome <70939751+qingxinhome@users.noreply.github.com>
    Co-authored-by: gouhongshen <gouhongshen@hotmail.com>
    Co-authored-by: zengyan1 <93656539+zengyan1@users.noreply.github.com>
    Co-authored-by: LiuBo <g.user.lb@gmail.com>
    Co-authored-by: yangj1211 <153493538+yangj1211@users.noreply.github.com>
    Co-authored-by: Arjun Sunil Kumar <arjunsk@users.noreply.github.com>
    XuPeng-SH added a commit to XuPeng-SH/matrixone that referenced this pull request Jun 12, 2024
    * fix bvt test (matrixorigin#16605)
    
    fix bvt test
    
    Approved by: @heni02
    
    * remove duplicates in object list for flushing (matrixorigin#16677)
    
    - reduce the size of tombstone files
    
    Approved by: @XuPeng-SH
    
    * fix stats for prefix_eq function (matrixorigin#16666)
    
    由于索引表总是将主键序列化进去,导致ndv很高,索引表的过滤度估计严重错误,会导致优化器错判tp/ap语句
    现在改成利用原始过滤条件的过滤度去计算prefix_eq函数的过滤度
    
    Approved by: @aunjgr
    
    * Fix condition to ignore delete booking if no transfer needed (matrixorigin#16633)
    
    Add a log statement to improve traceability when the transfer is not needed
    
    Approved by: @XuPeng-SH
    
    * make sure all pipeline run in single parallel for tp query (matrixorigin#16685)
    
    make sure all pipeline run in single parallel for tp query
    
    Approved by: @ouyuanning, @aunjgr
    
    * [Cherry-pick] handle null in convertRowsIntoBatch (matrixorigin#16676)
    
    handle null in convertRowsIntoBatch
    
    Approved by: @daviszhen
    
    * Fix enumtype system variable check  (matrixorigin#16691)
    
    Fix enumtype system variable check
    
    Approved by: @daviszhen
    
    * support query replica count of special cn (matrixorigin#16642)
    
    support query replica count of special cn
    
    Approved by: @reusee, @daviszhen
    
    * split build operator into merge and build operators (matrixorigin#16673)
    
    把收发数据的功能从build算子中拆开,拆成merge+build,为后续的重构做准备
    
    Approved by: @m-schen, @ouyuanning, @aunjgr
    
    * fileservice: add caching dns resolver (matrixorigin#16702)
    
    fileservice: longer timeout for http client
    
    Approved by: @zhangxu19830126
    
    * Fix-16620 (matrixorigin#16681)
    
    1.  Reuse latest partition state.
    
    Approved by: @badboynt1, @m-schen, @XuPeng-SH
    
    * rmTag16601_16597 (matrixorigin#16700)
    
    rm  tag 16601 and 16597
    
    Approved by: @heni02
    
    * optimize top operator in pipeline for tp query (matrixorigin#16704)
    
    optimize top operator in pipeline for tp query, don't need mergetop
    
    Approved by: @m-schen
    
    * optimize limit operator in pipeline for tp query (matrixorigin#16705)
    
    optimize limit operator in pipeline for tp query, don't need toplimit
    
    Approved by: @m-schen
    
    * add global system variable and session variable account isolation cases (matrixorigin#16694)
    
    add global system variable and session variable account isolation cases
    
    Approved by: @aressu1985
    
    * Add issue 16613 cases (matrixorigin#16719)
    
    Add issue 16613 cases
    
    Approved by: @aressu1985
    
    * add case for function hex() and unhex() (matrixorigin#16711)
    
    add case for hex() and unhex().
    
    Approved by: @heni02
    
    * optimize group operator in pipeline for tp query (matrixorigin#16717)
    
    optimize group operator in pipeline for tp query, don't need mergegroup
    
    Approved by: @m-schen
    
    * add debug info for panic (matrixorigin#16634)
    
    issue上的问题是事务状态异常。
    
    在出问题的调用栈上,增加事务状态的检测逻辑。
    
    txnIsValid 判断事务状态是否异常。
    
    Approved by: @badboynt1, @m-schen, @ouyuanning, @triump2020, @qingxinhome, @aunjgr
    
    * add optimizer hint exectype to force query to be ap or tp (matrixorigin#16722)
    
    add optimizer hint exectype to force query to be ap or tp
    
    Approved by: @ouyuanning
    
    * update bloom filter for the new prefix bf (matrixorigin#16684)
    
    support prefix bloom filter for object reader and writer
    
    Approved by: @XuPeng-SH
    
    * memorycache: code clean-ups (matrixorigin#16313)
    
    fileservice: remove IOEntry.ReadFromOSFile
    
    memorycache: remove RCBytes
    
    Approved by: @zhangxu19830126
    
    * optimize offset operator in pipeline for tp query (matrixorigin#16706)
    
    optimize limit operator in pipeline for tp query, don't need mergeoffset
    
    Approved by: @m-schen
    
    * add issue 16139 cases (matrixorigin#16733)
    
    add issue 16139 cases
    
    Approved by: @aressu1985
    
    * handle Restore Duplicate Entry (matrixorigin#16567)
    
    SQL执行时将事务WriteOffset与当前语句绑定,解决读数据万圣节问题
    
    MO Checkin Regression test susccess:
    https://github.com/matrixorigin/ci-test/actions/runs/9362961560
    https://github.com/matrixorigin/ci-test/actions/runs/9379340928
    
    Approved by: @daviszhen, @badboynt1, @m-schen, @reusee, @zhangxu19830126, @XuPeng-SH, @aunjgr, @triump2020
    
    * Handle Cancel Restore Statement Fail (matrixorigin#16735)
    
    handle `ctrl+c` failed to cancel during restore data
    
    Approved by: @daviszhen
    
    * fix a bug that cause ap performance regression on multi cn (matrixorigin#16737)
    
    fix a bug that cause ap performance regression on multi cn
    
    Approved by: @ouyuanning
    
    * optimize order operator in pipeline for tp query (matrixorigin#16742)
    
    把mergeorder算子拆成merge+mergeorder,把接收数据的功能独立出来。
    对于tp query,pipeline直接改成scan->order->mergeorder即可,不需要通过connector-merge进行连接
    
    Approved by: @ouyuanning, @m-schen
    
    * dashboard: refactor runtime dashboard (matrixorigin#16746)
    
    refine go runtime metrics dashboard
    
    Approved by: @zhangxu19830126
    
    * malloc: add profiler (matrixorigin#16699)
    
    malloc: refactor config
    
    malloc: add chainDeallocator, FuncDeallocator; optimize metrics allocator
    
    malloc: optimize metrics allocator
    
    malloc: enable metrics default
    
    Approved by: @zhangxu19830126
    
    * skip stats for create view (matrixorigin#16728)
    
    skip stats for create view
    
    Approved by: @daviszhen, @aunjgr
    
    * fileservice: add disk-based object storage (matrixorigin#16610)
    
    add local disk s3 fs object storage for testing purposes
    
    Approved by: @zhangxu19830126
    
    * block reader supports between filter. (matrixorigin#16674)
    
    block reader supports between filters.
    
    Approved by: @XuPeng-SH, @heni02, @aunjgr
    
    * change LIMIT,OFFSET's data type from int64 to uint64 (matrixorigin#16697)
    
    limit和offset 的数据类型改为uint64。
    
    Approved by: @m-schen, @reusee, @ouyuanning, @badboynt1, @aunjgr, @aressu1985
    
    * fix shard service panic (matrixorigin#16759)
    
    fix shard service panic
    
    Approved by: @reusee
    
    * make add txn error trace async (matrixorigin#16757)
    
    Fix hung when add txn error trace
    
    Approved by: @iamlinjunhong
    
    * Revert "resubmit pipeline client max connections is too large (matrixorigin#16209)" (matrixorigin#16754)
    
    Revert "resubmit pipeline client max connections is too large (matrixorigin#16209)"
    
    Approved by: @badboynt1, @daviszhen
    
    * [bug] launch: fix timeout mechanism when TN service startup (matrixorigin#16760)
    
    fix timeout mechanism when TN service startup:
    5m timeout for total wait and 5s timeout for each request.
    
    Approved by: @zhangxu19830126
    
    * add a code owner for vector (matrixorigin#16762)
    
    add XuPeng-SH as vector owner
    
    Approved by: @fengttt
    
    * refactor the block reader filter to support more expressions (matrixorigin#16756)
    
    1.  `>, >=, <, <=`
    2. `between and, prefix in, prefix between, prefix eq`
    3. `in, eq`
    
    Approved by: @XuPeng-SH, @aressu1985
    
    * update readme(1.2.0) (matrixorigin#16243)
    
    * Remove list checkpoint meta file (matrixorigin#16723)
    
    Remove list checkpoint meta file
    
    Approved by: @XuPeng-SH
    
    * fileservice: fix fd leak in LocalFS.read (matrixorigin#16748)
    
    fix fd leak in LocalFS.read
    
    Approved by: @fengttt
    
    * [BugFix]: Remove unnecessary projections in master index (matrixorigin#16766)
    
    Remove unnecessary projects from Master Index.
    
    ```sql
    mysql> explain analyze SELECT tbl.a100  FROM tbl  WHERE tbl.a75 = 'I2nJ0RqIQu';
    +---------------------------------------------------------------------------------------------------------------------------------------------------------+
    | AP QUERY PLAN ON MULTICN(10 core)                                                                                                                       |
    +---------------------------------------------------------------------------------------------------------------------------------------------------------+
    | Project                                                                                                                                                 |
    |   Analyze: timeConsumed=0ms waitTime=6ms inputRows=2 outputRows=1 InputSize=48bytes OutputSize=24bytes MemorySize=0bytes                                |
    |   ->  Join                                                                                                                                              |
    |         Analyze: timeConsumed=2ms waitTime=44ms inputRows=2 outputRows=1 InputSize=48bytes OutputSize=24bytes MemorySize=0bytes                         |
    |         Join Type: INDEX                                                                                                                                |
    |         Join Cond: (tbl.a100 = #[1,0])                                                                                                                  |
    |         Runtime Filter Build: #[-1,0]                                                                                                                   |
    |         ->  Table Scan on a.tbl [ForceOneCN]                                                                                                            |
    |               Analyze: timeConsumed=0ms waitTime=0ms inputBlocks=1 inputRows=1 outputRows=1 InputSize=48bytes OutputSize=24bytes MemorySize=50bytes     |
    |               Filter Cond: (tbl.a75 = 'I2nJ0RqIQu')                                                                                                     |
    |               Block Filter Cond: (tbl.a75 = 'I2nJ0RqIQu')                                                                                               |
    |               Runtime Filter Probe: tbl.a100                                                                                                            |
    |         ->  Table Scan on a.__mo_index_secondary_019003d8-3fd8-7455-b750-bd977ca13178 [ForceOneCN]                                                      |
    |               Analyze: timeConsumed=2ms waitTime=0ms inputBlocks=6 inputRows=49152 outputRows=1 InputSize=3mb OutputSize=24bytes MemorySize=696320bytes |
    |               Filter Cond: prefix_eq(#[0,0], 'F74 FI2nJ0RqIQu ')                                                                                      |
    |               Block Filter Cond: prefix_eq(#[0,0], 'F74 FI2nJ0RqIQu ')                                                                                |
    +---------------------------------------------------------------------------------------------------------------------------------------------------------+
    16 rows in set (0.01 sec)
    
    
    mysql> explain analyze SELECT tbl.a100  FROM tbl  WHERE tbl.a37 = '3Tfm6CEXy5' AND tbl.a94 = '6PRBdXpsVB';
    +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
    | AP QUERY PLAN ON MULTICN(10 core)                                                                                                                                                                         |
    +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
    | Project                                                                                                                                                                                                   |
    |   Analyze: timeConsumed=0ms waitTime=15ms inputRows=3 outputRows=1 InputSize=72bytes OutputSize=24bytes MemorySize=0bytes                                                                                 |
    |   ->  Join                                                                                                                                                                                                |
    |         Analyze: timeConsumed=4ms waitTime=72ms inputRows=2 outputRows=1 InputSize=48bytes OutputSize=24bytes MemorySize=0bytes                                                                           |
    |         Join Type: INDEX                                                                                                                                                                                  |
    |         Join Cond: (tbl.a100 = #[1,0])                                                                                                                                                                    |
    |         Runtime Filter Build: #[-1,0]                                                                                                                                                                     |
    |         ->  Table Scan on a.tbl [ForceOneCN]                                                                                                                                                              |
    |               Analyze: timeConsumed=0ms waitTime=0ms inputBlocks=1 inputRows=1 outputRows=1 InputSize=72bytes OutputSize=24bytes MemorySize=75bytes                                                       |
    |               Filter Cond: (tbl.a94 = '6PRBdXpsVB'), (tbl.a37 = '3Tfm6CEXy5')                                                                                                                             |
    |               Block Filter Cond: (tbl.a94 = '6PRBdXpsVB'), (tbl.a37 = '3Tfm6CEXy5')                                                                                                                       |
    |               Runtime Filter Probe: tbl.a100                                                                                                                                                              |
    |         ->  Join                                                                                                                                                                                          |
    |               Analyze: timeConsumed=4ms probe_time=[total=0ms,min=0ms,max=0ms,dop=10] build_time=[4ms] waitTime=57ms inputRows=2 outputRows=1 InputSize=48bytes OutputSize=24bytes MemorySize=361187bytes |
    |               Join Type: INNER                                                                                                                                                                            |
    |               Join Cond: (#[0,0] = #[1,0])                                                                                                                                                                |
    |               ->  Table Scan on a.__mo_index_secondary_019003d8-3fd8-7455-b750-bd977ca13178 [ForceOneCN]                                                                                                  |
    |                     Analyze: timeConsumed=3ms waitTime=0ms inputBlocks=4 inputRows=32768 outputRows=1 InputSize=2mb OutputSize=24bytes MemorySize=679936bytes                                             |
    |                     Filter Cond: prefix_eq(#[0,0], 'F36 F3Tfm6CEXy5 ')                                                                                                                                  |
    |                     Block Filter Cond: prefix_eq(#[0,0], 'F36 F3Tfm6CEXy5 ')                                                                                                                            |
    |               ->  Table Scan on a.__mo_index_secondary_019003d8-3fd8-7455-b750-bd977ca13178 [ForceOneCN]                                                                                                  |
    |                     Analyze: timeConsumed=3ms waitTime=0ms inputBlocks=6 inputRows=49152 outputRows=1 InputSize=3mb OutputSize=24bytes MemorySize=696320bytes                                             |
    |                     Filter Cond: prefix_eq(#[0,0], 'F93 F6PRBdXpsVB ')                                                                                                                                  |
    |                     Block Filter Cond: prefix_eq(#[0,0], 'F93 F6PRBdXpsVB ')                                                                                                                            |
    +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
    24 rows in set (0.00 sec)
    
    ```
    
    Approved by: @badboynt1
    
    * add SERVER_MORE_RESULTS_EXISTS judgment (matrixorigin#16712)
    
    补全响应client时的SERVER_MORE_RESULTS_EXISTS 设置
    
    Approved by: @daviszhen
    
    * fileservice: tune metrics dashboard (matrixorigin#16770)
    
    add metrics for io.ReadAll
    
    Approved by: @zhangxu19830126
    
    * [opt] retain IN expression in prepared stmt (matrixorigin#16744)
    
    don't convert IN expression to OR list in prepared statements
    
    Approved by: @ouyuanning
    
    * Refactor view scope execute (matrixorigin#15984)
    
    降低创建视图与创建表的操作的耦合,增强代码可读性,
    
    Approved by: @badboynt1, @ouyuanning, @m-schen, @aunjgr
    
    * optimize join pipeline for tp query (matrixorigin#16773)
    
    对于tp query,将build端的pipeline优化掉,直接将build算子添加到右子树上,不需要connector->merge进行连接
    并且直接在compile时完成,不需要在运行时再对pipeline做修改
    
    Approved by: @ouyuanning
    
    * add optimizer hints  (matrixorigin#16782)
    
    增加了一个optimizer hints选项,可以强制所有的right join改成left join
    修改了对query ap/tp hint的实现方式
    
    Approved by: @aunjgr
    
    * [BugFix]: Add ColName to MasterIndexScan for Filter PushDown (matrixorigin#16778)
    
    - Adding ColName to MasterIndex Optimizer Plan for Filter Pushdown.
    - With this change, master index performance is now reasonable. Hence removing the Experimental Flag.
    
    1 Filter Query QPS
    - No index: 500
    - 1 Master: 2820
    - 100 Secondary: 2958
    
    <details>
    <summary> Query Plan </summary>
    
    ```sql
    -- Master Index
    
    mysql> explain analyze SELECT tbl.a100  FROM tbl  WHERE tbl.a48 = 'b92k7dWP5t';
    +-----------------------------------------------------------------------------------------------------------------------------------------------------+
    | AP QUERY PLAN ON MULTICN(10 core)                                                                                                                   |
    +-----------------------------------------------------------------------------------------------------------------------------------------------------+
    | Project                                                                                                                                             |
    |   Analyze: timeConsumed=0ms waitTime=4ms inputRows=2 outputRows=1 InputSize=48bytes OutputSize=24bytes MemorySize=0bytes                            |
    |   ->  Join                                                                                                                                          |
    |         Analyze: timeConsumed=1ms waitTime=29ms inputRows=2 outputRows=1 InputSize=48bytes OutputSize=24bytes MemorySize=0bytes                     |
    |         Join Type: INDEX                                                                                                                            |
    |         Join Cond: (tbl.a100 = __mo_index_secondary_01900572-3257-76e5-9a7d-e4eaa2a28f17.__mo_index_pri_col)                                        |
    |         Runtime Filter Build: #[-1,0]                                                                                                               |
    |         ->  Table Scan on a.tbl [ForceOneCN]                                                                                                        |
    |               Analyze: timeConsumed=0ms waitTime=0ms inputBlocks=1 inputRows=1 outputRows=1 InputSize=48bytes OutputSize=24bytes MemorySize=50bytes |
    |               Filter Cond: (tbl.a48 = 'b92k7dWP5t')                                                                                                 |
    |               Block Filter Cond: (tbl.a48 = 'b92k7dWP5t')                                                                                           |
    |               Runtime Filter Probe: tbl.a100                                                                                                        |
    |         ->  Table Scan on a.__mo_index_secondary_01900572-3257-76e5-9a7d-e4eaa2a28f17 [ForceOneCN]                                                  |
    |               Analyze: timeConsumed=1ms waitTime=0ms inputBlocks=1 inputRows=1 outputRows=1 InputSize=79bytes OutputSize=24bytes MemorySize=80bytes |
    |               Filter Cond: prefix_eq(__mo_index_secondary_01900572-3257-76e5-9a7d-e4eaa2a28f17.__mo_index_idx_col, 'F47 Fb92k7dWP5t ')            |
    |               Block Filter Cond: prefix_eq(__mo_index_secondary_01900572-3257-76e5-9a7d-e4eaa2a28f17.__mo_index_idx_col, 'F47 Fb92k7dWP5t ')      |
    +-----------------------------------------------------------------------------------------------------------------------------------------------------+
    16 rows in set (0.00 sec)
    
    
    -- Secondary Index
    
    mysql> explain analyze SELECT tbl.a100  FROM tbl  WHERE tbl.a48 = 'b92k7dWP5t';
    +-----------------------------------------------------------------------------------------------------------------------------------------------------+
    | TP QURERY PLAN                                                                                                                                      |
    +-----------------------------------------------------------------------------------------------------------------------------------------------------+
    | Project                                                                                                                                             |
    |   Analyze: timeConsumed=0ms waitTime=0ms inputRows=2 outputRows=1 InputSize=48bytes OutputSize=24bytes MemorySize=0bytes                            |
    |   ->  Join                                                                                                                                          |
    |         Analyze: timeConsumed=0ms waitTime=1ms inputRows=1 outputRows=1 InputSize=24bytes OutputSize=24bytes MemorySize=0bytes                      |
    |         Join Type: INDEX                                                                                                                            |
    |         Join Cond: (tbl.a100 = __mo_index_secondary_01900577-f81d-7a7b-802c-b61a09a28067.__mo_index_pri_col)                                        |
    |         Runtime Filter Build: #[-1,0]                                                                                                               |
    |         ->  Table Scan on a.tbl [ForceOneCN]                                                                                                        |
    |               Analyze: timeConsumed=0ms waitTime=0ms inputBlocks=1 inputRows=1 outputRows=1 InputSize=48bytes OutputSize=24bytes MemorySize=50bytes |
    |               Filter Cond: (tbl.a48 = 'b92k7dWP5t')                                                                                                 |
    |               Block Filter Cond: (tbl.a48 = 'b92k7dWP5t')                                                                                           |
    |               Runtime Filter Probe: tbl.a100                                                                                                        |
    |         ->  Table Scan on a.__mo_index_secondary_01900577-f81d-7a7b-802c-b61a09a28067 [ForceOneCN]                                                  |
    |               Analyze: timeConsumed=0ms waitTime=0ms inputBlocks=1 inputRows=1 outputRows=1 InputSize=74bytes OutputSize=24bytes MemorySize=75bytes |
    |               Filter Cond: prefix_eq(__mo_index_secondary_01900577-f81d-7a7b-802c-b61a09a28067.__mo_index_idx_col, 'Fb92k7dWP5t ')                 |
    |               Block Filter Cond: prefix_eq(__mo_index_secondary_01900577-f81d-7a7b-802c-b61a09a28067.__mo_index_idx_col, 'Fb92k7dWP5t ')           |
    +-----------------------------------------------------------------------------------------------------------------------------------------------------+
    16 rows in set (0.00 sec)
    
    
    
    ```
    </details>
    
    
    2 Filter Query QPS
    - No Index:
    - 1 Master: 1335 (right now we use both the filters using inner join)
    - 100 Secondary: 1725 (we only make use of one secondary index table)
    
    <details>
    <summary> Query Plan </summary>
    
    
    ```sql
    -- master index
    mysql> explain analyze SELECT tbl.a100  FROM tbl  WHERE tbl.a89 = '40u4JSeGvz' AND tbl.a31 = '3X5ZOcJbol';
    +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
    | AP QUERY PLAN ON MULTICN(10 core)                                                                                                                                                                            |
    +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
    | Project                                                                                                                                                                                                      |
    |   Analyze: timeConsumed=0ms waitTime=36ms inputRows=3 outputRows=1 InputSize=72bytes OutputSize=24bytes MemorySize=0bytes                                                                                    |
    |   ->  Join                                                                                                                                                                                                   |
    |         Analyze: timeConsumed=11ms waitTime=167ms inputRows=2 outputRows=1 InputSize=48bytes OutputSize=24bytes MemorySize=0bytes                                                                            |
    |         Join Type: INDEX                                                                                                                                                                                     |
    |         Join Cond: (tbl.a100 = __mo_index_secondary_01900572-3257-76e5-9a7d-e4eaa2a28f17.__mo_index_pri_col)                                                                                                 |
    |         Runtime Filter Build: #[-1,0]                                                                                                                                                                        |
    |         ->  Table Scan on a.tbl [ForceOneCN]                                                                                                                                                                 |
    |               Analyze: timeConsumed=0ms waitTime=0ms inputBlocks=1 inputRows=1 outputRows=1 InputSize=72bytes OutputSize=24bytes MemorySize=75bytes                                                          |
    |               Filter Cond: (tbl.a89 = '40u4JSeGvz'), (tbl.a31 = '3X5ZOcJbol')                                                                                                                                |
    |               Block Filter Cond: (tbl.a89 = '40u4JSeGvz'), (tbl.a31 = '3X5ZOcJbol')                                                                                                                          |
    |               Runtime Filter Probe: tbl.a100                                                                                                                                                                 |
    |         ->  Join                                                                                                                                                                                             |
    |               Analyze: timeConsumed=11ms probe_time=[total=0ms,min=0ms,max=0ms,dop=10] build_time=[11ms] waitTime=145ms inputRows=2 outputRows=1 InputSize=48bytes OutputSize=24bytes MemorySize=361187bytes |
    |               Join Type: INNER                                                                                                                                                                               |
    |               Join Cond: (__mo_index_secondary_01900572-3257-76e5-9a7d-e4eaa2a28f17.__mo_index_pri_col = __mo_index_secondary_01900572-3257-76e5-9a7d-e4eaa2a28f17.__mo_index_pri_col)                       |
    |               ->  Table Scan on a.__mo_index_secondary_01900572-3257-76e5-9a7d-e4eaa2a28f17 [ForceOneCN]                                                                                                     |
    |                     Analyze: timeConsumed=5ms waitTime=0ms inputBlocks=1 inputRows=1 outputRows=1 InputSize=79bytes OutputSize=24bytes MemorySize=80bytes                                                    |
    |                     Filter Cond: prefix_eq(__mo_index_secondary_01900572-3257-76e5-9a7d-e4eaa2a28f17.__mo_index_idx_col, 'F30 F3X5ZOcJbol ')                                                               |
    |                     Block Filter Cond: prefix_eq(__mo_index_secondary_01900572-3257-76e5-9a7d-e4eaa2a28f17.__mo_index_idx_col, 'F30 F3X5ZOcJbol ')                                                         |
    |               ->  Table Scan on a.__mo_index_secondary_01900572-3257-76e5-9a7d-e4eaa2a28f17 [ForceOneCN]                                                                                                     |
    |                     Analyze: timeConsumed=11ms waitTime=0ms inputBlocks=1 inputRows=1 outputRows=1 InputSize=79bytes OutputSize=24bytes MemorySize=80bytes                                                   |
    |                     Filter Cond: prefix_eq(__mo_index_secondary_01900572-3257-76e5-9a7d-e4eaa2a28f17.__mo_index_idx_col, 'F88 F40u4JSeGvz ')                                                               |
    |                     Block Filter Cond: prefix_eq(__mo_index_secondary_01900572-3257-76e5-9a7d-e4eaa2a28f17.__mo_index_idx_col, 'F88 F40u4JSeGvz ')                                                         |
    +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
    24 rows in set (0.03 sec)
    
    
    
    -- secondary index
    
    mysql> explain analyze SELECT tbl.a100  FROM tbl  WHERE tbl.a89 = '40u4JSeGvz' AND tbl.a31 = '3X5ZOcJbol';
    +-----------------------------------------------------------------------------------------------------------------------------------------------------+
    | TP QURERY PLAN                                                                                                                                      |
    +-----------------------------------------------------------------------------------------------------------------------------------------------------+
    | Project                                                                                                                                             |
    |   Analyze: timeConsumed=0ms waitTime=0ms inputRows=2 outputRows=1 InputSize=48bytes OutputSize=24bytes MemorySize=0bytes                            |
    |   ->  Join                                                                                                                                          |
    |         Analyze: timeConsumed=0ms waitTime=1ms inputRows=1 outputRows=1 InputSize=24bytes OutputSize=24bytes MemorySize=0bytes                      |
    |         Join Type: INDEX                                                                                                                            |
    |         Join Cond: (tbl.a100 = __mo_index_secondary_01900568-4e08-7f3a-9fb9-755355944df6.__mo_index_pri_col)                                        |
    |         Runtime Filter Build: #[-1,0]                                                                                                               |
    |         ->  Table Scan on a.tbl [ForceOneCN]                                                                                                        |
    |               Analyze: timeConsumed=0ms waitTime=0ms inputBlocks=1 inputRows=1 outputRows=1 InputSize=72bytes OutputSize=24bytes MemorySize=75bytes |
    |               Filter Cond: (tbl.a89 = '40u4JSeGvz'), (tbl.a31 = '3X5ZOcJbol')                                                                       |
    |               Block Filter Cond: (tbl.a89 = '40u4JSeGvz'), (tbl.a31 = '3X5ZOcJbol')                                                                 |
    |               Runtime Filter Probe: tbl.a100                                                                                                        |
    |         ->  Table Scan on a.__mo_index_secondary_01900568-4e08-7f3a-9fb9-755355944df6 [ForceOneCN]                                                  |
    |               Analyze: timeConsumed=0ms waitTime=0ms inputBlocks=1 inputRows=1 outputRows=1 InputSize=74bytes OutputSize=24bytes MemorySize=75bytes |
    |               Filter Cond: prefix_eq(__mo_index_secondary_01900568-4e08-7f3a-9fb9-755355944df6.__mo_index_idx_col, 'F3X5ZOcJbol ')                 |
    |               Block Filter Cond: prefix_eq(__mo_index_secondary_01900568-4e08-7f3a-9fb9-755355944df6.__mo_index_idx_col, 'F3X5ZOcJbol ')           |
    +-----------------------------------------------------------------------------------------------------------------------------------------------------+
    16 rows in set (0.00 sec)
    ```
    </details>
    
    Approved by: @daviszhen, @badboynt1, @heni02
    
    * dashboard: fix runtime dashboard (matrixorigin#16771)
    
    refine runtime metrics dashboard
    
    Approved by: @zhangxu19830126
    
    * remove 1PC commands (matrixorigin#16786)
    
    remove 1PC subcommands
    
    Approved by: @XuPeng-SH
    
    * do not apply delete when flush (matrixorigin#16731)
    
    do not apply delete when flush
    
    Approved by: @XuPeng-SH
    
    * fix merge
    
    * rm 1pc
    
    ---------
    
    Co-authored-by: YANGGMM <www.yangzhao123@gmail.com>
    Co-authored-by: aptend <49832303+aptend@users.noreply.github.com>
    Co-authored-by: nitao <badboynt@126.com>
    Co-authored-by: Wei Ziran <weiziran125@gmail.com>
    Co-authored-by: Kai Cao <ck89119@users.noreply.github.com>
    Co-authored-by: fagongzi <zhangxu19830126@gmail.com>
    Co-authored-by: reusee <reusee@gmail.com>
    Co-authored-by: triump2020 <63033222+triump2020@users.noreply.github.com>
    Co-authored-by: Ariznawlll <ariznawl@163.com>
    Co-authored-by: heni02 <113406637+heni02@users.noreply.github.com>
    Co-authored-by: davis zhen <daviszhen007@gmail.com>
    Co-authored-by: GreatRiver <2552853833@qq.com>
    Co-authored-by: qingxinhome <70939751+qingxinhome@users.noreply.github.com>
    Co-authored-by: gouhongshen <gouhongshen@hotmail.com>
    Co-authored-by: zengyan1 <93656539+zengyan1@users.noreply.github.com>
    Co-authored-by: LiuBo <g.user.lb@gmail.com>
    Co-authored-by: XuPeng-SH <xupeng3112@163.com>
    Co-authored-by: yangj1211 <153493538+yangj1211@users.noreply.github.com>
    Co-authored-by: Arjun Sunil Kumar <arjunsk@users.noreply.github.com>
    Co-authored-by: CJKkkk_ <66134511+CJKkkk-315@users.noreply.github.com>
    Co-authored-by: bRong Njam <longran1989@gmail.com>
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    Bug fix kind/bug Something isn't working Review effort [1-5]: 3 size/M Denotes a PR that changes [100,499] lines Tests
    Projects
    None yet
    Development

    Successfully merging this pull request may close these issues.

    8 participants