Skip to content

[fix](Dictionary-codec) heap overflow with in-predicate on nullable columns (#14319)#14641

Merged
BiteTheDDDDt merged 1 commit intoapache:masterfrom
freemandealer:updated_new_dict_heap_overflow
Nov 29, 2022
Merged

[fix](Dictionary-codec) heap overflow with in-predicate on nullable columns (#14319)#14641
BiteTheDDDDt merged 1 commit intoapache:masterfrom
freemandealer:updated_new_dict_heap_overflow

Conversation

@freemandealer
Copy link
Copy Markdown
Contributor

@freemandealer freemandealer commented Nov 28, 2022

Losing segmentid info will mess up the _segment_id_to_value_in_dict_flags map in InListPredicate,
causing two distinct segments to collide and crash the BE at last.

Root causes of the problem:

  1. NullableColumn use nested column but the rowsetid&segmentid info cannot pass on to the nested one.
    So overrided set function should be provided instead of use the dummy virtual one in the base class.
  2. Set function only called in short_cir_eval and pre_eval while delete_range and delete_bloom_filter are left.

Signed-off-by: freemandealer freeman.zhang1992@gmail.com

Co-authored-by: Pxl pxl290@qq.com

Proposed changes

Issue Number: close #14319

Problem summary

Describe your changes.

Checklist(Required)

  1. Does it affect the original behavior:
    • Yes
    • No
    • I don't know
  2. Has unit tests been added:
    • Yes
    • No
    • No Need
  3. Has document been added or modified:
    • Yes
    • No
    • No Need
  4. Does it need to update dependencies:
    • Yes
    • No
  5. Are there any changes that cannot be rolled back:
    • Yes (If Yes, please explain WHY)
    • No

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@dataroaring dataroaring added the bug/sqlsmith Bug detected by sqlsmith label Nov 28, 2022
@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented Nov 28, 2022

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 35.17 seconds
load time: 458 seconds
storage size: 17122921389 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221129095227_clickbench_pr_54921.html

…olumns (apache#14319)

Losing segmentid info will mess up the _segment_id_to_value_in_dict_flags map
in InListPredicate, causing two distinct segments to collide and crash the BE
at last.

Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
@freemandealer freemandealer force-pushed the updated_new_dict_heap_overflow branch from fa943b5 to c48cb47 Compare November 29, 2022 09:02
Copy link
Copy Markdown
Contributor

@BiteTheDDDDt BiteTheDDDDt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 29, 2022
@github-actions
Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@BiteTheDDDDt
Copy link
Copy Markdown
Contributor

need merge this to 1.2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. area/vectorization bug/sqlsmith Bug detected by sqlsmith dev/1.2.1-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] heap overflow with in-predicate on a nullable column

6 participants