Skip to content

Conversation

@kaijchen
Copy link
Member

Proposed changes

The sink who creates the delta writer may be closed while other sinks still using this delta writer.
The parent profile is deconstructed and when the last sink trying to update the profile, it will meet use-after-free.

To address this issue, we record the profile number in delta writer,
and the last sink who close the delta writer will create and update the profile.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@kaijchen
Copy link
Member Author

run buildall

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

}

Status DeltaWriterV2Map::close() {
Status DeltaWriterV2Map::close(RuntimeProfile* profile) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: method 'close' can be made static [readability-convert-member-functions-to-static]

be/src/vec/sink/delta_writer_v2_pool.h:72:

-     Status close(RuntimeProfile* profile);
+     static Status close(RuntimeProfile* profile);

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.28 seconds
stream load tsv: 556 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size: 17161969709 Bytes

@kaijchen
Copy link
Member Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.36% (8468/22665)
Line Coverage: 29.66% (68507/231008)
Region Coverage: 28.23% (35584/126028)
Branch Coverage: 25.09% (18210/72568)
Coverage Report: http://coverage.selectdb-in.cc/coverage/343b9def5086a12e93dfce80bfb6df8d4027f7ec_343b9def5086a12e93dfce80bfb6df8d4027f7ec/report/index.html

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46 seconds
stream load tsv: 552 seconds loaded 74807831229 Bytes, about 129 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162106405 Bytes

Copy link
Contributor

@yiguolei yiguolei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Oct 31, 2023
@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Collaborator

@wm1581066 wm1581066 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yiguolei yiguolei merged commit 8f32094 into apache:master Oct 31, 2023
dutyu pushed a commit to dutyu/doris that referenced this pull request Nov 4, 2023
…#26110)

The sink who creates the delta writer may be closed while other sinks still using this delta writer.
The parent profile is deconstructed and when the last sink trying to update the profile, it will meet use-after-free.

To address this issue, we record the profile number in delta writer,
and the last sink who close the delta writer will create and update the profile.
seawinde pushed a commit to seawinde/doris that referenced this pull request Nov 13, 2023
…#26110)

The sink who creates the delta writer may be closed while other sinks still using this delta writer.
The parent profile is deconstructed and when the last sink trying to update the profile, it will meet use-after-free.

To address this issue, we record the profile number in delta writer,
and the last sink who close the delta writer will create and update the profile.
XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
…#26110)

The sink who creates the delta writer may be closed while other sinks still using this delta writer.
The parent profile is deconstructed and when the last sink trying to update the profile, it will meet use-after-free.

To address this issue, we record the profile number in delta writer,
and the last sink who close the delta writer will create and update the profile.
@kaijchen kaijchen deleted the fix-mm-profile branch July 31, 2025 06:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants