Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bugfix](fd) Recycle the segment file fds directly when delete stale rowset #22705

Merged
merged 3 commits into from
Aug 9, 2023

Conversation

Lchangliang
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

When load frequently, BE will do compaciton frequently. And because of set enable_vertical_compaction is true, when doing vectical compaction, it will cache the segments in segment cache. When the stale rowset is evicted, the segment cache doesn't recycle the fds in time because the recycle is lazy. So it will make mistakes because of "Too many open file".
Fix by tow methods:

  1. Recycle fds directly when delete stale rowsets.
  2. Don't cache fd when doing vectical compaction.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@Lchangliang
Copy link
Contributor Author

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Aug 8, 2023

clang-tidy review says "All clean, LGTM! 👍"

@Lchangliang
Copy link
Contributor Author

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Aug 8, 2023

clang-tidy review says "All clean, LGTM! 👍"

@hello-stephen
Copy link
Contributor

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.07 seconds
stream load tsv: 538 seconds loaded 74807831229 Bytes, about 132 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size: 17162196986 Bytes

@Lchangliang
Copy link
Contributor Author

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Aug 8, 2023

clang-tidy review says "All clean, LGTM! 👍"

@hello-stephen
Copy link
Contributor

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.17 seconds
stream load tsv: 513 seconds loaded 74807831229 Bytes, about 139 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.4 seconds inserted 10000000 Rows, about 340K ops/s
storage size: 17162109471 Bytes

Copy link
Contributor

@yiguolei yiguolei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Aug 8, 2023

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Aug 8, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Aug 8, 2023

PR approved by anyone and no changes requested.

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 1d1077c into apache:master Aug 9, 2023
16 of 17 checks passed
Lchangliang added a commit to Lchangliang/incubator-doris that referenced this pull request Aug 9, 2023
dataroaring pushed a commit to dataroaring/incubator-doris that referenced this pull request Aug 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.1-merged merge_conflict reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants