Skip to content

[Improvement](pipeline) support send eos on local exchange and remove some unused code#22086

Merged
BiteTheDDDDt merged 2 commits into
apache:masterfrom
BiteTheDDDDt:dev_0721
Jul 24, 2023
Merged

[Improvement](pipeline) support send eos on local exchange and remove some unused code#22086
BiteTheDDDDt merged 2 commits into
apache:masterfrom
BiteTheDDDDt:dev_0721

Conversation

@BiteTheDDDDt
Copy link
Copy Markdown
Contributor

Proposed changes

support send eos on local exchange and remove some unused code

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@BiteTheDDDDt
Copy link
Copy Markdown
Contributor Author

run buildall

@github-actions
Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@hello-stephen
Copy link
Copy Markdown
Contributor

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.84 seconds
stream load tsv: 504 seconds loaded 74807831229 Bytes, about 141 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17164139361 Bytes

@BiteTheDDDDt
Copy link
Copy Markdown
Contributor Author

run buildall

@github-actions
Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@hello-stephen
Copy link
Copy Markdown
Contributor

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.19 seconds
stream load tsv: 509 seconds loaded 74807831229 Bytes, about 140 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 30.0 seconds inserted 10000000 Rows, about 333K ops/s
storage size: 17167040153 Bytes

Copy link
Copy Markdown
Contributor

@xinyiZzz xinyiZzz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Jul 21, 2023
@github-actions
Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

SCOPED_CONSUME_MEM_TRACKER(_parent->_mem_tracker.get());
status = send_block((PBlock*)nullptr, true);
if (is_local()) {
if (_recvr_is_valid()) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if just remove the sender not send a null block to receiver. Then the receiver may not get eof signal?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The receiver just judge the eof by sender_number == 0;

Copy link
Copy Markdown
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@BiteTheDDDDt BiteTheDDDDt merged commit 19ba6be into apache:master Jul 24, 2023
@xiaokang xiaokang added dev/2.0.0 2.0.0 release dev/2.0.0-merged and removed dev/2.0.0 2.0.0 release labels Jul 24, 2023
xiaokang pushed a commit that referenced this pull request Jul 24, 2023
… some unused code (#22086)

support send eos on local exchange and remove some unused code
@BiteTheDDDDt BiteTheDDDDt deleted the dev_0721 branch January 20, 2025 06:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.0-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants