Skip to content

[enhancement](session-variable)add a use_fix_replica session variable to fix query replica#17101

Merged
yiguolei merged 3 commits intoapache:masterfrom
Cai-Yao:dev_02_24
Feb 27, 2023
Merged

[enhancement](session-variable)add a use_fix_replica session variable to fix query replica#17101
yiguolei merged 3 commits intoapache:masterfrom
Cai-Yao:dev_02_24

Conversation

@Cai-Yao
Copy link
Contributor

@Cai-Yao Cai-Yao commented Feb 24, 2023

Proposed changes

Issue Number: close #xxx

Problem summary

Add use_fix_replica session variable, so that we can be better debug replica inconsistencies problem.
If use_fix_replica default is -1, which means not fix,
else we will choose the {use_fix_replica} smallest replica.

Checklist(Required)

  • Does it affect the original behavior
  • Has unit tests been added
  • Has document been added or modified
  • Does it need to update dependencies
  • Is this PR support rollback (If NO, please explain WHY)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@github-actions github-actions bot added the area/planner Issues or PRs related to the query planner label Feb 24, 2023

public static final String MAX_TABLE_COUNT_USE_CASCADES_JOIN_REORDER = "max_table_count_use_cascades_join_reorder";

// fix replic to query. If num = 1, query the smallest replic, if 2 is the second smallest replic.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo error replic ---> replica

Collections.shuffle(replicas);
} else {
LOG.debug("use fix replica, value: {}, replica num: {}", useFixReplica, replicas.size());
// sort by replica id
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the logic is too complicated!!!
if (useFixReplica == -1) {
Collections.shuffle(replicas);
} else {
LOG.debug("use fix replica, value: {}, replica num: {}", useFixReplica, replicas.size());
// sort by replica id
replicas.sort(Replica.ID_COMPARATOR);
Replica replica = replicas.get(useFixReplica >= replicas.size() : replicas.size() - 1 ? useFixReplica);
replicas.clear();
replicas.add(replica);
}
Then other code does not need change.

@github-actions github-actions bot added the kind/docs Categorizes issue or PR as related to documentation. label Feb 25, 2023
@Cai-Yao
Copy link
Contributor Author

Cai-Yao commented Feb 25, 2023

run buildall

@hello-stephen
Copy link
Contributor

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.6 seconds
stream load tsv: 472 seconds loaded 74807831229 Bytes, about 151 MB/s
stream load json: 36 seconds loaded 2358488459 Bytes, about 62 MB/s
stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 27 seconds loaded 861443392 Bytes, about 30 MB/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230225032452_clickbench_pr_104291.html

Copy link
Contributor

@yiguolei yiguolei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 27, 2023
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@yiguolei yiguolei merged commit f228cfd into apache:master Feb 27, 2023
yiguolei pushed a commit that referenced this pull request Feb 28, 2023
Yulei-Yang pushed a commit to Yulei-Yang/doris that referenced this pull request Mar 5, 2023
… to fix query replica (apache#17101)

Add use_fix_replica session variable, so that we can be better debug replica inconsistencies problem.
If use_fix_replica default is -1, which means not fix,
else we will choose the {use_fix_replica} smallest replica.
yagagagaga pushed a commit to yagagagaga/doris that referenced this pull request Mar 9, 2023
… to fix query replica (apache#17101)

Add use_fix_replica session variable, so that we can be better debug replica inconsistencies problem.
If use_fix_replica default is -1, which means not fix,
else we will choose the {use_fix_replica} smallest replica.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. area/planner Issues or PRs related to the query planner dev/1.2.3-merged kind/docs Categorizes issue or PR as related to documentation. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants