Skip to content

/pd

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

schedule: actively push operator #1536

Merged
merged 11 commits into from May 19, 2019

Conversation

@nolouch
Copy link
Member

nolouch commented May 15, 2019 •
edited

What problem does this PR solve?

For tikv/tikv#4591 and speed up scheduling, we need actively push the operator.

What is changed and how it works?

use a heap to get the needed push operator.

Check List

Tests

  • Unit test
  • Manualy test
    TiKV use this branch: tikv/tikv#4684
    Without the improve:
    Screen Shot 2019-05-16 at 12 51 58 AM
    The branch:
    Screen Shot 2019-05-16 at 12 53 48 AM
nolouch added 2 commits May 15, 2019
@nolouch
schedule: actively push operator
217d93e 
Signed-off-by: nolouch <nolouch@gmail.com>
@nolouch
Merge remote-tracking branch 'origin' into active-operator
Loading status checks…
f3422f7
@nolouch nolouch added the WIP label May 15, 2019
@nolouch
remove limit
Loading status checks…
b8998b1 
Signed-off-by: nolouch <nolouch@gmail.com>
@nolouch nolouch force-pushed the nolouch:active-operator branch from a0c116c to b8998b1 May 15, 2019
nolouch added 2 commits May 16, 2019
@nolouch
add source
Loading status checks…
dc4ee6a 
Signed-off-by: nolouch <nolouch@gmail.com>
@nolouch
adjust intervel
Loading status checks…
d49f65f 
Signed-off-by: nolouch <nolouch@gmail.com>
@disksing
disksing reviewed May 16, 2019
View changes
server/coordinator.go Outdated Show resolved Hide resolved
server/coordinator.go Show resolved Hide resolved
server/coordinator.go Outdated Show resolved Hide resolved
server/schedule/operator_controller.go Outdated Show resolved Hide resolved
server/schedule/operator_queue.go Outdated Show resolved Hide resolved
server/schedule/operator_queue.go Outdated Show resolved Hide resolved
@nolouch
address comments
Loading status checks…
cfd3b57 
Signed-off-by: nolouch <nolouch@gmail.com>
@nolouch nolouch removed the WIP label May 16, 2019
@nolouch
Merge branch 'master' into active-operator
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
44f601d
@codecov-io

This comment has been minimized.

Sign in to view
Copy link

codecov-io commented May 16, 2019 •
edited

Codecov Report

❗️ No coverage uploaded for pull request base (master@6febb24). Click here to learn what that means.
The diff coverage is 91.46%.

Impacted file tree graph

@@            Coverage Diff            @@
##             master    #1536   +/-   ##
=========================================
  Coverage          ?   67.75%           
=========================================
  Files             ?      159           
  Lines             ?    15740           
  Branches          ?        0           
=========================================
  Hits              ?    10664           
  Misses            ?     4112           
  Partials          ?      964
Impacted Files Coverage Δ
tests/cluster.go 84.27% <100%> (ø)
server/schedule/operator_queue.go 100% <100%> (ø)
server/cluster_worker.go 48.33% <100%> (ø)
server/coordinator.go 85.39% <100%> (ø)
server/schedule/operator_controller.go 92.17% <86.53%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6febb24...93d3142. Read the comment docs.
@nolouch nolouch requested review from rleungx and disksing May 16, 2019
@disksing
disksing approved these changes May 16, 2019
View changes
@rleungx
rleungx reviewed May 16, 2019
View changes
server/schedule/operator_controller.go Outdated Show resolved Hide resolved
server/schedule/operator_controller.go Outdated Show resolved Hide resolved
server/schedule/operator_queue.go Outdated Show resolved Hide resolved
server/schedule/operator_controller.go Outdated Show resolved Hide resolved
server/schedule/operator_controller_test.go Outdated Show resolved Hide resolved
@nolouch
address comments
Loading status checks…
8016601 
Signed-off-by: nolouch <nolouch@gmail.com>
@nolouch nolouch force-pushed the nolouch:active-operator branch from 7608f63 to 8016601 May 17, 2019
@rleungx
rleungx approved these changes May 17, 2019
View changes
@nolouch nolouch force-pushed the nolouch:active-operator branch from 93d3142 to 8016601 May 17, 2019
@nolouch
Merge branch 'master' into active-operator
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
69b28b9
@shafreeck
shafreeck requested changes May 17, 2019
View changes
Copy link
Contributor

shafreeck left a comment

I think the main logic in this PR is implementing a timer with a priority queue, so why not use time.Timer instead?
server/coordinator.go Outdated Show resolved Hide resolved
server/schedule/operator_controller.go Show resolved Hide resolved
server/schedule/operator_queue.go Show resolved Hide resolved
nolouch added 2 commits May 17, 2019
@nolouch
address comments
9fc5ae4 
Signed-off-by: nolouch <nolouch@gmail.com>
@nolouch
Merge remote-tracking branch 'nolouch/active-operator' into active-op…
Loading status checks…
caa045d 
…erator
@shafreeck
shafreeck approved these changes May 17, 2019
View changes
Copy link
Contributor

shafreeck left a comment

LGTM
@nolouch

This comment has been minimized.

Sign in to view
Copy link
Member Author

nolouch commented May 17, 2019

/run-all-tests
@nolouch

This comment has been minimized.

Sign in to view
Copy link
Member Author

nolouch commented May 19, 2019

/ok-to-test
@nolouch nolouch merged commit b6150ca into pingcap:master May 19, 2019
8 checks passed
8 checks passed
ci/circleci Your tests passed on CircleCI!
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
idc-jenkins-ci-pd/integration-common-test Jenkins job succeeded.
Details
idc-jenkins-ci-pd/integration-compatibility-test Jenkins job succeeded.
Details
idc-jenkins-ci-pd/integration-ddl-test Jenkins job succeeded.
Details
idc-jenkins-ci/build Jenkins job succeeded.
Details
idc-jenkins-ci/test Jenkins job succeeded.
Details
license/cla Contributor License Agreement is signed.
Details
@nolouch nolouch deleted the nolouch:active-operator branch May 19, 2019
nolouch added a commit that referenced this pull request May 23, 2019
@nolouch
*: rebase release 3.0 to master (#1547)
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
9505c6f 
* tiny clean up code (#1526)

Signed-off-by: Ryan Leung <rleungx@gmail.com>

* config:  enable use region storage by default (#1524)

* config: default enable use region storage

* schedulers: let hot region balance not affect by balance-region-scheduler-limit (#1522)

Signed-off-by: nolouch <nolouch@gmail.com>

* log: do not add stack to error log (#1532)

* *: replace gofail with pingcap/failpoint (#1534)

* *: replace gofail with pingcap/failpoint

Signed-off-by: disksing <i@disksing.com>

* fix typo

Signed-off-by: disksing <i@disksing.com>

* fix typo

Signed-off-by: disksing <i@disksing.com>

* Update server/cluster_test.go

Co-Authored-By: Lonng <chris@lonng.org>

* Update server/tso.go

Co-Authored-By: Lonng <chris@lonng.org>

* Update server/tso.go

Co-Authored-By: Lonng <chris@lonng.org>

* check enable failpoint result

Signed-off-by: disksing <i@disksing.com>

* Update server/cluster.go

Co-Authored-By: Lonng <chris@lonng.org>

* server: set timeout for MoveLeader (#1533)

* server: set timeout for MoveLeader

Signed-off-by: disksing <i@disksing.com>

* client, server: add ScanRegions gRPC protocol support (#1535)

* client, server: support ScanRegions gRPC protocol

Signed-off-by: disksing <i@disksing.com>

* schedule: actively push operator  (#1536)

* schedule: actively push operator

Signed-off-by: nolouch <nolouch@gmail.com>

* *: update some dead links (#1543)

* update links

Signed-off-by: Ryan Leung <rleungx@gmail.com>

* Add windows build script (#1540)

Signed-off-by: Ana Hobden <operator@hoverbear.org>

* fix operator timeout metrics (#1541)

Signed-off-by: Ryan Leung <rleungx@gmail.com>

* *:Rollback config in store when kv.persist failed (#1476)

* tests: independent region sync test (#1545)

* tests: independent syncer region test

Signed-off-by: nolouch <nolouch@gmail.com>

* schedule: operator limit for stores (#1474)

* add store limit for scheduling

Signed-off-by: Ryan Leung <rleungx@gmail.com>

* statistic: add the statistic of flow (#1548)

* store_statistic: add the statistic of flow

Signed-off-by: nolouch <nolouch@gmail.com>
nolouch added a commit that referenced this pull request Aug 16, 2019
@nolouch
schedule: actively push operator (#1536)
559000c 
* schedule: actively push operator

Signed-off-by: nolouch <nolouch@gmail.com>
nolouch added a commit that referenced this pull request Aug 21, 2019
@nolouch
schedule: actively push operator (#1536) (#1686)
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
1cfe67b 
* schedule: actively push operator  (#1536)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@shafreeck shafreeck

@disksing disksing

@rleungx rleungx

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
5 participants
@nolouch @codecov-io @shafreeck @disksing @rleungx
You can’t perform that action at this time.