New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

scheduler: refactor clusterInfo to extract a BasicCluster, and add mockCluster to easy testing #782

Merged
merged 7 commits into from Oct 17, 2017

Conversation

Projects
None yet
5 participants
@qqsun8819
Collaborator

qqsun8819 commented Oct 1, 2017

This pr resolves issue issue 752
1 add mockCluster in schedulers , and implements basic interface in schedule.Cluster.
2 move balance_test.go into schedulers
3 remove balance.go in server for duplicate code of schedulers/util.go

On the other hand, this pr imports some new bad smell code:
1 mockCluster has some duplicate code of clusterInfo in server/cache.go, due to dependency problem.
2 coordinator_test.go also some duplicate code of mockCluster, this is a remaining problem after we move balance_test.go out of server.
I plan to propose a new pr to resolve these two problems after this pr is merged

@qqsun8819 qqsun8819 requested review from siddontang, nolouch and disksing Oct 1, 2017

@sre-bot

This comment has been minimized.

Show comment
Hide comment
@sre-bot

sre-bot Oct 1, 2017

Hi contributor, thanks for your PR.

This patch needs to be approved by someone of admins. They should reply with "/ok-to-test" to accept this PR for running test automatically.

sre-bot commented Oct 1, 2017

Hi contributor, thanks for your PR.

This patch needs to be approved by someone of admins. They should reply with "/ok-to-test" to accept this PR for running test automatically.

// NewMockCluster creates a new mockCluster
func newMockCluster(id *core.MockIDAllocator) *mockCluster {
return &mockCluster{

This comment has been minimized.

@nolouch

nolouch Oct 4, 2017

Member

maybe we can let clusterinfo be public and use it directly

@nolouch

nolouch Oct 4, 2017

Member

maybe we can let clusterinfo be public and use it directly

@qqsun8819

This comment has been minimized.

Show comment
Hide comment
@qqsun8819

qqsun8819 Oct 7, 2017

Collaborator

@disksing @nolouch We need to discuss about how we deal with clusterInfo in cache.go, as @nolouch proposed in his comment.
As I mentioned in pr description, now mockCluster has many duplicate code as clusterInfo, so we should extract clusterInfo along into a separate package; but as clusterInfo depends on many data structure in server package ( struct in replication.go etc.) , these module also has to be moved into separate package like core. This change is somewhat big...
According to these problems , I have some proposals we can discuss so we can proceed for further refactor:
1 move clusterInfo into "schedules" package, or into a new package like "cluster".
2 resolve all the dependency problems caused by the movement of clusterInfo in 1.
3 whether 1 and 2 should be done in a new pr after this pr is merged, or do it just in this pr.
As 1 and 2 caused many changes, what I prefer is open a new pr to do this, and I need your suggestion.

Collaborator

qqsun8819 commented Oct 7, 2017

@disksing @nolouch We need to discuss about how we deal with clusterInfo in cache.go, as @nolouch proposed in his comment.
As I mentioned in pr description, now mockCluster has many duplicate code as clusterInfo, so we should extract clusterInfo along into a separate package; but as clusterInfo depends on many data structure in server package ( struct in replication.go etc.) , these module also has to be moved into separate package like core. This change is somewhat big...
According to these problems , I have some proposals we can discuss so we can proceed for further refactor:
1 move clusterInfo into "schedules" package, or into a new package like "cluster".
2 resolve all the dependency problems caused by the movement of clusterInfo in 1.
3 whether 1 and 2 should be done in a new pr after this pr is merged, or do it just in this pr.
As 1 and 2 caused many changes, what I prefer is open a new pr to do this, and I need your suggestion.

@disksing

This comment has been minimized.

Show comment
Hide comment
@disksing

disksing Oct 9, 2017

Member

How about extract a cluster struct in core or scheduler contains following four fields:

stores          *core.StoresInfo
regions         *core.RegionsInfo
writeStatistics cache.Cache
readStatistics  cache.Cache

Then embed it into clusterInfo and mockClusterInfo.

@qqsun8819

Member

disksing commented Oct 9, 2017

How about extract a cluster struct in core or scheduler contains following four fields:

stores          *core.StoresInfo
regions         *core.RegionsInfo
writeStatistics cache.Cache
readStatistics  cache.Cache

Then embed it into clusterInfo and mockClusterInfo.

@qqsun8819

@siddontang

This comment has been minimized.

Show comment
Hide comment
@siddontang

siddontang Oct 11, 2017

Member

Any update @qqsun8819

Member

siddontang commented Oct 11, 2017

Any update @qqsun8819

@siddontang siddontang changed the title from scheduler: add mockCluster to ease scheduler testing to [WIP] scheduler: add mockCluster to ease scheduler testing Oct 13, 2017

@qqsun8819

This comment has been minimized.

Show comment
Hide comment
@qqsun8819

qqsun8819 Oct 14, 2017

Collaborator

@siddontang I'm working on it , plan to complete it this weekend according to @disksing 's suggestion

Collaborator

qqsun8819 commented Oct 14, 2017

@siddontang I'm working on it , plan to complete it this weekend according to @disksing 's suggestion

@qqsun8819 qqsun8819 changed the title from [WIP] scheduler: add mockCluster to ease scheduler testing to scheduler: add mockCluster to ease scheduler testing Oct 14, 2017

@qqsun8819 qqsun8819 changed the title from scheduler: add mockCluster to ease scheduler testing to scheduler: refactor clusterInfo to extract a BasicCluster, and add mockCluster to easy testing Oct 14, 2017

@qqsun8819

This comment has been minimized.

Show comment
Hide comment
@qqsun8819

qqsun8819 Oct 14, 2017

Collaborator

@disksing @nolouch @siddontang PR is updated according to review comments, PTAL if you have time

Collaborator

qqsun8819 commented Oct 14, 2017

@disksing @nolouch @siddontang PR is updated according to review comments, PTAL if you have time

Show outdated Hide outdated server/cache_test.go
Show outdated Hide outdated server/cache_test.go
return &MockIDAllocator{base: 0}
}
// Alloc return a new id

This comment has been minimized.

@disksing

disksing Oct 15, 2017

Member

Alloc returns a new id.

@disksing

disksing Oct 15, 2017

Member

Alloc returns a new id.

"github.com/pingcap/kvproto/pkg/pdpb"
"github.com/pingcap/pd/server/cache"
"github.com/pingcap/pd/server/core"
"github.com/pingcap/pd/server/namespace"
"github.com/pingcap/pd/server/schedule"
_ "github.com/pingcap/pd/server/schedulers" // Register schedulers for tests.
"math"

This comment has been minimized.

@disksing

disksing Oct 15, 2017

Member

We usually put builtin packages before 3rd-part ones.

@disksing

disksing Oct 15, 2017

Member

We usually put builtin packages before 3rd-part ones.

This comment has been minimized.

@qqsun8819

qqsun8819 Oct 15, 2017

Collaborator

done

@qqsun8819

qqsun8819 Oct 15, 2017

Collaborator

done

@disksing

This comment has been minimized.

Show comment
Hide comment
@disksing

disksing Oct 15, 2017

Member

LGTM.

Member

disksing commented Oct 15, 2017

LGTM.

@nolouch

LGTM

@disksing

This comment has been minimized.

Show comment
Hide comment
@disksing

disksing Oct 17, 2017

Member

/run-all-tests

Member

disksing commented Oct 17, 2017

/run-all-tests

@disksing disksing merged commit efc52b5 into master Oct 17, 2017

7 checks passed

ci/circleci Your tests passed on CircleCI!
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
jenkins-ci-pd/build Jenkins job succeeded.
Details
jenkins-ci-pd/integration-common-test Jenkins job succeeded.
Details
jenkins-ci-pd/integration-ddl-test Jenkins job succeeded.
Details
jenkins-ci-pd/unit-test Jenkins job succeeded.
Details
license/cla Contributor License Agreement is signed.
Details

@disksing disksing deleted the qqsun8819/mockcluster branch Oct 17, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment