Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

*:Rollback config in store when kv.persist failed #1476

Merged
merged 34 commits into from May 22, 2019

Conversation

@bradyjoestar
Copy link
Contributor

bradyjoestar commented Mar 25, 2019

What problem does this PR solve?

#1475

What is changed and how it works?

rollback the store when persist config failed

Check List

Tests

  • Unit test
bradyjoestar and others added 6 commits Mar 20, 2019
merged
fix CI and switch to golangci-lint (#1471)
@sre-bot

This comment has been minimized.

Copy link
Contributor

sre-bot commented Mar 25, 2019

Hi contributor, thanks for your PR.

This patch needs to be approved by someone of admins. They should reply with "/ok-to-test" to accept this PR for running test automatically.

1 similar comment
@sre-bot

This comment has been minimized.

Copy link
Contributor

sre-bot commented Mar 25, 2019

Hi contributor, thanks for your PR.

This patch needs to be approved by someone of admins. They should reply with "/ok-to-test" to accept this PR for running test automatically.

@siddontang siddontang changed the title Issue 1475 Rollback config in store when kv.persist failed Mar 26, 2019
@bradyjoestar bradyjoestar changed the title Rollback config in store when kv.persist failed *:Rollback config in store when kv.persist failed Mar 27, 2019
@codecov-io

This comment has been minimized.

Copy link

codecov-io commented Mar 27, 2019

Codecov Report

Merging #1476 into master will increase coverage by 0.14%.
The diff coverage is 79.06%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1476      +/-   ##
==========================================
+ Coverage   67.87%   68.02%   +0.14%     
==========================================
  Files         159      159              
  Lines       15781    15843      +62     
==========================================
+ Hits        10712    10777      +65     
  Misses       4104     4104              
+ Partials      965      962       -3
Impacted Files Coverage Δ
server/server.go 81.19% <77.19%> (+2.97%) ⬆️
server/option.go 90.55% <82.75%> (+0.13%) ⬆️
pkg/etcdutil/etcdutil.go 86.04% <0%> (-9.31%) ⬇️
server/region_syncer/client.go 79.74% <0%> (-3.8%) ⬇️
server/leader.go 80.81% <0%> (-3.27%) ⬇️
server/util.go 75.49% <0%> (-2.65%) ⬇️
server/region_syncer/server.go 91.37% <0%> (-1.73%) ⬇️
server/cluster.go 74.46% <0%> (-0.54%) ⬇️
server/handler.go 55.47% <0%> (+0.5%) ⬆️
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update acf3e28...b710e24. Read the comment docs.

swbsin added 3 commits Mar 27, 2019
@bradyjoestar

This comment has been minimized.

Copy link
Contributor Author

bradyjoestar commented Mar 27, 2019

/rebuild

@bradyjoestar bradyjoestar changed the title *:Rollback config in store when kv.persist failed *:[WIP]Rollback config in store when kv.persist failed Mar 27, 2019
swbsin and others added 3 commits Mar 27, 2019
@bradyjoestar bradyjoestar changed the title *:[WIP]Rollback config in store when kv.persist failed *:Rollback config in store when kv.persist failed Mar 27, 2019
@@ -622,8 +648,12 @@ func (s *Server) SetNamespaceConfig(name string, cfg NamespaceConfig) error {
func (s *Server) DeleteNamespaceConfig(name string) error {
if n, ok := s.scheduleOpt.ns[name]; ok {
cfg := n.load()
delete(s.scheduleOpt.ns, name)
s.scheduleOpt.ns[name].store(&NamespaceConfig{})

This comment has been minimized.

Copy link
@rleungx

rleungx Mar 28, 2019

Member

Why not just use delete?

This comment has been minimized.

Copy link
@bradyjoestar

bradyjoestar Mar 29, 2019

Author Contributor

Delete is not atomic, It may cause DATA RACE error.
I have met it in running test.

This comment has been minimized.

Copy link
@nolouch

nolouch Apr 9, 2019

Member

But this way may get error config? all default is zero.

This comment has been minimized.

Copy link
@bradyjoestar

bradyjoestar Apr 11, 2019

Author Contributor

pd/server/server.go

Lines 578 to 582 in 9ba0e44

func (s *Server) GetNamespaceConfig(name string) *NamespaceConfig {
if _, ok := s.scheduleOpt.ns[name]; !ok {
return &NamespaceConfig{}
}

If ns[name] doesnt exist,it will return a &NamespaceConfig{} , so in a certain sense they are equivalent, I thought

This comment has been minimized.

Copy link
@nolouch

nolouch Apr 15, 2019

Member

got it.

@@ -68,7 +68,7 @@ func (s *serverTestSuite) TestRegionSyncer(c *C) {
c.Assert(err, IsNil)
}
// ensure flush to region kv
time.Sleep(3 * time.Second)
time.Sleep(5 * time.Second)

This comment has been minimized.

Copy link
@rleungx

This comment has been minimized.

Copy link
@bradyjoestar

bradyjoestar Mar 29, 2019

Author Contributor

Once I have met the following logs in jenkins

obtain value: 91
expect value:110

Maybe the time isnt enough? I guess. 😀 :

This comment has been minimized.

Copy link
@rleungx

rleungx Mar 29, 2019

Member

I doubt that this change can solve this problem actually. /cc @nolouch

This comment has been minimized.

Copy link
@nolouch

nolouch Apr 9, 2019

Member

I'm not sure, maybe relate to the ci environment.

This comment has been minimized.

Copy link
@bradyjoestar

bradyjoestar Apr 11, 2019

Author Contributor

restore to 3s.

@@ -536,6 +536,11 @@ func (s *Server) SetScheduleConfig(cfg ScheduleConfig) error {
old := s.scheduleOpt.load()
s.scheduleOpt.store(&cfg)
if err := s.scheduleOpt.persist(s.kv); err != nil {
s.scheduleOpt.store(old)
log.Error("schedule config updated failed",

This comment has been minimized.

Copy link
@rleungx

rleungx Mar 28, 2019

Member

How about changing the error message to failed to update xxx?

This comment has been minimized.

Copy link
@bradyjoestar

bradyjoestar Mar 29, 2019

Author Contributor

Sure. 👌

@rleungx rleungx added the contributor label Mar 28, 2019
swbsin and others added 4 commits Mar 29, 2019
bradyjoestar and others added 2 commits Apr 11, 2019
@nolouch

This comment has been minimized.

Copy link
Member

nolouch commented Apr 15, 2019

==================
WARNING: DATA RACE
Write at 0x00c001db4630 by goroutine 1160:
  runtime.mapassign_faststr()
      /usr/local/go/src/runtime/map_faststr.go:202 +0x0
  github.com/pingcap/pd/server.(*Server).SetNamespaceConfig()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/server.go:633 +0x1017
  github.com/pingcap/pd/server.(*testClusterSuite).TestSetScheduleOpt()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/cluster_test.go:644 +0x931
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /usr/local/go/src/fmt/scan.go:114 +0x193
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/storage/file_storage.go:458 +0x2b3
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xcf
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /usr/local/go/src/fmt/scan.go:114 +0x193
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/storage/file_storage.go:458 +0x2b3
  github.com/syndtr/goleveldb/leveldb.(*DB).checkAndCleanFiles()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/db_util.go:52 +0x2e1
  github.com/syndtr/goleveldb/leveldb.openDB()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/db.go:130 +0x9f3
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/storage/file_storage.go:458 +0x2b3
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xcf
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /usr/local/go/src/fmt/scan.go:114 +0x193
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/storage/file_storage.go:458 +0x2b3
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xcf
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /usr/local/go/src/fmt/scan.go:114 +0x286
  fmt.(*ss).doScanf()
      /usr/local/go/src/fmt/scan.go:1193 +0x377
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xcf
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /usr/local/go/src/fmt/scan.go:114 +0x193
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/storage/file_storage.go:458 +0x2b3
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xcf
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /usr/local/go/src/fmt/scan.go:114 +0x193
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/storage/file_storage.go:458 +0x2b3
  github.com/syndtr/goleveldb/leveldb.(*DB).recoverJournal()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/db.go:476 +0xc5
  github.com/syndtr/goleveldb/leveldb.openDB()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/db.go:125 +0x9cf
  github.com/syndtr/goleveldb/leveldb.Open()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/db.go:197 +0x1e9
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xcf
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /usr/local/go/src/fmt/scan.go:114 +0x193
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/storage/file_storage.go:458 +0x2b3
  github.com/syndtr/goleveldb/leveldb.(*session).recover.func1()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/session.go:113 +0xf4
  github.com/syndtr/goleveldb/leveldb.(*session).recover()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/session.go:121 +0x18f4
  github.com/syndtr/goleveldb/leveldb.Open()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/db.go:183 +0xd7
  github.com/syndtr/goleveldb/leveldb.OpenFile()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/db.go:219 +0xa6
  github.com/pingcap/pd/server/core.newLeveldbKV()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/core/levedb_kv.go:30 +0x52
  github.com/pingcap/pd/server/core.NewRegionKV()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/core/region_kv.go:52 +0x5d
  github.com/pingcap/pd/server.(*Server).startServer()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/server.go:225 +0x752
  github.com/pingcap/pd/server.(*Server).Run()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/server.go:306 +0xd5
  github.com/pingcap/pd/server.NewTestServer()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/testutil.go:52 +0xdc
  github.com/pingcap/pd/server.(*testClusterSuite).TestSetScheduleOpt()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/cluster_test.go:616 +0x53
  runtime.call32()
      /usr/local/go/src/runtime/asm_amd64.s:519 +0x3a
  reflect.Value.Call()
      /usr/local/go/src/reflect/value.go:308 +0xc0
  github.com/pingcap/check.(*suiteRunner).forkTest.func1()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20190102082844-67f458068fc8/check.go:836 +0x9fc
  github.com/pingcap/check.(*suiteRunner).forkCall.func1()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20190102082844-67f458068fc8/check.go:730 +0xb7

Previous read at 0x00c001db4630 by goroutine 502:
  runtime.mapaccess2_faststr()
      /usr/local/go/src/runtime/map_faststr.go:107 +0x0
  github.com/pingcap/pd/server.(*scheduleOption).GetMaxReplicas()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/option.go:65 +0x8c
  github.com/pingcap/pd/server.(*storeStatistics).Collect()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/store_statistics.go:128 +0xcee
  github.com/pingcap/pd/server.(*storeStatisticsMap).Collect()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/store_statistics.go:203 +0x112
  github.com/pingcap/pd/server.(*RaftCluster).collectMetrics()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/cluster.go:612 +0x251
  github.com/pingcap/pd/server.(*RaftCluster).runBackgroundJobs()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/cluster.go:650 +0x13a

Goroutine 1160 (running) created at:
  github.com/pingcap/check.(*suiteRunner).forkCall()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20190102082844-67f458068fc8/check.go:727 +0x4a7
  github.com/pingcap/check.(*suiteRunner).forkTest()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20190102082844-67f458068fc8/check.go:818 +0x1b9
  github.com/pingcap/check.(*suiteRunner).doRun()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20190102082844-67f458068fc8/check.go:659 +0x12d
  github.com/pingcap/check.(*suiteRunner).run()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20190102082844-67f458068fc8/check.go:689 +0x176
  github.com/pingcap/check.Run()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20190102082844-67f458068fc8/run.go:142 +0x5a
  github.com/pingcap/check.RunAll()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20190102082844-67f458068fc8/run.go:105 +0xf32
  github.com/pingcap/check.TestingT()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20190102082844-67f458068fc8/run.go:91 +0x770
  github.com/pingcap/pd/server.TestServer()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/server_test.go:27 +0x4f
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:865 +0x163

Goroutine 502 (running) created at:
  github.com/pingcap/pd/server.(*RaftCluster).start()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/cluster.go:127 +0x69f
  github.com/pingcap/pd/server.(*Server).bootstrapCluster()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/server.go:427 +0x1639
  github.com/pingcap/pd/server.(*testClusterSuite).TestSetScheduleOpt()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/cluster_test.go:624 +0x36c
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /usr/local/go/src/fmt/scan.go:114 +0x193
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/storage/file_storage.go:458 +0x2b3
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xcf
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /usr/local/go/src/fmt/scan.go:114 +0x193
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/storage/file_storage.go:458 +0x2b3
  github.com/syndtr/goleveldb/leveldb.(*DB).checkAndCleanFiles()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/db_util.go:52 +0x2e1
  github.com/syndtr/goleveldb/leveldb.openDB()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/db.go:130 +0x9f3
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/storage/file_storage.go:458 +0x2b3
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xcf
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /usr/local/go/src/fmt/scan.go:114 +0x193
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/storage/file_storage.go:458 +0x2b3
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xcf
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /usr/local/go/src/fmt/scan.go:114 +0x286
  fmt.(*ss).doScanf()
      /usr/local/go/src/fmt/scan.go:1193 +0x377
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xcf
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /usr/local/go/src/fmt/scan.go:114 +0x193
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/storage/file_storage.go:458 +0x2b3
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xcf
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /usr/local/go/src/fmt/scan.go:114 +0x193
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/storage/file_storage.go:458 +0x2b3
  github.com/syndtr/goleveldb/leveldb.(*DB).recoverJournal()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/db.go:476 +0xc5
  github.com/syndtr/goleveldb/leveldb.openDB()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/db.go:125 +0x9cf
  github.com/syndtr/goleveldb/leveldb.Open()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/db.go:197 +0x1e9
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xcf
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /usr/local/go/src/fmt/scan.go:114 +0x193
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/storage/file_storage.go:458 +0x2b3
  github.com/syndtr/goleveldb/leveldb.(*session).recover.func1()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/session.go:113 +0xf4
  github.com/syndtr/goleveldb/leveldb.(*session).recover()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/session.go:121 +0x18f4
  github.com/syndtr/goleveldb/leveldb.Open()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/db.go:183 +0xd7
  github.com/syndtr/goleveldb/leveldb.OpenFile()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/db.go:219 +0xa6
  github.com/pingcap/pd/server/core.newLeveldbKV()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/core/levedb_kv.go:30 +0x52
  github.com/pingcap/pd/server/core.NewRegionKV()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/core/region_kv.go:52 +0x5d
  github.com/pingcap/pd/server.(*Server).startServer()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/server.go:225 +0x752
  github.com/pingcap/pd/server.(*Server).Run()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/server.go:306 +0xd5
  github.com/pingcap/pd/server.NewTestServer()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/testutil.go:52 +0xdc
  github.com/pingcap/pd/server.(*testClusterSuite).TestSetScheduleOpt()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/cluster_test.go:616 +0x53
  runtime.call32()
      /usr/local/go/src/runtime/asm_amd64.s:519 +0x3a
  reflect.Value.Call()
      /usr/local/go/src/reflect/value.go:308 +0xc0
  github.com/pingcap/check.(*suiteRunner).forkTest.func1()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20190102082844-67f458068fc8/check.go:836 +0x9fc
  github.com/pingcap/check.(*suiteRunner).forkCall.func1()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20190102082844-67f458068fc8/check.go:730 +0xb7
==================

@bradyjoestar , here also has a race problem, we need let SetNamespaceConfig be a thread-safe function.

@bradyjoestar

This comment has been minimized.

Copy link
Contributor Author

bradyjoestar commented Apr 17, 2019

/rebuild

@bradyjoestar

This comment has been minimized.

Copy link
Contributor Author

bradyjoestar commented Apr 17, 2019

ns map[string]*namespaceOption is not atomic although *namespaceOption is atomic.

Same error will meet

pd/server/option.go

Lines 309 to 321 in 22032cb

if isExist {
o.store(&cfg.Schedule)
o.rep.store(&cfg.Replication)
for name, nsCfg := range cfg.Namespace {
nsCfg := nsCfg
o.ns[name] = newNamespaceOption(&nsCfg)
}
o.labelProperty.Store(cfg.LabelProperty)
o.clusterVersion.Store(cfg.ClusterVersion)
o.pdServerConfig.Store(&cfg.PDServerCfg)
}
return nil
}

@nolouch

This comment has been minimized.

Copy link
Member

nolouch commented Apr 25, 2019

@@ -16,6 +16,7 @@ package server
import (
"context"
"fmt"
"github.com/pkg/errors"

This comment has been minimized.

Copy link
@disksing

disksing May 21, 2019

Member

Seems not in good order. We recommend to put it with other 3rd part packages.

}
o.ns.Range(f)

This comment has been minimized.

Copy link
@disksing

disksing May 21, 2019

Member

em, this part is exactly the same with the code block in persist. How about extract a function?

This comment has been minimized.

Copy link
@bradyjoestar

bradyjoestar May 21, 2019

Author Contributor

Sure!👌

}
s.scheduleOpt.ns.Range(f)

This comment has been minimized.

Copy link
@disksing

disksing May 21, 2019

Member

The same pattern comes again :)

@disksing

This comment has been minimized.

Copy link
Member

disksing commented May 21, 2019

The rest LGTM.

@disksing

This comment has been minimized.

Copy link
Member

disksing commented May 21, 2019

@nolouch nolouch merged commit 1268d8b into pingcap:master May 22, 2019
5 checks passed
5 checks passed
ci/circleci Your tests passed on CircleCI!
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
idc-jenkins-ci/build Jenkins job succeeded.
Details
idc-jenkins-ci/test Jenkins job succeeded.
Details
license/cla Contributor License Agreement is signed.
Details
nolouch added a commit that referenced this pull request May 23, 2019
* tiny clean up code (#1526)

Signed-off-by: Ryan Leung <rleungx@gmail.com>

* config:  enable use region storage by default (#1524)

* config: default enable use region storage

* schedulers: let hot region balance not affect by balance-region-scheduler-limit (#1522)

Signed-off-by: nolouch <nolouch@gmail.com>

* log: do not add stack to error log (#1532)

* *: replace gofail with pingcap/failpoint (#1534)

* *: replace gofail with pingcap/failpoint

Signed-off-by: disksing <i@disksing.com>

* fix typo

Signed-off-by: disksing <i@disksing.com>

* fix typo

Signed-off-by: disksing <i@disksing.com>

* Update server/cluster_test.go

Co-Authored-By: Lonng <chris@lonng.org>

* Update server/tso.go

Co-Authored-By: Lonng <chris@lonng.org>

* Update server/tso.go

Co-Authored-By: Lonng <chris@lonng.org>

* check enable failpoint result

Signed-off-by: disksing <i@disksing.com>

* Update server/cluster.go

Co-Authored-By: Lonng <chris@lonng.org>

* server: set timeout for MoveLeader (#1533)

* server: set timeout for MoveLeader

Signed-off-by: disksing <i@disksing.com>

* client, server: add ScanRegions gRPC protocol support (#1535)

* client, server: support ScanRegions gRPC protocol

Signed-off-by: disksing <i@disksing.com>

* schedule: actively push operator  (#1536)

* schedule: actively push operator

Signed-off-by: nolouch <nolouch@gmail.com>

* *: update some dead links (#1543)

* update links

Signed-off-by: Ryan Leung <rleungx@gmail.com>

* Add windows build script (#1540)

Signed-off-by: Ana Hobden <operator@hoverbear.org>

* fix operator timeout metrics (#1541)

Signed-off-by: Ryan Leung <rleungx@gmail.com>

* *:Rollback config in store when kv.persist failed (#1476)

* tests: independent region sync test (#1545)

* tests: independent syncer region test

Signed-off-by: nolouch <nolouch@gmail.com>

* schedule: operator limit for stores (#1474)

* add store limit for scheduling

Signed-off-by: Ryan Leung <rleungx@gmail.com>

* statistic: add the statistic of flow (#1548)

* store_statistic: add the statistic of flow

Signed-off-by: nolouch <nolouch@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.