Skip to content

Commit

Permalink
Wait for orderer to terminate before restarting (#460)
Browse files Browse the repository at this point in the history 
If a dying orderer is still alive when the replacement is started, it
can result in a ledger panic.

[e][OrdererOrg.orderer1] 2020-01-07 13:32:59.912 EST [orderer.common.server] initializeServerConfig -> INFO 00f Starting orderer with TLS enabled
[e][OrdererOrg.orderer1] panic: Error opening leveldb: resource temporarily unavailable
[e][OrdererOrg.orderer1]
[e][OrdererOrg.orderer1] goroutine 1 [running]:
[e][OrdererOrg.orderer1] github.com/hyperledger/fabric/common/ledger/util/leveldbhelper.(*DB).Open(0xc000178e10)
[e][OrdererOrg.orderer1] 	/Users/sykesm/workspace/fabric/src/github.com/hyperledger/fabric/common/ledger/util/leveldbhelper/leveldb_helper.go:75 +0x285
[e][OrdererOrg.orderer1] github.com/hyperledger/fabric/common/ledger/util/leveldbhelper.openDBAndCheckFormat(0xc000361580, 0x0, 0x0, 0x0)
[e][OrdererOrg.orderer1] 	/Users/sykesm/workspace/fabric/src/github.com/hyperledger/fabric/common/ledger/util/leveldbhelper/leveldb_provider.go:63 +0x11b
[e][OrdererOrg.orderer1] github.com/hyperledger/fabric/common/ledger/util/leveldbhelper.NewProvider(0xc000361580, 0xc000361580, 0x4, 0xc00014c700)
[e][OrdererOrg.orderer1] 	/Users/sykesm/workspace/fabric/src/github.com/hyperledger/fabric/common/ledger/util/leveldbhelper/leveldb_provider.go:51 +0x2f
[e][OrdererOrg.orderer1] github.com/hyperledger/fabric/common/ledger/blkstorage/fsblkstorage.NewProvider(0xc000361540, 0xc000361560, 0x4df0e80, 0x55ed2e0, 0xc00014c690, 0x6d, 0x0, 0x0)
[e][OrdererOrg.orderer1] 	/Users/sykesm/workspace/fabric/src/github.com/hyperledger/fabric/common/ledger/blkstorage/fsblkstorage/fs_blockstore_provider.go:43 +0x158
[e][OrdererOrg.orderer1] github.com/hyperledger/fabric/common/ledger/blockledger/fileledger.New(0xc00014ca80, 0x61, 0x4df0e80, 0x55ed2e0, 0x8c00008, 0x0, 0xc000429630, 0x400f158)
[e][OrdererOrg.orderer1] 	/Users/sykesm/workspace/fabric/src/github.com/hyperledger/fabric/common/ledger/blockledger/fileledger/factory.go:71 +0x107
[e][OrdererOrg.orderer1] github.com/hyperledger/fabric/orderer/common/server.createLedgerFactory(0xc0002f7200, 0x4df0e80, 0x55ed2e0, 0xc000014700, 0x45b, 0x65b, 0xc000018300, 0xf1, 0x2f1)
[e][OrdererOrg.orderer1] 	/Users/sykesm/workspace/fabric/src/github.com/hyperledger/fabric/orderer/common/server/util.go:29 +0xf8
[e][OrdererOrg.orderer1] github.com/hyperledger/fabric/orderer/common/server.Main()
[e][OrdererOrg.orderer1] 	/Users/sykesm/workspace/fabric/src/github.com/hyperledger/fabric/orderer/common/server/main.go:113 +0x664
[e][OrdererOrg.orderer1] main.main()
[e][OrdererOrg.orderer1] 	/Users/sykesm/workspace/fabric/src/github.com/hyperledger/fabric/cmd/orderer/main.go:15 +0x20
[d][OrdererOrg.orderer2] spawned /var/folders/tw/dsbwtnf50w7dhvhh0tkg7fwr0000gn/T/gexec_artifacts750850912/g028006345/orderer (pid: 15850)

Waiting for the orderer to terminate helps ensure the DB lock is
released.

FAB-17309

Change-Id: Ic53e61deceb0121e1d57e80e1a59abcac57e2eda
Signed-off-by: Matthew Sykes <sykesmat@us.ibm.com>
  • Loading branch information
@sykesm @yacovm
sykesm authored and yacovm committed Jan 7, 2020
1 parent 92af234 commit 571842f
Showing 1 changed file with 4 additions and 0 deletions.
4 changes: 4 additions & 0 deletions integration/raft/cft_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -411,6 +411,9 @@ var _ = Describe("EndToEnd Crash Fault Tolerance", func() {
o1Proc.Signal(syscall.SIGTERM)
o2Proc.Signal(syscall.SIGTERM)
o3Proc.Signal(syscall.SIGTERM)
Eventually(o1Proc.Wait(), network.EventuallyTimeout).Should(Receive())
Eventually(o2Proc.Wait(), network.EventuallyTimeout).Should(Receive())
Eventually(o3Proc.Wait(), network.EventuallyTimeout).Should(Receive())

By("Launching orderers again")
o1Runner = network.OrdererRunner(o1)
Expand Down Expand Up @@ -538,6 +541,7 @@ var _ = Describe("EndToEnd Crash Fault Tolerance", func() {

By("Killing orderer")
ordererProc.Signal(syscall.SIGTERM)
Eventually(ordererProc.Wait(), network.EventuallyTimeout).Should(Receive())

By("Launching orderers again")
runner = network.OrdererRunner(orderer)
Expand Down

0 comments on commit 571842f

Please sign in to comment.