Wait for goroutine to finish before db close

Purge managment for private data state prefetches the information
about the data that is going to be expired in the next upcoming
block. This task is perfomed in a background goroutine. When the
ledger is closed, this task is not stopped and causes a nil pointer
panic inside goleveldb code.

This is a timing based issue and is observed in quite a few tines in
unit tests failures in CI. When the db.Close (issued by ledger close)
and db.NewIterator (issued by the background goroutine) functions are
invoked on goleveldb in parallel, this may happen that the Close function
makes the internal mem-buffers of goleveldb as nil followed by an access
to theses buffers by the function NewIterator.

As a fix, this CR makes the Close function to wait for the background
routine to finish.

FAB-11974 #done

Change-Id: I331476a6814ed2e6effcf10cc9310adb387722ff
Signed-off-by: manish <manish.sethi@gmail.com>

core/ledger/kvledger/kv_ledger_test.go

@@ -33,6 +33,7 @@ func TestMain(m *testing.M) {
 	flogging.SetModuleLevel("statevalidator", "debug")
 	flogging.SetModuleLevel("valimpl", "debug")
 	flogging.SetModuleLevel("confighistory", "debug")
+	flogging.SetModuleLevel("pvtstatepurgemgmt", "debug")
 	viper.Set("peer.fileSystemPath", "/tmp/fabric/ledgertests/kvledger")
 	viper.Set("ledger.history.enableHistoryDatabase", true)
 	os.Exit(m.Run())

core/ledger/kvledger/txmgmt/txmgr/lockbasedtxmgr/lockbased_txmgr.go

@@ -142,6 +142,9 @@ func (txmgr *LockBasedTxMgr) invokeNamespaceListeners() error {
 
 // Shutdown implements method in interface `txmgmt.TxMgr`
 func (txmgr *LockBasedTxMgr) Shutdown() {
+	// wait for background go routine to finish else the timing issue causes a nil pointer inside goleveldb code
+	// see FAB-11974
+	txmgr.pvtdataPurgeMgr.WaitForPrepareToFinish()
 	txmgr.db.Close()
 }