Losing HighestBefore events on shutdown (Vector clock error)

[thaarok]

vecmt.Index wraps the epoch store database into VecFlushable wrapper, which postpones all writes until the Flush() call.

Such behavior is correct. (It allows to revert invalid events insertion - only the correct ones are flushed in the saveAndProcessEvent method.)

However VecFlushable in lachesis-base does not flush directly into the backing database, it flushes into its in-memory backedMap, which is being flushed to the backing on-disk database only when the map size exceeds IndexConfig.Caches.DBCache (around 10MiB). This leads to keeping the "HighestBefore" only in the memory even when Opera shuts down, which sometimes leads to following error on the next Opera start:

Vector clock error  err: Event A=2:2:875e5a not found

It seems this occur mostly (or maybe only) when "pebble-flg" instead of "pebble-fsh" is being used - probably the fsh transaction avoid writing other db records of the event, so this error does not occur.

In go-opera-norma I have fixed this by creating a new VecFlushable, which flushes directly to the backing database.

[thaarok]

Stacktrace of the panic for reference:

panic: panic of Vector clock: Event A=2:2:875e5a not found

goroutine 1 [running]:
github.com/Fantom-foundation/go-opera/integration.rawMakeEngine.panics.func1({0x1c8f8c0?, 0xc00562bf40})
	/home/jkalina/Fantom/go-opera-norma/integration/assembly.go:56 +0xa5
github.com/Fantom-foundation/lachesis-base/vecfc.(*Index).forklessCause(0xc000140120, {0x0, 0x0, 0x0, 0x2, 0x0, 0x0, 0x0, 0x2, 0x87, ...}, ...)
	/home/jkalina/Fantom/lachesis-base/vecfc/forkless_cause.go:45 +0x430
github.com/Fantom-foundation/lachesis-base/vecfc.(*Index).ForklessCause(0xc000140120, {0x0, 0x0, 0x0, 0x2, 0x0, 0x0, 0x0, 0x2, 0x87, ...}, ...)
	/home/jkalina/Fantom/lachesis-base/vecfc/forkless_cause.go:35 +0x136
github.com/Fantom-foundation/lachesis-base/abft/election.(*Election).observedRootsMap(0xc0053fc990, {0x0, 0x0, 0x0, 0x2, 0x0, 0x0, 0x0, 0x2, 0x87, ...}, ...)
	/home/jkalina/Fantom/lachesis-base/abft/election/election.go:119 +0xf9
github.com/Fantom-foundation/lachesis-base/abft/election.(*Election).ProcessRoot(0xc0053fc990, {{0x0, 0x0, 0x0, 0x2, 0x0, 0x0, 0x0, 0x2, 0x87, ...}, ...})
	/home/jkalina/Fantom/lachesis-base/abft/election/election_math.go:34 +0x37f
github.com/Fantom-foundation/lachesis-base/abft.(*Orderer).processKnownRoots(0xc005704370)
	/home/jkalina/Fantom/lachesis-base/abft/event_processing.go:133 +0xcc
github.com/Fantom-foundation/lachesis-base/abft.(*Orderer).bootstrapElection(0xc0054d56b0?)
	/home/jkalina/Fantom/lachesis-base/abft/event_processing.go:104 +0x25
github.com/Fantom-foundation/lachesis-base/abft.(*Orderer).Bootstrap(0xc005704370, {0xc00562be60?, 0x0?})
	/home/jkalina/Fantom/lachesis-base/abft/bootstrap.go:53 +0x249
github.com/Fantom-foundation/lachesis-base/abft.(*Lachesis).BootstrapWithOrderer(...)
	/home/jkalina/Fantom/lachesis-base/abft/lachesis.go:92
github.com/Fantom-foundation/lachesis-base/abft.(*Lachesis).Bootstrap(0xc0003b2140, {0x64?})
	/home/jkalina/Fantom/lachesis-base/abft/lachesis.go:88 +0x65
github.com/Fantom-foundation/go-opera/cmd/opera/launcher.makeNode(0xc0054046e0, 0xc005426000, 0x0)
	/home/jkalina/Fantom/go-opera-norma/cmd/opera/launcher/launcher.go:367 +0xe7c
github.com/Fantom-foundation/go-opera/cmd/opera/launcher.lachesisMain(0xc0054046e0)
	/home/jkalina/Fantom/go-opera-norma/cmd/opera/launcher/launcher.go:284 +0xba
gopkg.in/urfave/cli%2ev1.HandleAction({0x16ce6a0?, 0x1af6f80?}, 0xc00540e000?)
	/home/jkalina/go/pkg/mod/gopkg.in/urfave/cli.v1@v1.20.0/app.go:490 +0x50
gopkg.in/urfave/cli%2ev1.(*App).Run(0xc000a82820, {0xc0000401e0, 0x6, 0x6})
	/home/jkalina/go/pkg/mod/gopkg.in/urfave/cli.v1@v1.20.0/app.go:264 +0x6be
github.com/Fantom-foundation/go-opera/cmd/opera/launcher.Launch(...)
	/home/jkalina/Fantom/go-opera-norma/cmd/opera/launcher/launcher.go:263
main.main()
	/home/jkalina/Fantom/go-opera-norma/cmd/opera/main.go:11 +0x2f

Note: the err occurs during the engine.Bootstrap phase of MakeNode.