Diagnose GOC halt

[shaspitz]

Problem

Lots of slack discussion, this is me documenting efforts to fully diagnose why the GOC halt happened.

Closing criteria

Be able to consistently replicate the GOC bug, and confirm that a specific patch prevents the bug from occurring.

Problem details

Our best guess at what happened is a deadlock caused by writing within iterators. Unfortunately, there are a good amount of unit and e2e tests covering the suspect code. But the halt/hang only occurred in the testnet.

[shaspitz]

First step is to try out e2e, unit tests, and diff tests with a goleveldb backend

[shaspitz]

First step is to try out e2e, unit tests, and diff tests with a goleveldb backend

Tried this out in a hacky way by using

db, err := tmdb.NewGoLevelDB("testdb", "e2e")
if err != nil {
	panic(fmt.Sprintf("failed to create db: %v", err))
}

in specific_setup.go and running each e2e test independently. All throttle e2e tests passed, including one that recvs 500 packets

[jtremback]

This is done, right?

[shaspitz]

yes, will be closed by #462