Integration tests failing due to order of tests

[ccdle12]

Background

While working on a PR found that integration tests were failing, specifically the position of testRevokedCloseRetribution causes testRevokedCloseRetributionZeroValueRemoteOutput to fail in lnd_test.go

Your environment

• lnd version 0.5.0-beta commit=v0.5-beta-301-g3ff6055a622b10ea13ac266d5e6d61a770501462-dirty
• OSX
• btcd

Steps to reproduce

make itest

Result:

  --- PASS: TestLightningNetworkDaemon/revoked_uncooperative_close_retribution (16.18s)
    --- PASS: TestLightningNetworkDaemon/failing_link (6.46s)
    --- PASS: TestLightningNetworkDaemon/garbage_collect_link_nodes (64.94s)
    --- PASS: TestLightningNetworkDaemon/abandonchannel (0.45s)
    --- FAIL: TestLightningNetworkDaemon/revoked_uncooperative_close_retribution_zero_value_remote_output (34.79s)
        lnd_test.go:74: Failed: (revoked uncooperative close retribution zero value remote output): exited with error:
            *errors.errorString unable to find Carol's force close tx in mempool: wanted 1, found 2 txs in mempool: [a5f1345c7de7ed26da6f5958d741f786190aaeee0b965e8989d98a8e2b371556 fc6ac77e008821e245dc2b707a8cc6803ea6a2ea5c7ef391b52620fd73e6199a]
            /Users/christophercoverdale/go/src/github.com/lightningnetwork/lnd/lnd_test.go:6417 (0x17adf7a)
                testRevokedCloseRetributionZeroValueRemoteOutput: t.Fatalf("unable to find Carol's force close tx in mempool: %v",
            /Users/christophercoverdale/go/src/github.com/lightningnetwork/lnd/lnd_test.go:99 (0x1788bf6)
                (*harnessTest).RunTestCase: testCase.test(net, h)
            /Users/christophercoverdale/go/src/github.com/lightningnetwork/lnd/lnd_test.go:12591 (0x17e1d54)
                TestLightningNetworkDaemon.func3: ht.RunTestCase(testCase, lndHarness)
            /usr/local/Cellar/go/1.11.1/libexec/src/testing/testing.go:827 (0x10f590f)
                tRunner: fn(t)
            /usr/local/Cellar/go/1.11.1/libexec/src/runtime/asm_amd64.s:1333 (0x105dcb1)
                goexit: BYTE    $0x90   // NOP
FAIL
exit status 1
FAIL    github.com/lightningnetwork/lnd 573.513s
make: *** [itest] Error 1

Original order of tests:

	{
		// TODO(roasbeef): test always needs to be last as Bob's state
		// is borked since we trick him into attempting to cheat Alice?
		name: "revoked uncooperative close retribution",
		test: testRevokedCloseRetribution,
	},
	{
		name: "failing link",
		test: testFailingChannel,
	},
	{
		name: "garbage collect link nodes",
		test: testGarbageCollectLinkNodes,
	},
	{
		name: "abandonchannel",
		test: testAbandonChannel,
	},
	{
		name: "revoked uncooperative close retribution zero value remote output",
		test: testRevokedCloseRetributionZeroValueRemoteOutput,
	},
	{
		name: "revoked uncooperative close retribution remote hodl",
		test: testRevokedCloseRetributionRemoteHodl,
	},
	{
		name: "data loss protection",
		test: testDataLossProtection,
	},
	{
		name: "query routes",
		test: testQueryRoutes,
	},
	{
		name: "route fee cutoff",
		test: testRouteFeeCutoff,
	},
	{
		name: "send update disable channel",
		test: testSendUpdateDisableChannel,
	},

Moving testRevokedCloseRetribution to end of tests slice causes all the tests to pass:

	{
		name: "failing link",
		test: testFailingChannel,
	},
	{
		name: "garbage collect link nodes",
		test: testGarbageCollectLinkNodes,
	},
	{
		name: "abandonchannel",
		test: testAbandonChannel,
	},
	{
		name: "revoked uncooperative close retribution zero value remote output",
		test: testRevokedCloseRetributionZeroValueRemoteOutput,
	},
	{
		name: "revoked uncooperative close retribution remote hodl",
		test: testRevokedCloseRetributionRemoteHodl,
	},
	{
		name: "data loss protection",
		test: testDataLossProtection,
	},
	{
		name: "query routes",
		test: testQueryRoutes,
	},
	{
		name: "route fee cutoff",
		test: testRouteFeeCutoff,
	},
	{
		name: "send update disable channel",
		test: testSendUpdateDisableChannel,
	},
	{
		// TODO(roasbeef): test always needs to be last as Bob's state
		// is borked since we trick him into attempting to cheat Alice?
		name: "revoked uncooperative close retribution",
		test: testRevokedCloseRetribution,
	},

result:

    --- PASS: TestLightningNetworkDaemon/failing_link (7.01s)
    --- PASS: TestLightningNetworkDaemon/garbage_collect_link_nodes (68.78s)
    --- PASS: TestLightningNetworkDaemon/abandonchannel (0.47s)
    --- PASS: TestLightningNetworkDaemon/revoked_uncooperative_close_retribution_zero_value_remote_output (20.25s)
    --- PASS: TestLightningNetworkDaemon/revoked_uncooperative_close_retribution_remote_hodl (21.16s)
    --- PASS: TestLightningNetworkDaemon/data_loss_protection (17.69s)
    --- PASS: TestLightningNetworkDaemon/query_routes (17.33s)
    --- PASS: TestLightningNetworkDaemon/route_fee_cutoff (14.20s)
    --- PASS: TestLightningNetworkDaemon/send_update_disable_channel (23.74s)
    --- PASS: TestLightningNetworkDaemon/revoked_uncooperative_close_retribution (15.52s)
PASS
ok      github.com/lightningnetwork/lnd 668.665s

Expected behaviour

Test should all pass, preferably regardless of order.

Actual behaviour

Tests fail with original order of tests in lnd_test.go.

By moving testRevokedCloseRetribution to the end of the test slice the tests all pass.

[halseth]

This will be fixed by #2153

[ccdle12]

Thanks @halseth, I'll continue working on a PR with that mind :)