Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testing: Delay in test causes timeouts in tests that follow it. #11106

Closed
eZanmoto opened this issue Jun 6, 2015 · 1 comment

Comments

Projects
None yet
3 participants
@eZanmoto
Copy link

commented Jun 6, 2015

1 go version: go version go1.4.2 linux/amd64

2
OS: Linux Mint 16
uname -a: Linux mint 3.11.0-12-generic #19-Ubuntu SMP Wed Oct 9 16:20:46 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

3 Ran go test with the following test file:

package p

import (
    "net"
    "testing"
    "time"
)

func Test1(t *testing.T) {
    testTunnelWithinSecond(t, time.Millisecond)
}

func Test2(t *testing.T) {
    testTunnelWithinSecond(t, time.Millisecond)
}

func Test3(t *testing.T) {
    testTunnelWithinSecond(t, 0)
}

func Test4(t *testing.T) {
    testTunnelWithinSecond(t, time.Millisecond)
}

func Test5(t *testing.T) {
    testTunnelWithinSecond(t, time.Millisecond)
}

func testTunnelWithinSecond(t *testing.T, timeout time.Duration) {
    ch := make(chan struct{}, 1)

    go func() {
        in, err := pipeConn()
        if err != nil {
            t.Fatalf("Unexpected error: %v", err)
        }

        for err == nil {
            in[1].SetReadDeadline(time.Now().Add(timeout))
            _, err = in[1].Read(make([]byte, 0))
            if e, ok := err.(net.Error); ok && e.Timeout() {
                err = nil
                continue
            }
        }

        close(ch)
    }()

    select {
    case <-time.After(time.Millisecond):
        t.Fatalf("Test timed out")
    case <-ch:
    }
}

// `pipeConn` returns a pair of `net.Conn`s that are connected to each other.
func pipeConn() ([2]net.Conn, error) {
    errs := make(chan error, 1)
    ports := make(chan int, 1)
    conns := make(chan net.Conn, 1)

    go func() {
        ln, err := net.ListenTCP("tcp4", &net.TCPAddr{IP: localhost})
        if err != nil {
            errs <- err
        } else {
            ports <- ln.Addr().(*net.TCPAddr).Port

            conn, err := ln.Accept()
            conns <- conn
            errs <- err
        }
    }()

    select {
    case err := <-errs:
        return [2]net.Conn{nil, nil}, err
    case port := <-ports:
        conn, _ := net.DialTCP(
            "tcp4",
            nil,
            &net.TCPAddr{IP: localhost, Port: port},
        )
        return [2]net.Conn{conn, <-conns}, <-errs
    }
}

var localhost = net.IPv4(127, 0, 0, 1)

4 I expected all tests to pass except for Test3, which I expected to fail with "Test timed out".

5 Test1 and Test2 always passed, but Test4 and Test5 intermittently (but with high probability) failed with "Test timed out". The following script was run to break down the frequency of failures:

import collections
import pprint
import re
import subprocess

output = collections.defaultdict(int)
for i in xrange(100):
    try:
        subprocess.check_output(["go", "test"])
    except subprocess.CalledProcessError, e:
        output[re.sub("0\.\d*s", "", e.output)] += 1

for k, v in output.items():
    print "%d%%" % v
    print '\t' + k.replace('\n', '\n\t')

The above script gave the following results on a sample run, providing preliminary evidence to the presence of cross-contamination of tests.

18%
    --- FAIL: Test3 ()
        tunnel_test.go:52: Test timed out
    --- FAIL: Test4 ()
        tunnel_test.go:52: Test timed out
    FAIL
    exit status 1
    FAIL    bitbucket.com/ezanmoto/repogate/lab 

69%
    --- FAIL: Test3 ()
        tunnel_test.go:52: Test timed out
    --- FAIL: Test4 ()
        tunnel_test.go:52: Test timed out
    --- FAIL: Test5 ()
        tunnel_test.go:52: Test timed out
    FAIL
    exit status 1
    FAIL    bitbucket.com/ezanmoto/repogate/lab 

1%
    --- FAIL: Test3 ()
        tunnel_test.go:52: Test timed out
    FAIL
    exit status 1
    FAIL    bitbucket.com/ezanmoto/repogate/lab 

12%
    --- FAIL: Test3 ()
        tunnel_test.go:52: Test timed out
    --- FAIL: Test5 ()
        tunnel_test.go:52: Test timed out
    FAIL
    exit status 1
    FAIL    bitbucket.com/ezanmoto/repogate/lab 
@minux

This comment has been minimized.

Copy link
Member

commented Jun 7, 2015

@minux minux closed this Jun 7, 2015

@golang golang locked and limited conversation to collaborators Jun 25, 2016

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.