Panic in TestCancelAndReadd

[enj]

TestCancelAndReadd has a race condition that causes a panic due to:

1. A very short wait-to-start-time of 100 * time.Millisecond which causes wg.Done() to be called too many times
2. Not using a fake clock such as the code in https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/util/queue/work_queue_test.go

=== RUN   TestCancelAndReadd
panic: sync: negative WaitGroup counter

goroutine 320 [running]:
panic(0x23a2620, 0xc420240440)
	/usr/local/go/src/runtime/panic.go:500 +0x1ae
sync.(*WaitGroup).Add(0xc420218230, 0xffffffffffffffff)
	/usr/local/go/src/sync/waitgroup.go:75 +0x27c
sync.(*WaitGroup).Done(0xc420218230)
	/usr/local/go/src/sync/waitgroup.go:100 +0x42
github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/node.TestCancelAndReadd.func1(0xc4202bfa40, 0x1, 0xc400000020)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/node/timed_workers_test.go:118 +0x65
github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/node.(*TimedWorkerQueue).getWrappedWorkerFunc.func1(0xc4202bfa40, 0x0, 0x0)
	github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/node/_test/_obj_test/timed_workers.go:90 +0xb6
github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/node.CreateWorker.func1()
	github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/node/_test/_obj_test/timed_workers.go:49 +0x72
created by time.goFunc
	/usr/local/go/src/time/sleep.go:154 +0x78

cc @gmarek @davidopp @timothysc @liggitt @stevekuznetsov

xref: #41311

[gmarek]

cc @smarterclayton

[erictune]

@yujuhong you wrote this test. Can you take a look?
@kubernetes/sig-node-bugs

[gmarek]

No, I think it's mine. I'll take a look tomorrow. The quickest "fix" would be to increase the timeout.

[grodrigues3]

@yujuhong we're trying to make sure all issues have associated sigs. If you remove a sig label, can you add the correct sig/label.

[gmarek]

I'll fix this tomorrow - no need to apply labels here.

[yujuhong]

@yujuhong we're trying to make sure all issues have associated sigs. If you remove a sig label, can you add the correct sig/label.

I honestly don't know what the correct sig is for controllers.

[enj]

@gmarek I have started to see this again:

openshift/origin#16071 -> https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/16071/test_pull_request_origin_unit/1869/