kubelet: Fix racy kubelet tests.

[yifan-gu]

Add fakePodWorkders to run syncPod() in serial for testing.

Fix #7788

//cc @yujuhong @vmarmol @dchen1107

[yujuhong]

Thanks @yifan-gu for the PR! Looks good overall.

My only concern is that we would be testing the behaviors of the pod workers in kubelet_tests.go without using the actually pod workers. The fake pod_worker may eventually deviate from the actually implementation as time goes by. Ideally, we would want to move away from examining all docker calls in details in kubelet_test.go, but this may not be done for a while.

Maybe we can add one or two tests for the fake pod worker to verify that it's issuing the same docker calls as the actual pod worker? What do you think about it? We can do this in a followup PR, or if you are busy with other things, feel free to file an issue. I can grab it when I have time :)

[dchen1107]

LGTM overall. I am ok to merge the pr with TODOs based on @yujuhong's comments. Thanks!

[yifan-gu]

@yujuhong Yes I see your concern. I think we can add tests for the fake one to verify it invokes syncPodFn for the same number of times, and with the same args as the real one.
I can do that in this PR.

[yifan-gu]

Tests added, PTAL, thanks! @yujuhong @dchen1107

[yujuhong]

LGTM. Thanks!

[yujuhong]

@yifan-gu, did you kick off shippable/travis build or are they just taking forever to finish?

[yifan-gu]

@yujuhong NM, I just noticed there is a build failure by my fault, so I fixed it and rebased. Wait for travis now :)

[yujuhong]

Flake in one of the shippable build, but the other build passed. Merging.

[yifan-gu]

@yujuhong What flakes? It that related?

[yujuhong]

Nope. It's the good old pod status not being reported in time bug #6651.