e2e conformance tests no longer run as documented in 1.5.0, regression from 1.4.x

[spiffxp]

Is this a request for help? no

What keywords did you search in Kubernetes issues before filing this one? e2e, conformance

Is this a BUG REPORT or FEATURE REQUEST? BUG REPORT

Kubernetes version (use kubectl version): 1.5.0, 1.5.1

Environment: 1.5.0 GKE cluster, OSX 10.11.x workstation

What happened: the documented approach of running conformance tests doesn't work with 1.5.1 release artifacts, but does work with 1.4.6

aaron@cloudtop:kubernetes-1.5.1$ GINKGO_PARALLEL=y go run hack/e2e.go -v --test --test_args="--ginkgo.focus=\[Conformance\] --ginkgo.skip=\[Serial\]"
2016/12/15 10:41:53 e2e.go:722: Running: kubectl version
It looks as if you don't have a compiled kubectl binary

What you expected to happen:

aaron@cloudtop:kubernetes-1.4.6$ GINKGO_PARALLEL=y go run hack/e2e.go -v --test --test_args="--ginkgo.focus=\[Conformance\] --ginkgo.skip=\[Serial\]"
2016/12/15 10:43:08 e2e.go:449: Running: get status
Skeleton Provider: prepare-e2e not implemented
Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.6", GitCommit:"e569a27d02001e343cb68086bc06d47804f62af6", GitTreeState:"clean", BuildDate:"2016-11-12T05:22:15Z", GoVersion:"go1.7.1", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.6", GitCommit:"e569a27d02001e343cb68086bc06d47804f62af6", GitTreeState:"clean", BuildDate:"2016-11-12T05:16:27Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
2016/12/15 10:43:09 e2e.go:451: Step 'get status' finished in 592.839581ms

How to reproduce it (as minimally and precisely as possible):

Attempt to run conformance tests (as documented at https://github.com/kubernetes/community/blob/master/contributors/devel/e2e-tests.md#conformance-tests) from a directory that contains the extracted contents of the v1.5.1 kubernetes.tar.gz and kubernetes-test.tar.gz artifacts. Have an alternate kubectl in your PATH

Anything else do we need to know:

[spiffxp]

/cc @kubernetes/sig-testing

[ixdy]

This sounds like it might be related to #38725. kubernetes.tar.gz no longer includes server or client binaries, which have to be downloaded separately.

[spiffxp]

@ixdy yup, that's it... e2e.go and its related scripts either need to look at PATH implicitly, or fetch the client binaries

[ixdy]

@spiffxp alternately, change the instructions to "download and extract kubernetes.tar.gz, then run DOWNLOAD_TESTS_TAR=y cluster/get-kube-binaries.sh"

[spiffxp]

ref #35737, /cc @david-mcmahon @saad-ali @pwittrock

I'm less than pleased that this didn't get flagged as "action required" or "notable change" in the release notes. What confuses me is the referenced PR's release note of

"Arch-specific server and client binaries are no longer included in kubernetes.tar.gz. You must manually download the necessary architecture-specific tarballs or use kubernetes/cluster/get-kube-binaries.sh to download them for you."

didn't get included in CHANGELOG.md. The innocuous PR title of "stop including arch-specific binaries" was used instead

[david-mcmahon]

@spiffxp Looks like when the template was edited, a space was added between the block start and the release-note text. kubernetes/release#233 should account for such subtle changes in the future.

[spiffxp]

/sig testing
mumbles something about kubetest but I guess I should see if there is a way to run tests "as documented" on 1.5.x before closing this

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[spiffxp]

/close
This has aged out, as we no longer support kuberentes < 1.6.x