Support e2e testing via a bastion host for SSH commands #72286

smarterclayton · Dec 21, 2018

Issue: e2e tests should succeed even if nodes are not publicly available from where the tests are run

Until we port the remaining e2e tests to use exec host pods instead of SSH, it should be possible to use bastion host when clusters don't expose public IPs for nodes. Testers can use the KUBE_SSH_BASTION environment variable set to a host:port to specify the host that the tests will use as a jump host.

If KUBE_SSH_BASTION is specified, the test will first connect to the bastion using SSH and the provider private key, then establish a tunnel from the bastion to the target node by its IP. If the node has an external IP, the external IP must be routable from the bastion.

Also support KUBE_SSH_KEY_PATH so there is a single cross-provider env var for setting the full path to the private key, which simplifies testing multiple clouds from the same infra.

/kind bug

e2e tests that require SSH may be used against clusters that have nodes without external IP addresses by setting the environment variable `KUBE_SSH_BASTION` to the `host:port` of a machine that is allowed to SSH to those nodes.  The same private key that the test would use is used for the bastion host.  The test connects to the bastion and then tunnels another SSH connection to the node.

k8s-ci-robot · Dec 21, 2018

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: smarterclayton

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~test/OWNERS~~ [smarterclayton]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

smarterclayton · Dec 21, 2018

@kubernetes/sig-testing-pr-reviews we have configurations that have private nodes and I would like to ensure that e2e tests running as prow jobs on a k8s cluster can reach into those other clusters to do the full e2e suite

smarterclayton · Dec 21, 2018

/retest

smarterclayton · Dec 22, 2018

/retest

smarterclayton · Jan 10, 2019

This is a more complete solution for #68792

smarterclayton · Jan 11, 2019

/retest

…

On Thu, Jan 10, 2019 at 6:10 PM Kubernetes Prow Robot < ***@***.***> wrote: @smarterclayton <https://github.com/smarterclayton>: The following tests *failed*, say /retest to rerun them all: Test name Commit Details Rerun command pull-kubernetes-integration 88cac12 <88cac12> link <https://gubernator.k8s.io/build/kubernetes-jenkins/pr-logs/pull/72286/pull-kubernetes-integration/40880/> /test pull-kubernetes-integration pull-kubernetes-e2e-kops-aws 88cac12 <88cac12> link <https://gubernator.k8s.io/build/kubernetes-jenkins/pr-logs/pull/72286/pull-kubernetes-e2e-kops-aws/118909/> /test pull-kubernetes-e2e-kops-aws Full PR test history <https://gubernator.k8s.io/pr/72286>. Your PR dashboard <https://gubernator.k8s.io/pr/smarterclayton>. Please help us cut down on flakes by linking to <https://git.k8s.io/community/contributors/devel/flaky-tests.md#filing-issues-for-flaky-tests> an open issue <https://github.com/kubernetes/kubernetes/issues?q=is:issue+is:open> when you hit one in your PR. Instructions for interacting with me using PR comments are available here <https://git.k8s.io/community/contributors/guide/pull-requests.md>. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra <https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:> repository. I understand the commands that are listed here <https://go.k8s.io/bot-commands>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#72286 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG_p-eq2s_hsDkz7yzPovpGu_ezIX8uks5vB8h8gaJpZM4ZfGrH> .

smarterclayton · Jan 11, 2019

/retest

smarterclayton · Jan 16, 2019

@kubernetes/sig-node-pr-reviews need some eyes on this - nodes shouldn't be required to have external SSH access, and I think we're starting to get to the point where we shouldn't assume the e2e tests run from inside the security zone of a cluster. A bastion is reasonably simple with go-ssh (have been running this code in openshift for a while) and the code path is isolated so that if someone does hit bugs in this new path it doesn't impact others.

rphillips · Jan 18, 2019

rphillips reviewed Jan 18, 2019

View changes

rphillips · Jan 18, 2019

test/e2e/framework/ssh.go

+	}
+	bastionClient, err := ssh.Dial("tcp", bastion, config)
+	if err != nil {
+		err = wait.Poll(5*time.Second, 20*time.Second, func() (bool, error) {


would be nice to have the constants at the top of the file

the code in this case is a copy of sshutil.RunSSHCommand and I'm trying to keep the structure of the method consistent (can't reuse the constant across two packages since we don't want to expose it)

rphillips · Jan 18, 2019

rphillips reviewed Jan 18, 2019

View changes

rphillips · Jan 18, 2019

test/e2e/framework/ssh.go

+			err = fmt.Errorf("failed running `%s` on %s@%s: '%v'", cmd, user, host, err)
+		}
+	}
+	return bout.String(), berr.String(), code, err


is it ok to buffer the entire stdout/stderr into memory? We may want to put the data into a limited buffer.

This is no different than the other SSH stub, but generally yes, this is for test code only and the usage is constraint. We also dump the full output to the build logs, which is a disincentive to use this for a chatty channel.

So it's not great, but it's consistent with our existing flow, but in the future we should improve this structure (these are convenience commands that aren't really designed for anything other than quick and dirty testing).

rphillips · Jan 18, 2019

rphillips reviewed Jan 18, 2019

View changes

rphillips · Jan 18, 2019

test/e2e/framework/ssh.go

+
+func IssueSSHCommand(cmd, provider string, node *v1.Node) error {
+	_, err := IssueSSHCommandWithResult(cmd, provider, node)
+	if err != nil {


just return err here

this was a straight move, would prefer not to change the code.

rphillips · Jan 21, 2019

/lgtm

rphillips · Jan 21, 2019

/retest

k8s-ci-robot added release-note-none kind/bug size/L cncf-cla: yes needs-sig needs-priority labels Dec 21, 2018

smarterclayton force-pushed the smarterclayton:ssh branch from b615bf4 to 190ec3b Dec 21, 2018

k8s-ci-robot added the approved label Dec 21, 2018

k8s-ci-robot requested review from kow3ns and soltysh Dec 21, 2018

k8s-ci-robot added sig/testing and removed needs-sig labels Dec 21, 2018

smarterclayton referenced this pull request Dec 21, 2018
Merged
Support tunneling to masters over SSH for e2e tests #21700
3 of 3 tasks complete

k8s-ci-robot added release-note and removed release-note-none labels Dec 21, 2018

smarterclayton referenced this pull request Dec 24, 2018
Merged
Set KUBE_SSH_BASTION and KUBE_SSH_KEY_PATH in installer tests #2469

smarterclayton force-pushed the smarterclayton:ssh branch from 190ec3b to 8472568 Dec 27, 2018

k8s-ci-robot added the needs-rebase label Jan 10, 2019

smarterclayton force-pushed the smarterclayton:ssh branch from 8472568 to 88cac12 Jan 10, 2019

k8s-ci-robot removed the needs-rebase label Jan 10, 2019

smarterclayton referenced this pull request Jan 10, 2019
Merged
e2e: Fallback to internal IPs when using SSH in tests #68792

k8s-ci-robot added the sig/node label Jan 16, 2019

k8s-ci-robot assigned rphillips Jan 21, 2019

k8s-ci-robot added the lgtm label Jan 21, 2019

kubernetes/kubernetes

Join GitHub today

Support e2e testing via a bastion host for SSH commands #72286

Conversation

smarterclayton commented Dec 21, 2018 • edited

k8s-ci-robot added release-note-none kind/bug size/L cncf-cla: yes needs-sig needs-priority labels Dec 21, 2018

smarterclayton force-pushed the smarterclayton:ssh branch from b615bf4 to 190ec3b Dec 21, 2018

This comment has been minimized.

k8s-ci-robot commented Dec 21, 2018

k8s-ci-robot added the approved label Dec 21, 2018

k8s-ci-robot requested review from kow3ns and soltysh Dec 21, 2018

k8s-ci-robot added sig/testing and removed needs-sig labels Dec 21, 2018

smarterclayton referenced this pull request Dec 21, 2018

k8s-ci-robot added release-note and removed release-note-none labels Dec 21, 2018

This comment has been minimized.

smarterclayton commented Dec 21, 2018

This comment has been minimized.

smarterclayton commented Dec 21, 2018

This comment has been minimized.

smarterclayton commented Dec 22, 2018

smarterclayton referenced this pull request Dec 24, 2018

smarterclayton force-pushed the smarterclayton:ssh branch from 190ec3b to 8472568 Dec 27, 2018

k8s-ci-robot added the needs-rebase label Jan 10, 2019

smarterclayton added some commits Dec 21, 2018

smarterclayton force-pushed the smarterclayton:ssh branch from 8472568 to 88cac12 Jan 10, 2019

k8s-ci-robot removed the needs-rebase label Jan 10, 2019

This comment has been minimized.

smarterclayton commented Jan 10, 2019

smarterclayton referenced this pull request Jan 10, 2019

This comment has been minimized.

smarterclayton commented Jan 11, 2019

This comment has been minimized.

smarterclayton commented Jan 11, 2019

This comment has been minimized.

smarterclayton commented Jan 16, 2019 • edited

k8s-ci-robot added the sig/node label Jan 16, 2019

rphillips reviewed Jan 18, 2019 View changes

This comment has been minimized.

This comment has been minimized.

rphillips reviewed Jan 18, 2019 View changes

This comment has been minimized.

This comment has been minimized.

rphillips reviewed Jan 18, 2019 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

rphillips commented Jan 21, 2019

k8s-ci-robot assigned rphillips Jan 21, 2019

k8s-ci-robot added the lgtm label Jan 21, 2019

This comment has been minimized.

rphillips commented Jan 21, 2019

Hide details View details k8s-ci-robot merged commit c336229 into kubernetes:master Jan 21, 2019 18 checks passed

18 checks passed

smarterclayton commented Dec 21, 2018 •

edited

smarterclayton force-pushed the smarterclayton:ssh branch from `b615bf4` to `190ec3b` Dec 21, 2018

smarterclayton force-pushed the smarterclayton:ssh branch from `190ec3b` to `8472568` Dec 27, 2018

smarterclayton force-pushed the smarterclayton:ssh branch from `8472568` to `88cac12` Jan 10, 2019

smarterclayton commented Jan 16, 2019 •

edited

rphillips reviewed Jan 18, 2019

View changes

rphillips reviewed Jan 18, 2019

View changes

rphillips reviewed Jan 18, 2019

View changes

k8s-ci-robot merged commit `c336229` into kubernetes:master Jan 21, 2019
18 checks passed