Launch kubemark with an existing Kubemark master #44394

rthallisey · 2017-04-12T15:16:46Z

In order to expand the use of kubemark, allow developers to use kubemark with a pre-existing Kubernetes cluster.

Ref issue #44393

k8s-reviewable · 2017-04-12T15:16:52Z

This change is

k8s-ci-robot · 2017-04-12T15:16:53Z

Hi @rthallisey. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with @k8s-bot ok to test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

rthallisey · 2017-05-09T12:58:12Z

@shyamjvs do you have any comments?

shyamjvs · 2017-05-09T13:33:14Z

@rthallisey Thank you for the PR. So I assume you intend to enable running kubemark on-premise without any particular cloud provider.
Before starting the review, what's the reason you haven't defined the skeleton functions "create-master-instance-with-resources", "delete-master-instance-and-resources" and "copy-files" (those here: https://github.com/kubernetes/kubernetes/blob/master/test/kubemark/skeleton/util.sh)
Do you want to let those functions be defined by the dev based on their environment?

rthallisey · 2017-05-09T13:42:35Z

@shyamjvs Yes. The goal would be to let the developer manage their vms/bm setup, but allow them to land a kubemark cluster on that infrastructure. So any infrastructure related functions would be handled by the user managing their infrastructure.

shyamjvs · 2017-05-09T13:45:04Z

I see you've defined ssh command, but didn't define copy-files (which is basically scp). Any reason?

rthallisey · 2017-05-09T13:54:14Z

I don't move any files around. What I do there is copy the credentials from the running Kubernetes cluster that the developer provided and pass that around to be used by the kubermark scripts.

shyamjvs · 2017-05-09T14:00:16Z

But we need a definition for this https://github.com/rthallisey/kubernetes/blob/d25d6298965aae2414540707c74859965a4d9fb0/test/kubemark/start-kubemark.sh#L115

shyamjvs

Clean and smooth @rthallisey ! Did a quick first pass. Also, have you verified if this works by running locally?

shyamjvs · 2017-05-09T13:49:24Z

cluster/pre-existing/util.sh

+
+function detect-project() {
+    if [[ -z "${MASTER_IP-}" ]]; then
+	echo "Set 'MASTER_IP' to the instance assigned to be the Kubernetes master" 1>&2


Fix indentation, here and below.

shyamjvs · 2017-05-09T13:55:36Z

cluster/pre-existing/util.sh

+    if [[ -z "${SERVICE_CLUSTER_IP_RANGE-}" ]]; then
+	octets=($(echo "${MASTER_IP}" | sed -e 's|/.*||' -e 's/\./ /g'))
+	octets[3]=0
+	cluster_range=$(echo "${octets[*]}" | sed 's/ /./g')


I guess you'd also have to add "/24" to make it a subnet.(https://github.com/kubernetes/kubernetes/blob/master/cluster/kubemark/gce/config-default.sh#L89)
Also why not assign a larger CIDR (like /16)?

I can live with the larger one.

shyamjvs · 2017-05-09T14:05:22Z

cluster/pre-existing/util.sh

+
+    if [[ -z "${SERVICE_CLUSTER_IP_RANGE-}" ]]; then
+	octets=($(echo "${MASTER_IP}" | sed -e 's|/.*||' -e 's/\./ /g'))
+	octets[3]=0


Maybe just cluster_range=$(echo "${MASTER_IP}" | awk -F '.' '{printf("%d.%d.%d.0", $1, $2, $3)}')

nice use of awk

shyamjvs · 2017-05-09T14:10:09Z

cluster/pre-existing/util.sh

+    fi
+}
+
+function run-cmd-with-retries {


Some bugs around this function's logic have been fixed recently. Please incorporate them.

shyamjvs · 2017-05-09T14:16:01Z

cluster/pre-existing/util.sh

+}
+
+function execute-cmd-on-master-with-retries() {
+  run-cmd-with-retries ssh kubernetes@"${MASTER_IP}" $@


I guess we have RETRIES="${2:-1}" before the command (so we can allow setting RETRIES to a value different from 3). For eg. https://github.com/kubernetes/kubernetes/blob/master/test/kubemark/start-kubemark.sh#L91

rthallisey · 2017-05-09T14:37:34Z

Ah I see where you're referring to. I don't define copy-files because I don't need any of those scripts since kubernetes is already running in the developer's infra. So that function just gets passed over.

rthallisey · 2017-05-09T14:43:51Z

The way I test this is by creating a 3 node kubernetes cluster with Kubeadm and pointing master_ip to the vm with the Kube master.

I tested this before I pushed, but I need to rebase and retest this since it's >20 days old.

shyamjvs · 2017-05-10T10:11:27Z

cluster/pre-existing/util.sh

@@ -0,0 +1,79 @@
+#!/bin/bash


Is there any reason why you've placed this file here? Given that it defines kubemark's cloud provider interface (the one in test/kubemark/skeleton), test/kubemark/pre-existing/util.sh seems to be the right path to put this.

Yes, the function create-certs in start-kubemark.sh needs to be overwritten. Instead of going out to the common script for its definition I define it here.

Makes sense. You can keep create-certs and detect-project here, but move the other two functions (which define kubemark interface) to test/kubemark/pre-existing/util.sh?

I'll test that out. Though this script only sources cluster scripts and not test/pre-existing/util.sh so I'm not positive it will work since creat-certs uses that func.

We would have it included through https://github.com/kubernetes/kubernetes/blob/master/test/kubemark/start-kubemark.sh#L24 in start-kubemark script.

Ok I'll add it to the next revision.

shyamjvs · 2017-05-10T11:07:42Z

The function won't get passed over. As you reach here (https://github.com/rthallisey/kubernetes/blob/d25d6298965aae2414540707c74859965a4d9fb0/test/kubemark/skeleton/util.sh#L52) and return with error. I guess you'd have to define it, at least as some empty function. And thanks for testing! But please make sure you've thoroughly done it against the new code too.

rthallisey · 2017-05-10T11:28:02Z

By passed over I mean it doesn't do anything. Sorry I should've been more specific. All it does is echo to stderr, which doesn't stop the script from running. But, I can add a blank function if that's an issue.

shyamjvs · 2017-05-10T11:50:09Z

Sorry, I was expecting it to exit with error when I defined the interface, but seems like it just stderrs (similar to cluster/skeleton). That's fine then, no need to redefine it.

rthallisey · 2017-05-10T11:58:20Z

No problem. Just want to be sure we get it right.

rthallisey · 2017-05-10T13:42:13Z

I'll update this hopefully later today. At a conference this week :)

shyamjvs · 2017-05-10T13:44:12Z

Sure.. No problem.

rthallisey · 2017-05-12T20:26:25Z

@shyamjvs ok updated this.

When testing I get the error from hollow-node logs, which makes it seem like my local kubeconfig is busted. But, I can confirm my kubeconfig works from the CLI. I'll investigate further:
hollow-node.go:100] Failed to create a ClientConfig: error while loading kubeconfig from file /kubeconfig/kubelet.kubeconfig: [pos 2811]: json: error decoding base64 binary ... illegal base64 data at input byte 1368

shyamjvs · 2017-05-12T22:22:05Z

test/kubemark/pre-existing/util.sh

+
+# Leave the skeleton definition of execute-cmd-on-master-with-retries
+# so only the pre-existing provider functions will target this.
+function execute-cmd-on-pre-existing-master-with-retries() {


I don't understand this. Why have you named the function differently?
This would mean that the line calling start-kubemark-master.sh script on the master VM (inside the start-kubemark.sh function) would simply pass without doing anything.
How would we have our kubemark master setup then?

The function execute-cmd-on-master-with-retries will be passed because I don't want to write anything to the master. I only need to read the existing cluster credentials. I commented at the bottom to further explain.

Got it, thanks. The assumption of pre-existing kubemark master wasn't clear to me earlier.

shyamjvs · 2017-05-12T22:30:39Z

cluster/pre-existing/util.sh

+    exit 1
+  fi
+
+  if [[ -z "${PROJECT-}" ]]; then


You could replace this entire if statement with PROJECT="${PROJECT:-kubemark}"

True. I'll add it to the pre-existing defaults.

Since normally detect-project sets the value of PROJECT, I was imitating that here. But, adding PROJECT="${PROJECT:-kubemark}" to config-default.sh for the pre-existing provider will accomplish the same thing.

shyamjvs · 2017-05-12T22:35:37Z

cluster/pre-existing/util.sh

+source "${KUBE_ROOT}/hack/lib/util.sh"
+
+function detect-project() {
+  if [[ -z "${MASTER_IP-}" ]]; then


Replace '-' with ':-' ? Doesn't really matter in this case. But in general we follow the convention of using latter instead of the former mostly. Their difference (as stated by bash reference):

When not performing substring expansion, using the form described below (e.g., ‘:-’), Bash tests for a parameter that is unset or null. Omitting the colon results in a test only for a parameter that is unset. Put another way, if the colon is included, the operator tests for both parameter’s existence and that its value is not null; if the colon is omitted, the operator tests only for existence.

shyamjvs · 2017-05-12T22:36:36Z

cluster/pre-existing/util.sh

+    PROJECT="kubemark"
+  fi
+
+  if [[ -z "${SERVICE_CLUSTER_IP_RANGE-}" ]]; then


Same as above. Replace with ':-'.

shyamjvs · 2017-05-12T22:38:01Z

cluster/pre-existing/util.sh

+
+# A library of helper functions for a pre-existing Kubernetes cluster
+KUBE_ROOT=$(dirname "${BASH_SOURCE}")/../..
+source "${KUBE_ROOT}/cluster/common.sh"


Add newline before this.

shyamjvs · 2017-05-12T22:39:23Z

cluster/pre-existing/util.sh

+# limitations under the License.
+
+# A library of helper functions for a pre-existing Kubernetes cluster
+KUBE_ROOT=$(dirname "${BASH_SOURCE}")/../..


Add newline before this.

shyamjvs · 2017-05-12T22:42:19Z

cluster/pre-existing/util.sh

+  fi
+
+  if [[ -z "${PROJECT-}" ]]; then
+    PROJECT="kubemark"


IIRC this is used later in start-kubemark.sh for pushing the kubemark image to ${REGISTRY}/${PROJECT}/kubemark. Assuming some default value might be a bit too vague.

shyamjvs · 2017-05-12T22:54:08Z

cluster/kubemark/pre-existing/config-default.sh

+
+# Pre-existing cluster expects a MASTER_IP
+MASTER_IP="${MASTER_IP:-}"
+MASTER_IP_AND_PORT="${MASTER_IP_AND_PORT:-$MASTER_IP:6443}"


6443 is the default secure port anyway. Why do you want to add this port?
Unless you want to override the port, you can just let it stay as $MASTER_IP.

I set it to 6443 with the assumption of an existing Kubernetes cluster running that's using that port. This could be solved with the user setting MASTER_IP to include the port. So I suppose documentation here could be an alternative.

clusters: - name: kubemark cluster: certificate-authority-data: "${CA_CERT_BASE64}" server: https://${MASTER_IP_AND_PORT}

We were not having port as part of the kubeconfig earlier also. We just passed the IP address in the config and IIUC the port is defaulted. Not sure if hardcoding the default number (when it automatically happens) is a good idea. WDYT?

Is it really 6443? Isn't it 443?

Seems like it says 6443 here (https://github.com/kubernetes/apiserver/blob/master/pkg/server/config.go#L157) which is what also shows up in the documentation:
--secure-port int The port on which to serve HTTPS with authentication and authorization. If 0, don't serve HTTPS at all. (default 6443)

But actually the value being used is 443 (https://github.com/kubernetes/apiserver/blob/master/pkg/server/config.go#L199).
Guess it's a bug, will send a PR fixing this.

Seems like I was looking at the wrong place. It's 6443 here (https://github.com/kubernetes/kubernetes/blob/master/pkg/kubeapiserver/options/serving.go#L39).
I'm not sure I understand why we have this difference across the repos.

I believe 443 is the default. 6443 is the default secure port.

shyamjvs · 2017-05-12T23:06:51Z

test/kubemark/pre-existing/util.sh

+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+function run-cmd-with-retries {


It might be good to add a comment here saying where the logic of this function was copied from. So we can keep a track of it and also understand the weird-ish things we do inside it.

Why was it copied?
If you can't source the file where it is defined for some reason, we should refactor it to a separate file and source only that.

I'll add a common script to source. 👍

shyamjvs · 2017-05-12T23:15:43Z

test/kubemark/start-kubemark.sh

@@ -370,6 +370,7 @@ echo -e "${color_yellow}STARTING SETUP FOR MASTER${color_norm}"
 find-release-tars
 create-master-environment-file
 create-master-instance-with-resources
+MASTER_IP_AND_PORT="${MASTER_IP_AND_PORT:-$MASTER_IP}"


This is redundant. You're doing it inside the config-default.sh anyway.

Since I changed a few variables from MASTER_IP to MASTER_IP_AND_PORT in start-kubemark.sh, MASTER_IP_AND_PORT will need to be set to MASTER_IP in the case that the dev is not using the pre-existing provider. That's what this does. But I could also do what I mentioned above. I could document when using the pre-existing provider for the dev to include the port into the MASTER_IP variable.

Aah I see. Makes sense.

shyamjvs · 2017-05-12T23:28:21Z

@rthallisey Thanks for addressing the comments. One question I have is are you assuming your kubemark master VM to be all setup already (with the master components up and running)?

And regarding:

When testing I get the error from hollow-node logs, which makes it seem like my local kubeconfig is busted. But, I can confirm my kubeconfig works from the CLI. I'll investigate further:

You can try wrting the kubelet's kubeconfig (https://github.com/rthallisey/kubernetes/blob/9214e9edb415f225102dea9f7ba65caa7a741c54/test/kubemark/start-kubemark.sh#L208) to a file and then see what's wrong with it. Seems like some error with parsing json.

rthallisey · 2017-05-15T12:37:52Z

@shyamjvs I'm assuming the the developer has setup their infrastructure and deployed a Kubernetes cluster on it. That's why I'm not copying anything to the master and not ssh'ing to the master to do anything except gather the credentials of the existing Kubernetes cluster.

shyamjvs · 2017-05-15T13:07:20Z

@shyamjvs I'm assuming the the developer has setup their infrastructure and deployed a Kubernetes cluster on it.

This assumption is fine if you're talking just about the base cluster. But assuming that you've somehow magically configured the kubemark-master I feel is not apt. IMO you should make only the simplest assumption which is you have an SSH'able machine (identified by MASTER_IP) available to start your "kubemark"-master on it. Then you call start-kubemark.sh which does all the work needed to be done on that VM.

rthallisey · 2017-05-15T13:30:37Z

@shyamjvs I think I am making a pretty simple assumption. The Kubernetes Master exists at MASTER_IP. The only requirement from the kubemark side is gathering those credentials from the master and setting them correctly. This change isn't large, but it adds a ton of flexibility to the kubermark scripts because it provides a DIY option (do it yourself). Whereas the developer can build and test in an environment that's not currently supported without adding new code to the scripts.

shyamjvs · 2017-05-15T14:05:20Z

Whereas the developer can build and test in an environment that's not currently supported without adding new code to the scripts.

@rthallisey SG. That makes it clearer for me now :) However, the reason why I was saying about assuming just the VM is that it enables people not using any cloud provider to startup a kubemark cluster with just the VM available (for eg. on-prem setups). Besides, not everyone would have enough knowledge to start their own custom kubernetes master. So this would do all the magic for them. Anyway, we can probably let this be the same for now, and make this assumption pluggable in future.

Also @wojtek-t / @gmarek , if you have time could you give your opinion/feedback on this PR?

wojtek-t · 2017-05-26T07:33:14Z

test/kubemark/gce/util.sh

@@ -14,35 +14,13 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

+source test/kubemark/common/util.sh


If this is new file that you added, then you didn't "git add" it to this PR.

wojtek-t · 2017-05-26T07:34:24Z

test/kubemark/gce/util.sh

@@ -14,35 +14,13 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

+source test/kubemark/common/util.sh
+


Another thing is that please use the convention of sourcing files, e.g. like here:
https://github.com/kubernetes/kubernetes/pull/44394/files#diff-a705eb8c1cb0096615ba0ac989e939f1R21

wojtek-t · 2017-05-26T07:35:07Z

test/kubemark/pre-existing/README.md

+
+**Kubernetes Cluster**
+- A real Kubernetes Cluster that has master and minions. The hollow-node pods
+  are run in this cluster, but appear as nodes to the Kubemark Master


s/Master/cluster/

Wouldn't this be referred to as a cluster? Since it's likely you'll be running many hollow node pods, I'd expect that to require many real nodes. Is the purpose of your comment to make the syntax match the definition above it?

wojtek-t · 2017-05-26T07:35:29Z

test/kubemark/pre-existing/README.md

+# Kubemark Pre-existing Provider Guide
+
+**Kubemark Master**
+- A Kubernetes master running in a VM that will be used as the kubemark master


A set of control plance components running in a VM ....

wojtek-t · 2017-05-26T07:36:27Z

test/kubemark/pre-existing/README.md

+
+## Use Case
+
+The goal of the pre-existing provider is to use the kubemark tools with a


wojtek-t · 2017-05-26T07:37:50Z

test/kubemark/pre-existing/README.md

+KUBEMARK_IMAGE_MAKE_TARGET="push"
+CONTAINER_REGISTRY=docker.io
+PROJECT="rthallisey"
+KUBEMARK_IMAGE_MAKE_TARGET="push"


What's that?

looks like I had a duplicate

wojtek-t · 2017-05-26T07:40:12Z

test/kubemark/pre-existing/README.md

+The all in one Kubemark cluster looks like the following:
+ 1) A running Kubernetes cluster pointed to by the local kubeconfig
+ 2) Some hollow-nodes that run on the Kubernetes Cluster from #1
+ 3) The hollow-nodes are configured to talk with the master at #1


I don't think this setup makes any sense.
The setup I was trying to refer to in one of my comments was:
"kubemark master components (kubemark apiserver, kubemark scheduler, ...) are also running as pods in the base cluster".

The setup you describe is not only discouraged - it just doesn't make any sense to me...

I agree this setup may not be 100% practical for testing, but it's possible to do. Therefore, I figured I would document it anyway but provide a warning message.

In my opinion it's not that it's not practical for testing. It's just wrong.

In this setup you don't even have a control over how many nodes will be in the cluster. Because depending whether your hollow-node will be scheduled on a real node or on a different "hollow node" it will either be a real node, or not-running pod.

This is just bad and asking for troubles. We should really point that it's wrong. Not just discourage.

@gmarek FYI

+1 to @wojtek-t .
Reusing the real master as kubemark master is not a good idea. In my earlier comments I was seeking clarity about it and thought we agreed upon the same thing.(#44394 (comment)
#44394 (comment))

Sure I agree. I'll make the comment a stronger warning.

@rthallisey - it's not about making a comment. We shouldn't even mention such thing. We should remove this second at all.

If you want to mention a different setup, the only other reasonable one is that we don't create an additional VM for master control plane components, but we are running them as pods on this bas kubernetes cluster.

@wojtek-t 👍 That works for me. I misunderstood your last comment. Removing...

wojtek-t · 2017-05-26T07:40:50Z

test/kubemark/pre-existing/util.sh

+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+source test/kubemark/common/util.sh


No relative paths please - please source e.g. as here:
https://github.com/kubernetes/kubernetes/pull/44394/files#diff-a705eb8c1cb0096615ba0ac989e939f1R23

wojtek-t · 2017-05-26T07:42:09Z

cluster/kubemark/pre-existing/config-default.sh

+#
+MASTER_IP="${MASTER_IP:-}"
+
+# The container name and registry given to the kubemark container:


There is no container name below - it's just project and container registry.

Agreed, it should be project name.

wojtek-t · 2017-05-26T07:45:11Z

cluster/kubemark/pre-existing/config-default.sh

+
+NUM_NODES="${NUM_NODES:-1}"
+
+SERVICE_CLUSTER_IP_RANGE="${SERVICE_CLUSTER_IP_RANGE:-}"


Sorry, I still don't understand why do we need to define those vars.
Either we are sourcing another config farther in the flow (e.g. source cluster/kubemark/gce/config-default.sh) and then any of those should be needed. Or we are not sourcing it (and then we should source it here instead).

if [[ -z "${SERVICE_CLUSTER_IP_RANGE:-}" ]]; then cluster_range=$(echo "${MASTER_IP}" | awk -F '.' '{printf("%d.%d.%d.0", $1, $2, $3)}') SERVICE_CLUSTER_IP_RANGE="${SERVICE_CLUSTER_IP_RANGE:-$cluster_range/16}" fi

I can get rid of SERVICE_CLUSTER_IP_RANGE and DNS_DOMAIN. The remaining vars will never be defined if I don't set them.

The file cluster/kubemark/gce/config-default.sh does the exact same thing for some of these variables. For instance TEST_CLUSTER_API_CONTENT_TYPE="${TEST_CLUSTER_API_CONTENT_TYPE:-}" is in there. If we want to have a common place to set these instead of provider config files, it might make it a little more confusing IMO.

We still don't understand each other.
I know that we define those variables in this cluster/kubemark/gce/config-default.sh

And I really want to avoid having a second file that is basically exactly the same content.
My comment wasn't about this particular line - it's about the whole file starting from this line.

wojtek-t · 2017-05-26T07:45:46Z

@gmarek - FYI

wojtek-t · 2017-05-29T12:10:08Z

test/kubemark/gce/util.sh

@@ -14,35 +14,15 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

+KUBE_ROOT=$(cd $(dirname "${BASH_SOURCE}")/../../.. && pwd)


No pwd please. Really, the convention in all files is:

KUBE_ROOT=$(dirname "${BASH_SOURCE}")/../../..

wojtek-t · 2017-05-29T12:10:57Z

test/kubemark/pre-existing/README.md

+ 4) The hollow-nodes are configured to talk with the kubemark master at #2
+
+When using the pre-existing provider, the developer is responsible for creating
+#2.  Therefore, the kubemark scripts will not create any infrastructure or start


In fact he is responsible for creating both 1 and 2.

True. My assumption was the developer already knew that, but let's not assume.

wojtek-t · 2017-05-29T12:14:42Z

test/kubemark/pre-existing/README.md

+The all in one Kubemark cluster looks like the following:
+ 1) A running Kubernetes cluster pointed to by the local kubeconfig
+ 2) Some hollow-nodes that run on the Kubernetes Cluster from #1
+ 3) The hollow-nodes are configured to talk with the master at #1


In my opinion it's not that it's not practical for testing. It's just wrong.

In this setup you don't even have a control over how many nodes will be in the cluster. Because depending whether your hollow-node will be scheduled on a real node or on a different "hollow node" it will either be a real node, or not-running pod.

This is just bad and asking for troubles. We should really point that it's wrong. Not just discourage.

@gmarek FYI

rthallisey · 2017-06-05T18:54:00Z

Rebased. Anymore comments from folks? @wojtek-t @shyamjvs

shyamjvs · 2017-06-21T08:28:07Z

@rthallisey Sorry for the delay in response, I was busy with few things important for the release. The PR looks in a much better shape now. :) However it'd be good to get it finally lgtm'd by @wojtek-t (he'll come back from vacation at the start of july).

rthallisey · 2017-06-21T11:08:24Z

@shyamjvs Cool 👍. Thanks for the heads up.

wojtek-t

Just minor nits - other than that LGTM.

wojtek-t · 2017-07-04T11:50:50Z

cluster/kubemark/pre-existing/config-default.sh

+#   $CONTAINER_REGISTRY/$PROJECT/kubemark
+#
+PROJECT="${PROJECT:-}"
+CONTAINER_REGISTRY="${CONTAINER_REGISTRY:-}"


Can you please swap the order of those two - to reflect the order in which they are used. I mean:

CONTAINER_REGISTRY="${CONTAINER_REGISTRY:-}" PROJECT="${PROJECT:-}"

wojtek-t · 2017-07-04T11:52:39Z

test/kubemark/gce/util.sh

@@ -14,34 +14,15 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

+KUBE_ROOT=$(dirname "${BASH_SOURCE}")/../../..
+
+source ${KUBE_ROOT}/test/kubemark/common/util.sh


Use " for instructions.

In order to expand the use of kubemark, allow developers to use kubemark with a pre-existing Kubemark master.

rthallisey · 2017-07-05T16:47:29Z

@wojtek-t rebased and addressed your comments

wojtek-t · 2017-07-06T07:30:00Z

/retest

wojtek-t · 2017-07-06T07:30:06Z

/lgtm

wojtek-t · 2017-07-06T07:30:11Z

/approve no-issue

k8s-github-robot · 2017-07-06T07:30:16Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rthallisey, shyamjvs, wojtek-t

Associated issue: 44393

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

~~cluster/OWNERS~~ [wojtek-t]
~~test/kubemark/OWNERS~~ [shyamjvs,wojtek-t]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

wojtek-t · 2017-07-06T09:01:58Z

/test pull-kubernetes-e2e-gce-etcd3
/test pull-kubernetes-federation-e2e-gce

k8s-github-robot · 2017-07-06T11:41:52Z

Automatic merge from submit-queue

Kevin-fqh · 2018-04-09T17:21:31Z

@rthallisey @shyamjvs how does your kubemark master start? The kubemark master and the base k8s cluster's master can be the same VM? Colud you tell more detail about how to use the pre-existing ? Thanks

My step is as follow, but it not work. Could you give me some help?
1、I create 4 VMs
2、start etcd kube-apiserver, kibe-controller-manager,scheduler on the 1th VM, the ip is 192.168.132.131, play the role of kubemark master.
3、I build a base k8s cluster on 2th, 3rd, 4th VM(not any cloud provider), the master is 2th VM, the master ip is 192.168.132.129.
4、 192.168.132.129 can ssh kubernetes@ 192.168.132.131
5、on the 192.168.132.129, change the test/kubemark/cloud-provider-config.sh as pre-existing MASTER_IP="192.168.132.131:6443"
6. on the 192.168.132.129, excute the k8s.io/kubernetes/test/kubemark/start-kubemark.sh,but some unexpected errors occurs such as:
`
[root@f kubernetes]# test/kubemark/start-kubemark.sh

                  STARTING SETUP FOR MASTER
                   Created the environment file for master.
                   MASTER_IP: 192.168.132.131:6443
                  MASTER_NAME: 
                  kubernetes@192.168.132.131's password: 
                   Attempt 1 failed to ssh kubernetes@192.168.132.131 sudo. Retrying.
                  Failed to ssh kubernetes@192.168.132.131 sudo.

`

but I excuse the cmd ssh kubernetes@192.168.132.131 on shell directly is ok.

I use the binary to build base k8s cluster and kubemark master directly .
Does must start the k8s cluster or kubemark in a specific way, such as kubeadm?

gaorong · 2019-06-29T10:06:18Z

@rthallisey could you please help clarifying why we need a dedicated master (#2 master in readme file) used for hollow-nodes, it seems one master is enough?

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Apr 12, 2017

k8s-github-robot assigned jbeda and shyamjvs Apr 12, 2017

k8s-github-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. release-note-label-needed labels Apr 12, 2017

shyamjvs reviewed May 9, 2017

View reviewed changes

shyamjvs reviewed May 10, 2017

View reviewed changes

shyamjvs suggested changes May 12, 2017

View reviewed changes

rthallisey changed the title ~~Launch kubemark with an existing Kubernetes cluster~~ Launch kubemark with an existing Kubemark master May 25, 2017

wojtek-t reviewed May 26, 2017

View reviewed changes

wojtek-t reviewed May 29, 2017

View reviewed changes

shyamjvs mentioned this pull request May 30, 2017

Use kubemark to test k8s performance but get: !!! Error in ./hack/ginkgo-e2e.sh:117 #45288

Closed

k8s-github-robot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Jun 6, 2017

shyamjvs added this to the v1.8 milestone Jun 21, 2017

wojtek-t reviewed Jul 4, 2017

View reviewed changes

Ryan Hallisey added 2 commits July 5, 2017 09:14

Launch kubemark with an existing Kubemark Master

82e1d20

In order to expand the use of kubemark, allow developers to use kubemark with a pre-existing Kubemark master.

Add a README to the pre-existing provdier

f9354bf

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 6, 2017

k8s-github-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 6, 2017

k8s-github-robot merged commit 20e629b into kubernetes:master Jul 6, 2017

rthallisey deleted the pre-existing-provider branch July 6, 2017 20:59

Kevin-fqh mentioned this pull request Apr 10, 2018

Kubemark Test-How to point to Master cluster #61538

Closed

jpeach mentioned this pull request Jun 18, 2019

Document how to run kubemark against a pre-existing cluster kubernetes/community#3809

Closed

gaorong mentioned this pull request Jun 29, 2019

Enable setting up Kubemark master outside GCE. #78842

Closed


		## Use Case

		The goal of the pre-existing provider is to use the kubemark tools with a


		NUM_NODES="${NUM_NODES:-1}"

		SERVICE_CLUSTER_IP_RANGE="${SERVICE_CLUSTER_IP_RANGE:-}"

Launch kubemark with an existing Kubemark master #44394

Launch kubemark with an existing Kubemark master #44394

Conversation

rthallisey commented Apr 12, 2017

k8s-reviewable commented Apr 12, 2017

k8s-ci-robot commented Apr 12, 2017

rthallisey commented May 9, 2017

shyamjvs commented May 9, 2017 • edited

rthallisey commented May 9, 2017

shyamjvs commented May 9, 2017

rthallisey commented May 9, 2017

shyamjvs commented May 9, 2017

shyamjvs left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rthallisey commented May 9, 2017

rthallisey commented May 9, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shyamjvs commented May 10, 2017

rthallisey commented May 10, 2017

shyamjvs commented May 10, 2017

rthallisey commented May 10, 2017

rthallisey commented May 10, 2017

shyamjvs commented May 10, 2017

rthallisey commented May 12, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rthallisey May 15, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shyamjvs May 15, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shyamjvs commented May 12, 2017 • edited

rthallisey commented May 15, 2017

shyamjvs commented May 15, 2017 • edited

rthallisey commented May 15, 2017 • edited

shyamjvs commented May 15, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shyamjvs commented May 9, 2017 •

edited

rthallisey May 15, 2017 •

edited

shyamjvs May 15, 2017 •

edited

shyamjvs commented May 12, 2017 •

edited

shyamjvs commented May 15, 2017 •

edited

rthallisey commented May 15, 2017 •

edited