Enable setting up Kubernetes cluster in Ubuntu on Azure #21207

Merged
merged 2 commits into from Jun 27, 2016

Conversation

Projects
None yet
@WeixuZhuang
Contributor

WeixuZhuang commented Feb 13, 2016

Implement basic cloud provider functionality to deploy Kubernetes on
Azure. SaltStack is used to deploy Kubernetes on top of Ubuntu
virtual machines. OpenVpn provides network connectivity. For
kubelet authentication, we use basic authentication (username and
password). The scripts use the legacy Azure Service Management APIs.

We have set up a nightly test job in our Jenkins server for federated
testing to run the e2e test suite on Azure. With the cloud provider
scripts in this commit, 14 e2e test cases pass in this environment.
We plan to implement additional Azure functionality to support more
test cases.


This change is Reviewable

@googlebot googlebot added the cla: yes label Feb 13, 2016

@k8s-bot

This comment has been minimized.

Show comment
Hide comment
@k8s-bot

k8s-bot Feb 13, 2016

Can one of the admins verify that this patch is reasonable to test? (reply "ok to test", or if you trust the user, reply "add to whitelist")

If this message is too spammy, please complain to ixdy.

k8s-bot commented Feb 13, 2016

Can one of the admins verify that this patch is reasonable to test? (reply "ok to test", or if you trust the user, reply "add to whitelist")

If this message is too spammy, please complain to ixdy.

@k8s-bot

This comment has been minimized.

Show comment
Hide comment
@k8s-bot

k8s-bot Feb 13, 2016

Can one of the admins verify that this patch is reasonable to test? (reply "ok to test", or if you trust the user, reply "add to whitelist")

If this message is too spammy, please complain to ixdy.

k8s-bot commented Feb 13, 2016

Can one of the admins verify that this patch is reasonable to test? (reply "ok to test", or if you trust the user, reply "add to whitelist")

If this message is too spammy, please complain to ixdy.

@parantap2001

This comment has been minimized.

Show comment
Hide comment

@ixdy @fejta - thoughts ?

@ixdy

This comment has been minimized.

Show comment
Hide comment
@ixdy

ixdy Feb 16, 2016

Member

We're in code slush for 1.2, and this is a pretty big change, so it may take a little while to get this reviewed and merged (and it's unlikely to make 1.2). I'll look around to see who'd be appropriate to review.

Member

ixdy commented Feb 16, 2016

We're in code slush for 1.2, and this is a pretty big change, so it may take a little while to get this reviewed and merged (and it's unlikely to make 1.2). I'll look around to see who'd be appropriate to review.

@k8s-merge-robot

This comment has been minimized.

Show comment
Hide comment
@k8s-merge-robot

k8s-merge-robot Feb 22, 2016

Collaborator

PR needs rebase

Collaborator

k8s-merge-robot commented Feb 22, 2016

PR needs rebase

@k8s-merge-robot

This comment has been minimized.

Show comment
Hide comment
@k8s-merge-robot

k8s-merge-robot Mar 4, 2016

Collaborator

Adding label:do-not-merge because PR changes docs prohibited to auto merge
See http://kubernetes.github.io/docs/editdocs/ for information about editting docs

Collaborator

k8s-merge-robot commented Mar 4, 2016

Adding label:do-not-merge because PR changes docs prohibited to auto merge
See http://kubernetes.github.io/docs/editdocs/ for information about editting docs

@WeixuZhuang

This comment has been minimized.

Show comment
Hide comment
@WeixuZhuang

WeixuZhuang Mar 5, 2016

Contributor

Hi @ixdy , I found that there are some merged in cluster/azure which is some code for setting up kubernetes in azure cloud. My code does the same thing and it has been here for two weeks waiting for review....

Contributor

WeixuZhuang commented Mar 5, 2016

Hi @ixdy , I found that there are some merged in cluster/azure which is some code for setting up kubernetes in azure cloud. My code does the same thing and it has been here for two weeks waiting for review....

@ixdy

This comment has been minimized.

Show comment
Hide comment
@ixdy

ixdy Mar 7, 2016

Member

@WeixuZhuang sorry for the delay - as I mentioned earlier, we were trying to stabilize the tree in preparation for cutting the release-1.2 branch, and so now that that's done, there should be more traction on this PR. Note that you'll need to separate out the docs into a separate PR, as mentioned by the bot and in https://groups.google.com/forum/#!topic/kubernetes-dev/UedwIYb-fHg.

This PR also needs to be rebased. I'll look to see if there's someone who can review this after that's done.

Member

ixdy commented Mar 7, 2016

@WeixuZhuang sorry for the delay - as I mentioned earlier, we were trying to stabilize the tree in preparation for cutting the release-1.2 branch, and so now that that's done, there should be more traction on this PR. Note that you'll need to separate out the docs into a separate PR, as mentioned by the bot and in https://groups.google.com/forum/#!topic/kubernetes-dev/UedwIYb-fHg.

This PR also needs to be rebased. I'll look to see if there's someone who can review this after that's done.

parantap2001 referenced this pull request in colemickens/kubernetes Mar 7, 2016

@davidopp

This comment has been minimized.

Show comment
Hide comment
Member

davidopp commented Mar 7, 2016

@parantap2001

This comment has been minimized.

Show comment
Hide comment
@parantap2001

parantap2001 Mar 7, 2016

https://github.com/kubernetes/kubernetes/pull/22231/commits
here makes the entire cluster bring up on azure/e2e on kubernetes dependent on an external repository here - https://github.com/colemickens/azkube

@ixdy @colemickens @jeffmendoza @fejta @davidopp @WeixuZhuang

Why did we choose this approach rather than picking this existing approach similar to other clusters -
kubernetes as is here #21207

We can continue with the rebase to get this code in, however before we do that we wanted consensus on the best direction to move forward in terms on cluster bring up.

FYI. We have several E2E running here -
https://console.cloud.google.com/storage/browser/kube_azure_log/
and can easily upstream results to be shared in the dashboard of the nightly runs here -
http://storage.googleapis.com/kubernetes-test-history/static/index.html

https://github.com/kubernetes/kubernetes/pull/22231/commits
here makes the entire cluster bring up on azure/e2e on kubernetes dependent on an external repository here - https://github.com/colemickens/azkube

@ixdy @colemickens @jeffmendoza @fejta @davidopp @WeixuZhuang

Why did we choose this approach rather than picking this existing approach similar to other clusters -
kubernetes as is here #21207

We can continue with the rebase to get this code in, however before we do that we wanted consensus on the best direction to move forward in terms on cluster bring up.

FYI. We have several E2E running here -
https://console.cloud.google.com/storage/browser/kube_azure_log/
and can easily upstream results to be shared in the dashboard of the nightly runs here -
http://storage.googleapis.com/kubernetes-test-history/static/index.html

@davidopp

This comment has been minimized.

Show comment
Hide comment
@davidopp

davidopp Mar 7, 2016

Member

Very sorry, I was not aware of this PR (and I suspect @colemickens was not either). Although it may not have helped, filing an issue, and then attaching the PR to an issue, makes it easier for us to track stuff, because it will at least get assigned to a team. With 333 open PRs right now, it's very easy for issue-less PRs to get lost.

Member

davidopp commented Mar 7, 2016

Very sorry, I was not aware of this PR (and I suspect @colemickens was not either). Although it may not have helped, filing an issue, and then attaching the PR to an issue, makes it easier for us to track stuff, because it will at least get assigned to a team. With 333 open PRs right now, it's very easy for issue-less PRs to get lost.

@colemickens

This comment has been minimized.

Show comment
Hide comment
@colemickens

colemickens Mar 7, 2016

Contributor

I was also unaware of that pull request.

I'm working on getting e2e validation for the new Azure bring-up.

I had a preliminary version that was fully shell-script based, but it was error-prone and it's behavior was hard to get correct depending on the version of the azure-xplat-cli that was installed. The method used in my pull request minimized the number of dependencies needed ahead-of-time and increased the likelihood of a successful deployment. It also gives us some flexibility - the core piece (producing an ARM template, creating the Azure Service Principal, etc) can be re-used for different setups. For example, azkube currently creates CoreOS templates for Azure, but could be extended in the future to produce templates for Ubuntu or others, as well.

Additionally, I moved away from the Salt stack bringup based on indications that a more self-contained, docker-based bring up was the way forward.

Contributor

colemickens commented Mar 7, 2016

I was also unaware of that pull request.

I'm working on getting e2e validation for the new Azure bring-up.

I had a preliminary version that was fully shell-script based, but it was error-prone and it's behavior was hard to get correct depending on the version of the azure-xplat-cli that was installed. The method used in my pull request minimized the number of dependencies needed ahead-of-time and increased the likelihood of a successful deployment. It also gives us some flexibility - the core piece (producing an ARM template, creating the Azure Service Principal, etc) can be re-used for different setups. For example, azkube currently creates CoreOS templates for Azure, but could be extended in the future to produce templates for Ubuntu or others, as well.

Additionally, I moved away from the Salt stack bringup based on indications that a more self-contained, docker-based bring up was the way forward.

@parantap2001

This comment has been minimized.

Show comment
Hide comment
@parantap2001

parantap2001 Mar 7, 2016

point taken @davidopp. Miss on our part, we should have created an issue for it. Completely understandable given the huge churn for v1.2 as well.

@colemickens We are playing with the patches that you have and can evaluate whether generating a ubuntu template with your containerized methodology is simpler. However I am not convinced on the fact that any changes to cluster/azure now potentially means pull requests to two repositories.
Additionally I am a fan of uniformity so would prefer the methodology adopted and supported for other clusters (aws, gce..)

point taken @davidopp. Miss on our part, we should have created an issue for it. Completely understandable given the huge churn for v1.2 as well.

@colemickens We are playing with the patches that you have and can evaluate whether generating a ubuntu template with your containerized methodology is simpler. However I am not convinced on the fact that any changes to cluster/azure now potentially means pull requests to two repositories.
Additionally I am a fan of uniformity so would prefer the methodology adopted and supported for other clusters (aws, gce..)

@WeixuZhuang

This comment has been minimized.

Show comment
Hide comment
@WeixuZhuang

WeixuZhuang Mar 8, 2016

Contributor

@ixdy rebased/removed docs.

@davidopp created issue to track status #22715

@colemickens created a shell switch to have two different ways of setting up the cluster/azure kube-up. the switch is 'local' to retain the shell script methodology and 'docker' to use the container(using image colemickens/azkube) method for cluster/azure setup.

Contributor

WeixuZhuang commented Mar 8, 2016

@ixdy rebased/removed docs.

@davidopp created issue to track status #22715

@colemickens created a shell switch to have two different ways of setting up the cluster/azure kube-up. the switch is 'local' to retain the shell script methodology and 'docker' to use the container(using image colemickens/azkube) method for cluster/azure setup.

@zmerlynn

This comment has been minimized.

Show comment
Hide comment
@zmerlynn

zmerlynn Mar 21, 2016

Member

What's the status of this PR? I'm finally iterating my backlog and it's not clear if this PR is going to get acted on, or if we're headed a different way for Azure.

Member

zmerlynn commented Mar 21, 2016

What's the status of this PR? I'm finally iterating my backlog and it's not clear if this PR is going to get acted on, or if we're headed a different way for Azure.

@parantap2001

This comment has been minimized.

Show comment
Hide comment
@parantap2001

parantap2001 Mar 21, 2016

Changes in this PR are still current and rebase correctly on master.

The e2e tests on azure still work based on this patchset and are posted regularly here -
https://console.cloud.google.com/storage/browser/kube_azure_log/

We were hoping this change would go in as it does not break existing ways on deploying on azure (i.e externally maintained container with coreos + flannel introduced with a commit recently). In fact it adds a new way of deploying cluster/azure with ubuntu + openvpn.
This commit also provides a flag to switch between the method of deployment so both methodologies can co-exist.

Changes in this PR are still current and rebase correctly on master.

The e2e tests on azure still work based on this patchset and are posted regularly here -
https://console.cloud.google.com/storage/browser/kube_azure_log/

We were hoping this change would go in as it does not break existing ways on deploying on azure (i.e externally maintained container with coreos + flannel introduced with a commit recently). In fact it adds a new way of deploying cluster/azure with ubuntu + openvpn.
This commit also provides a flag to switch between the method of deployment so both methodologies can co-exist.

@davidopp

This comment has been minimized.

Show comment
Hide comment
@davidopp

davidopp Mar 21, 2016

Member

I'll reassign to myself for now.

Member

davidopp commented Mar 21, 2016

I'll reassign to myself for now.

@spiffxp

This comment has been minimized.

Show comment
Hide comment
@spiffxp

spiffxp Apr 5, 2016

Member

/cc @kubernetes/sig-testing

Would be in favor of seeing this eventually get exercised via a daily kubernetes-e2e-azure job, same as is currently done for AWS with kubernetes-e2e-aws

Member

spiffxp commented Apr 5, 2016

/cc @kubernetes/sig-testing

Would be in favor of seeing this eventually get exercised via a daily kubernetes-e2e-azure job, same as is currently done for AWS with kubernetes-e2e-aws

@colemickens

This comment has been minimized.

Show comment
Hide comment
@colemickens

colemickens Apr 5, 2016

Contributor

I have a slight concern that this is using the legacy Azure Service Management APIs rather than the new Azure Resource Manager APIs. This has large implications down the line for the cloudprovider implementation as well. Additionally, the heavy use of Salt, just as kube-deploy is being started with the intention of moving the deployment stuff out-of-tree and as I understand it, away from Salt.

I'm in progress on getting a kubernetes-e2e-azure job working for the new Azure bring-up that now exists in the tree. I'm adding it in a similar way to how the kubernetes-e2e-aws job is defined in the JJB definitions in hack/jenkins.

Contributor

colemickens commented Apr 5, 2016

I have a slight concern that this is using the legacy Azure Service Management APIs rather than the new Azure Resource Manager APIs. This has large implications down the line for the cloudprovider implementation as well. Additionally, the heavy use of Salt, just as kube-deploy is being started with the intention of moving the deployment stuff out-of-tree and as I understand it, away from Salt.

I'm in progress on getting a kubernetes-e2e-azure job working for the new Azure bring-up that now exists in the tree. I'm adding it in a similar way to how the kubernetes-e2e-aws job is defined in the JJB definitions in hack/jenkins.

@spiffxp

This comment has been minimized.

Show comment
Hide comment
@spiffxp

spiffxp Apr 5, 2016

Member

Additionally, the heavy use of Salt, just as kube-deploy is being started with the intention of moving the deployment stuff out-of-tree and as I understand it, away from Salt.

@colemickens I suspect that effort is going to take some time; if this more or less works today, it's something to at least iterate on, no?

Member

spiffxp commented Apr 5, 2016

Additionally, the heavy use of Salt, just as kube-deploy is being started with the intention of moving the deployment stuff out-of-tree and as I understand it, away from Salt.

@colemickens I suspect that effort is going to take some time; if this more or less works today, it's something to at least iterate on, no?

@davidopp

This comment has been minimized.

Show comment
Hide comment
@davidopp

davidopp Jun 7, 2016

Member

ok to test

Member

davidopp commented Jun 7, 2016

ok to test

@davidopp

This comment has been minimized.

Show comment
Hide comment
@davidopp

davidopp Jun 7, 2016

Member

LGTM

Member

davidopp commented Jun 7, 2016

LGTM

@davidopp davidopp added this to the v1.3 milestone Jun 8, 2016

@davidopp

This comment has been minimized.

Show comment
Hide comment
@davidopp

davidopp Jun 8, 2016

Member

Test failure is

Verifying ./hack/../hack/verify-boilerplate.sh
Boilerplate header is wrong for: /go/src/k8s.io/kubernetes/cluster/azure-legacy/templates/create-kubeconfig.sh
FAILED   ./hack/../hack/verify-boilerplate.sh   1s
Member

davidopp commented Jun 8, 2016

Test failure is

Verifying ./hack/../hack/verify-boilerplate.sh
Boilerplate header is wrong for: /go/src/k8s.io/kubernetes/cluster/azure-legacy/templates/create-kubeconfig.sh
FAILED   ./hack/../hack/verify-boilerplate.sh   1s

@davidopp davidopp removed this from the v1.3 milestone Jun 8, 2016

@k8s-bot

This comment has been minimized.

Show comment
Hide comment
@k8s-bot

k8s-bot Jun 8, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

k8s-bot commented Jun 8, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

@k8s-merge-robot k8s-merge-robot removed the lgtm label Jun 8, 2016

@davidopp davidopp added the lgtm label Jun 8, 2016

@WeixuZhuang

This comment has been minimized.

Show comment
Hide comment
@WeixuZhuang

WeixuZhuang Jun 8, 2016

Contributor

@davidopp I have fixed the issue with boilerplate check. All tests are passing now.

Thanks

Contributor

WeixuZhuang commented Jun 8, 2016

@davidopp I have fixed the issue with boilerplate check. All tests are passing now.

Thanks

@Krylon360

This comment has been minimized.

Show comment
Hide comment
@Krylon360

Krylon360 Jun 9, 2016

@WeixuZhuang would love to have the ARM templates added to this. ARM is pretty much the standard now with Azure. ASM still works; however many features are being depreciated very quickly. It's best to be ahead of the depreciation of ASM by having the ARM scripts ready to go.

@WeixuZhuang would love to have the ARM templates added to this. ARM is pretty much the standard now with Azure. ASM still works; however many features are being depreciated very quickly. It's best to be ahead of the depreciation of ASM by having the ARM scripts ready to go.

@parantap2001

This comment has been minimized.

Show comment
Hide comment
@parantap2001

parantap2001 Jun 9, 2016

@Krylon360 ARM Changes are ready. We are in the process of making a separate issue and adding commits to it.

For this commit/PR we are at a stable checkpoint, as the federated dashboard has these results upstreamed nightly now -
https://k8s-testgrid.appspot.com/k8s#azure

@Krylon360 ARM Changes are ready. We are in the process of making a separate issue and adding commits to it.

For this commit/PR we are at a stable checkpoint, as the federated dashboard has these results upstreamed nightly now -
https://k8s-testgrid.appspot.com/k8s#azure

@jamescw

This comment has been minimized.

Show comment
Hide comment
@jamescw

jamescw Jun 10, 2016

@WeixuZhuang the submit queue seems blocked, how can I find the install instructions for this in the meantime?

jamescw commented Jun 10, 2016

@WeixuZhuang the submit queue seems blocked, how can I find the install instructions for this in the meantime?

@k8s-bot

This comment has been minimized.

Show comment
Hide comment
@k8s-bot

k8s-bot Jun 14, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

k8s-bot commented Jun 14, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

@k8s-bot

This comment has been minimized.

Show comment
Hide comment
@k8s-bot

k8s-bot Jun 15, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

k8s-bot commented Jun 15, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

WeixuZhuang added some commits Jan 12, 2016

Azure/ubuntu/saltstack support re-instated
This first reverts commit 8e8437d.
Also resolves conflicts with docs on f334fc4
And resolves conflicts with https://github.com/kubernetes/kubernetes/pull/22231/commits
to make people switching between two different methods of setting up by
setting env variables.

Conflicts:
	cluster/get-kube.sh
	cluster/saltbase/salt/README.md
	cluster/saltbase/salt/kube-proxy/default
	cluster/saltbase/salt/top.sls
Implement Azure cloud provider scripts
Implement basic cloud provider functionality to deploy Kubernetes on
Azure.  SaltStack is used to deploy Kubernetes on top of Ubuntu
virtual machines.  OpenVpn provides network connectivity.  For
kubelet authentication, we use basic authentication (username and
password).  The scripts use the legacy Azure Service Management APIs.

We have set up a nightly test job in our Jenkins server for federated
testing to run the e2e test suite on Azure.  With the cloud provider
scripts in this commit, 14 e2e test cases pass in this environment.
We plan to implement additional Azure functionality to support more
test cases.
@travisnewhouse

This comment has been minimized.

Show comment
Hide comment
@travisnewhouse

travisnewhouse Jun 18, 2016

@davidopp Rebased again. Can you let us know if there are any other requirements before this can be merged? Thanks!

@davidopp Rebased again. Can you let us know if there are any other requirements before this can be merged? Thanks!

@davidopp

This comment has been minimized.

Show comment
Hide comment
@davidopp

davidopp Jun 19, 2016

Member

Hi Travis. I feel bad because it took us so long to review this, and then by the time it was finished we were in our "code freeze" and not accepting new PRs until we cut the 1.3 final release. Since this is a new feature and not fixing existing functionality, I think we should wait until the code freeze is over, which I believe will be around the end of this week. Sorry about the delay.

So in a nutshell, there's nothing blocking this other than the lifting of the code freeze.

Member

davidopp commented Jun 19, 2016

Hi Travis. I feel bad because it took us so long to review this, and then by the time it was finished we were in our "code freeze" and not accepting new PRs until we cut the 1.3 final release. Since this is a new feature and not fixing existing functionality, I think we should wait until the code freeze is over, which I believe will be around the end of this week. Sorry about the delay.

So in a nutshell, there's nothing blocking this other than the lifting of the code freeze.

@davidopp

This comment has been minimized.

Show comment
Hide comment
@davidopp

davidopp Jun 19, 2016

Member

ok to test

Member

davidopp commented Jun 19, 2016

ok to test

@k8s-bot

This comment has been minimized.

Show comment
Hide comment

k8s-bot commented Jun 19, 2016

GCE e2e build/test passed for commit e35c1cc.

@k8s-bot

This comment has been minimized.

Show comment
Hide comment
@k8s-bot

k8s-bot Jun 23, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

k8s-bot commented Jun 23, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

@davidopp

This comment has been minimized.

Show comment
Hide comment
@davidopp

davidopp Jun 27, 2016

Member

ok to test

Member

davidopp commented Jun 27, 2016

ok to test

@k8s-bot

This comment has been minimized.

Show comment
Hide comment
@@ -30,6 +30,8 @@
# * export KUBERNETES_PROVIDER=aws; wget -q -O - https://get.k8s.io | bash
# Libvirt (with CoreOS as a guest operating system)
# * export KUBERNETES_PROVIDER=libvirt-coreos; wget -q -O - https://get.k8s.io | bash
+# Microsoft Azure

This comment has been minimized.

@davidopp

davidopp Jun 27, 2016

Member

@colemickens you might want to add a comment for yours to this file too

@davidopp

davidopp Jun 27, 2016

Member

@colemickens you might want to add a comment for yours to this file too

@davidopp davidopp added the lgtm label Jun 27, 2016

@davidopp

This comment has been minimized.

Show comment
Hide comment
@davidopp

davidopp Jun 27, 2016

Member

LGTM

Member

davidopp commented Jun 27, 2016

LGTM

@k8s-bot

This comment has been minimized.

Show comment
Hide comment
@k8s-merge-robot

This comment has been minimized.

Show comment
Hide comment
@k8s-merge-robot

k8s-merge-robot Jun 27, 2016

Collaborator

Automatic merge from submit-queue

Collaborator

k8s-merge-robot commented Jun 27, 2016

Automatic merge from submit-queue

@k8s-merge-robot k8s-merge-robot merged commit 95a3737 into kubernetes:master Jun 27, 2016

7 of 8 checks passed

Jenkins GKE e2e Build triggered. sha1 is merged.
Details
Jenkins GCE Node e2e Build finished. 92 tests run, 12 skipped, 0 failed.
Details
Jenkins GCE e2e 323 tests run, 141 skipped, 0 failed.
Details
Jenkins GKE smoke e2e 323 tests run, 322 skipped, 0 failed.
Details
Jenkins unit/integration 3332 tests run, 14 skipped, 0 failed.
Details
Jenkins verification No test results found.
Details
Submit Queue Queued to run github e2e tests a second time.
Details
cla/google All necessary CLAs are signed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment