New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GCP: Support for ability to specify manual subnet #183

Closed
jshah-pivotal opened this Issue Mar 1, 2018 · 5 comments

Comments

Projects
None yet
5 participants
@jshah-pivotal

jshah-pivotal commented Mar 1, 2018

Deploying a cluster with manual subnets, the cloud-provider.ini needs to specify the subnetwork-name to be able to deploy
a Service of type LoadBalancer with the GCE annotation cloud.google.com/load-balancer-type of type Internal.

Without this, the IP address isn't correctly assigned on a subnet in the VPC.

The erb template
kube-apiserver/templates/config/cloud-provider.ini.erb

doesn't have this field nor does the
BOSH job spec
have an option for it.

cc @cholick

@cf-gitbot

This comment has been minimized.

Show comment
Hide comment
@cf-gitbot

cf-gitbot Mar 1, 2018

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/155638519

The labels on this github issue will be updated when the story is started.

cf-gitbot commented Mar 1, 2018

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/155638519

The labels on this github issue will be updated when the story is started.

@karampok karampok self-assigned this Mar 16, 2018

@karampok

This comment has been minimized.

Show comment
Hide comment
@karampok

karampok Mar 16, 2018

Member

Hi @jshah-pivotal ,
could you elaborate a bit more the issue?

"Deploying with manual subnets", where/what is the deploying refers to? bosh?

What is the error you get? or what is the output of the kubectl commands that shows
`the IP address isn't correctly assigned on a subnet in the VPC"

Thanks!

Member

karampok commented Mar 16, 2018

Hi @jshah-pivotal ,
could you elaborate a bit more the issue?

"Deploying with manual subnets", where/what is the deploying refers to? bosh?

What is the error you get? or what is the output of the kubectl commands that shows
`the IP address isn't correctly assigned on a subnet in the VPC"

Thanks!

@jshah-pivotal

This comment has been minimized.

Show comment
Hide comment
@jshah-pivotal

jshah-pivotal Mar 16, 2018

Bosh is the one deploying with manual subnets.
This is subnet config we are talking about.

https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/gce/gce.go#L167

https://github.com/cloudfoundry-incubator/kubo-release/blob/master/jobs/kube-apiserver/templates/config/cloud-provider.ini.erb#L13

add subnetwork-name here.

In the logs we see that subnet name comes out empty. When loadbalancer is created there is no frontend. It does not get IP assigned to it.

jshah-pivotal commented Mar 16, 2018

Bosh is the one deploying with manual subnets.
This is subnet config we are talking about.

https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/gce/gce.go#L167

https://github.com/cloudfoundry-incubator/kubo-release/blob/master/jobs/kube-apiserver/templates/config/cloud-provider.ini.erb#L13

add subnetwork-name here.

In the logs we see that subnet name comes out empty. When loadbalancer is created there is no frontend. It does not get IP assigned to it.

@bengtrj

This comment has been minimized.

Show comment
Hide comment
@bengtrj

bengtrj Aug 13, 2018

Contributor

This is a blocker for the Tracker team now. They're evaluating PKS (that depends on CFCR) by creating memcached service as an internal LoadBalancer and we had the same issue on GCP.

While K8S tries to create the LoadBalancer service on GCP, it needs the specific subnetwork-name - and since the config is not made available by CFCR, the service deployment fails.

A workaround that worked for us was ssh-ing into the master node, editing the cloud-config.ini located at /var/vcap/jobs/kube-controller-manager/config, and issuing a monit restart on the kube-controller-manager process. By looking at kubo-release code, the appropriate file to add the config would be the kube-apiserver/templates/config/cloud-provider.ini.erb because it's the one used by all other jobs on the release.

Related K8S issue:
kubernetes/kubernetes#60665

Contributor

bengtrj commented Aug 13, 2018

This is a blocker for the Tracker team now. They're evaluating PKS (that depends on CFCR) by creating memcached service as an internal LoadBalancer and we had the same issue on GCP.

While K8S tries to create the LoadBalancer service on GCP, it needs the specific subnetwork-name - and since the config is not made available by CFCR, the service deployment fails.

A workaround that worked for us was ssh-ing into the master node, editing the cloud-config.ini located at /var/vcap/jobs/kube-controller-manager/config, and issuing a monit restart on the kube-controller-manager process. By looking at kubo-release code, the appropriate file to add the config would be the kube-apiserver/templates/config/cloud-provider.ini.erb because it's the one used by all other jobs on the release.

Related K8S issue:
kubernetes/kubernetes#60665

@karampok karampok removed their assignment Aug 16, 2018

@cf-gitbot cf-gitbot added scheduled and removed unscheduled labels Aug 20, 2018

@cf-gitbot cf-gitbot added accepted and removed delivered labels Sep 20, 2018

@alex-slynko

This comment has been minimized.

Show comment
Hide comment
@alex-slynko

alex-slynko Oct 19, 2018

Member

The fix is in 0.23 release

Member

alex-slynko commented Oct 19, 2018

The fix is in 0.23 release

@cf-gitbot cf-gitbot removed the accepted label Oct 19, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment