New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

federation: Known issues #88

Open
irfanurrehman opened this Issue Oct 31, 2017 · 23 comments

Comments

Projects
None yet
5 participants
@irfanurrehman
Member

irfanurrehman commented Oct 31, 2017

Issue by nikhiljindal
Friday Dec 16, 2016 at 19:52 GMT
Originally opened as kubernetes/kubernetes#38893


Compiling a list of high level known issues that customers should be aware of while deciding to use federation:

cc @kubernetes/sig-federation-misc

@irfanurrehman

This comment has been minimized.

Show comment
Hide comment
@irfanurrehman

irfanurrehman Oct 31, 2017

Member

Comment by nikhiljindal
Friday Dec 16, 2016 at 19:52 GMT


cc @madhusudancs

Please feel free to add anything that I missed.
I plan to keep this as a running list. Will keep adding/removing stuff as we discover/fix issues

Member

irfanurrehman commented Oct 31, 2017

Comment by nikhiljindal
Friday Dec 16, 2016 at 19:52 GMT


cc @madhusudancs

Please feel free to add anything that I missed.
I plan to keep this as a running list. Will keep adding/removing stuff as we discover/fix issues

@irfanurrehman

This comment has been minimized.

Show comment
Hide comment
@irfanurrehman

irfanurrehman Oct 31, 2017

Member

Comment by valichek
Monday Dec 19, 2016 at 10:22 GMT


I tried to run federation control plane on aws but got many issues recently.

  1. script https://github.com/kubernetes/kubernetes/blob/v1.5.1/federation/deploy/deploy.sh doesn't support aws provider
  2. Tried to use old way (deploying manifests one by one), got federation-apiserverand federation-controller-manager running, but then it resulted to error in fetching secret: secrets "federation-apiserver-kubeconfig" not found at federation-controller-manager container, no description for this secret found.
  3. Couldn't find sources of gcr.io/madhusudancs-containers/federation-charts image
Member

irfanurrehman commented Oct 31, 2017

Comment by valichek
Monday Dec 19, 2016 at 10:22 GMT


I tried to run federation control plane on aws but got many issues recently.

  1. script https://github.com/kubernetes/kubernetes/blob/v1.5.1/federation/deploy/deploy.sh doesn't support aws provider
  2. Tried to use old way (deploying manifests one by one), got federation-apiserverand federation-controller-manager running, but then it resulted to error in fetching secret: secrets "federation-apiserver-kubeconfig" not found at federation-controller-manager container, no description for this secret found.
  3. Couldn't find sources of gcr.io/madhusudancs-containers/federation-charts image
@irfanurrehman

This comment has been minimized.

Show comment
Hide comment
@irfanurrehman

irfanurrehman Oct 31, 2017

Member

Comment by madhusudancs
Tuesday Dec 20, 2016 at 04:34 GMT


@valichek Could you try using kubefed - http://kubernetes.io/docs/admin/federation/kubefed/?

Member

irfanurrehman commented Oct 31, 2017

Comment by madhusudancs
Tuesday Dec 20, 2016 at 04:34 GMT


@valichek Could you try using kubefed - http://kubernetes.io/docs/admin/federation/kubefed/?

@irfanurrehman

This comment has been minimized.

Show comment
Hide comment
@irfanurrehman

irfanurrehman Oct 31, 2017

Member

Comment by valichek
Thursday Dec 22, 2016 at 12:52 GMT


@madhusudancs getting following errors when starting federation services with kubefed

Failed to attach volume "pvc-xxxx" on node "xxxx.us-west-2.compute.internal" with: Error attaching EBS volume: IncorrectState: vol-xxxx is not 'available'

First time there was a message for ec-2 instance: "One or more volumes attached to this instance is impaired". And volume stuck in attaching.
When tried again got the same error, but volume have been attached.

Member

irfanurrehman commented Oct 31, 2017

Comment by valichek
Thursday Dec 22, 2016 at 12:52 GMT


@madhusudancs getting following errors when starting federation services with kubefed

Failed to attach volume "pvc-xxxx" on node "xxxx.us-west-2.compute.internal" with: Error attaching EBS volume: IncorrectState: vol-xxxx is not 'available'

First time there was a message for ec-2 instance: "One or more volumes attached to this instance is impaired". And volume stuck in attaching.
When tried again got the same error, but volume have been attached.

@irfanurrehman

This comment has been minimized.

Show comment
Hide comment
@irfanurrehman

irfanurrehman Oct 31, 2017

Member

Comment by valichek
Thursday Dec 22, 2016 at 14:38 GMT


well, after some time, apiserver turned to be running, but not controller-manager,

Cloud provider could not be initialized: could not init DNS provider "google-clouddns": 
google: could not find default credentials. 
See https://developers.google.com/accounts/docs/application-default-credentials for more information.

I have check kubefed sources and found that --dns-provider and many other things are hardcoded and not configurable
https://github.com/kubernetes/kubernetes/blob/04a74570323eae3fc843ca7a6c34c28ada2847a9/federation/pkg/kubefed/init/init.go#L477

Member

irfanurrehman commented Oct 31, 2017

Comment by valichek
Thursday Dec 22, 2016 at 14:38 GMT


well, after some time, apiserver turned to be running, but not controller-manager,

Cloud provider could not be initialized: could not init DNS provider "google-clouddns": 
google: could not find default credentials. 
See https://developers.google.com/accounts/docs/application-default-credentials for more information.

I have check kubefed sources and found that --dns-provider and many other things are hardcoded and not configurable
https://github.com/kubernetes/kubernetes/blob/04a74570323eae3fc843ca7a6c34c28ada2847a9/federation/pkg/kubefed/init/init.go#L477

@irfanurrehman

This comment has been minimized.

Show comment
Hide comment
@irfanurrehman

irfanurrehman Oct 31, 2017

Member

Comment by madhusudancs
Thursday Dec 22, 2016 at 16:51 GMT


@valichek

I have check kubefed sources and found that --dns-provider and many other things are hardcoded and not configurable
https://github.com/kubernetes/kubernetes/blob/04a74570323eae3fc843ca7a6c34c28ada2847a9/federation/pkg/kubefed/init/init.go#L477

That's an old commit.

If you want to use AWS Route53 instead of Google Cloud DNS, pass --dns-provider=aws-route53 to kubefed init.

For now, you can configure everything you want by just editing the deployments directly after running kubefed init: kubectl --context=<host-cluster> --namespace=federation-system edit deployment federation-apiserver or kubectl --context=<host-cluster> --namespace=federation-system edit deployment federation-controller-manager.

We are collecting a list of things we need to make configurable. What else would you like to configure?

Member

irfanurrehman commented Oct 31, 2017

Comment by madhusudancs
Thursday Dec 22, 2016 at 16:51 GMT


@valichek

I have check kubefed sources and found that --dns-provider and many other things are hardcoded and not configurable
https://github.com/kubernetes/kubernetes/blob/04a74570323eae3fc843ca7a6c34c28ada2847a9/federation/pkg/kubefed/init/init.go#L477

That's an old commit.

If you want to use AWS Route53 instead of Google Cloud DNS, pass --dns-provider=aws-route53 to kubefed init.

For now, you can configure everything you want by just editing the deployments directly after running kubefed init: kubectl --context=<host-cluster> --namespace=federation-system edit deployment federation-apiserver or kubectl --context=<host-cluster> --namespace=federation-system edit deployment federation-controller-manager.

We are collecting a list of things we need to make configurable. What else would you like to configure?

@irfanurrehman

This comment has been minimized.

Show comment
Hide comment
@irfanurrehman

irfanurrehman Oct 31, 2017

Member

Comment by valichek
Thursday Dec 22, 2016 at 17:29 GMT


@madhusudancs well, for now it looks that I need to configure CIDR, I used kube-aws tool to run the cluster, and have following options:
instanceCIDR: "10.0.16.0/20"
serviceCIDR: "10.128.0.0/20"
podCIDR: "10.128.16.0/20"
dnsServiceIP: 10.128.0.10

If I understand the config properly, I have to change from --service-cluster-ip-range=10.0.0.0/16"
to --service-cluster-ip-range=10.128.112.0/20", or should I leave it same as for cluster - serviceCIDR: "10.128.0.0/20" ?

And thank you.

Member

irfanurrehman commented Oct 31, 2017

Comment by valichek
Thursday Dec 22, 2016 at 17:29 GMT


@madhusudancs well, for now it looks that I need to configure CIDR, I used kube-aws tool to run the cluster, and have following options:
instanceCIDR: "10.0.16.0/20"
serviceCIDR: "10.128.0.0/20"
podCIDR: "10.128.16.0/20"
dnsServiceIP: 10.128.0.10

If I understand the config properly, I have to change from --service-cluster-ip-range=10.0.0.0/16"
to --service-cluster-ip-range=10.128.112.0/20", or should I leave it same as for cluster - serviceCIDR: "10.128.0.0/20" ?

And thank you.

@irfanurrehman

This comment has been minimized.

Show comment
Hide comment
@irfanurrehman

irfanurrehman Oct 31, 2017

Member

Comment by valichek
Thursday Dec 22, 2016 at 17:44 GMT


And --dns-zone-id too

Member

irfanurrehman commented Oct 31, 2017

Comment by valichek
Thursday Dec 22, 2016 at 17:44 GMT


And --dns-zone-id too

@irfanurrehman

This comment has been minimized.

Show comment
Hide comment
@irfanurrehman

irfanurrehman Oct 31, 2017

Member

Comment by rbtcollins
Tuesday Jan 24, 2017 at 20:08 GMT


@madhusudancs Is there a means to move the federation control plane to another cluster? We're trying to avoid in-place upgrades of k8s clusters - should I file a separate bug on kubefed?

Member

irfanurrehman commented Oct 31, 2017

Comment by rbtcollins
Tuesday Jan 24, 2017 at 20:08 GMT


@madhusudancs Is there a means to move the federation control plane to another cluster? We're trying to avoid in-place upgrades of k8s clusters - should I file a separate bug on kubefed?

@irfanurrehman

This comment has been minimized.

Show comment
Hide comment
@irfanurrehman

irfanurrehman Oct 31, 2017

Member

Comment by madhusudancs
Wednesday Jan 25, 2017 at 20:31 GMT


@valichek --service-cluster-ip-range in federation API server does nothing. So just leave the defaults as is. In the next release (v1.6), we are removing that flag from federation.

Member

irfanurrehman commented Oct 31, 2017

Comment by madhusudancs
Wednesday Jan 25, 2017 at 20:31 GMT


@valichek --service-cluster-ip-range in federation API server does nothing. So just leave the defaults as is. In the next release (v1.6), we are removing that flag from federation.

@irfanurrehman

This comment has been minimized.

Show comment
Hide comment
@irfanurrehman

irfanurrehman Oct 31, 2017

Member

Comment by madhusudancs
Wednesday Jan 25, 2017 at 20:34 GMT


@rbtcollins It is possible, but you have to orchestrate that manually. You just need to start the federation API server and controller manager pods in the new cluster and attach the etcd volume from the old cluster to those pods.

Please feel free to file an issue for kubefed. We will look into it when we design/implement HA support for federation control plane via kubefed.

Member

irfanurrehman commented Oct 31, 2017

Comment by madhusudancs
Wednesday Jan 25, 2017 at 20:34 GMT


@rbtcollins It is possible, but you have to orchestrate that manually. You just need to start the federation API server and controller manager pods in the new cluster and attach the etcd volume from the old cluster to those pods.

Please feel free to file an issue for kubefed. We will look into it when we design/implement HA support for federation control plane via kubefed.

@irfanurrehman

This comment has been minimized.

Show comment
Hide comment
@irfanurrehman

irfanurrehman Oct 31, 2017

Member

Comment by quinton-hoole
Friday Sep 08, 2017 at 02:32 GMT


We can update this issue to track all of the remaining GA items.

Member

irfanurrehman commented Oct 31, 2017

Comment by quinton-hoole
Friday Sep 08, 2017 at 02:32 GMT


We can update this issue to track all of the remaining GA items.

@irfanurrehman

This comment has been minimized.

Show comment
Hide comment
@irfanurrehman

irfanurrehman Oct 31, 2017

Member

Comment by xtophs
Tuesday Oct 24, 2017 at 12:25 GMT


Because deploy.sh is specific to gce and gke, federation e2e tests cannot run as documented on Azure (or aws).

Other documentation on how to run federation tests manually is not available

Member

irfanurrehman commented Oct 31, 2017

Comment by xtophs
Tuesday Oct 24, 2017 at 12:25 GMT


Because deploy.sh is specific to gce and gke, federation e2e tests cannot run as documented on Azure (or aws).

Other documentation on how to run federation tests manually is not available

@irfanurrehman

This comment has been minimized.

Show comment
Hide comment
@irfanurrehman
Member

irfanurrehman commented Oct 31, 2017

@fejta-bot

This comment has been minimized.

Show comment
Hide comment
@fejta-bot

fejta-bot Jan 29, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot commented Jan 29, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@fejta-bot

This comment has been minimized.

Show comment
Hide comment
@fejta-bot

fejta-bot May 24, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot commented May 24, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@fejta-bot

This comment has been minimized.

Show comment
Hide comment
@fejta-bot

fejta-bot Jun 23, 2018

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

fejta-bot commented Jun 23, 2018

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

@fejta-bot

This comment has been minimized.

Show comment
Hide comment
@fejta-bot

fejta-bot Jul 23, 2018

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

fejta-bot commented Jul 23, 2018

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

@luceos

This comment has been minimized.

Show comment
Hide comment
@luceos

luceos Jul 25, 2018

/reopen
/remove-lifecycle rotten

Isn't this issue about the state of federation?

luceos commented Jul 25, 2018

/reopen
/remove-lifecycle rotten

Isn't this issue about the state of federation?

@k8s-ci-robot

This comment has been minimized.

Show comment
Hide comment
@k8s-ci-robot

k8s-ci-robot Jul 25, 2018

Contributor

@luceos: you can't re-open an issue/PR unless you authored it or you are assigned to it.

In response to this:

/reopen
/remove-lifecycle rotten

Isn't this issue about the state of federation?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Contributor

k8s-ci-robot commented Jul 25, 2018

@luceos: you can't re-open an issue/PR unless you authored it or you are assigned to it.

In response to this:

/reopen
/remove-lifecycle rotten

Isn't this issue about the state of federation?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@nikhita

This comment has been minimized.

Show comment
Hide comment
@nikhita

nikhita Aug 27, 2018

Member

/assign

Member

nikhita commented Aug 27, 2018

/assign

@nikhita

This comment has been minimized.

Show comment
Hide comment
@nikhita

nikhita Aug 27, 2018

Member

/reopen

Member

nikhita commented Aug 27, 2018

/reopen

@k8s-ci-robot k8s-ci-robot reopened this Aug 27, 2018

@nikhita

This comment has been minimized.

Show comment
Hide comment
@nikhita

nikhita Aug 27, 2018

Member

/unassign

Member

nikhita commented Aug 27, 2018

/unassign

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment