federatedclusters is not reachable

[kc004]

Kubernetes version: v1.11.3
Cluster joining method: helm
Kubernetes using Rancher

root@:~/go/src/github.com/kubernetes-sigs/federation-v2# ./bin/kubefed2 join on-prem --cluster-context on-prem --host-cluster-context on-prem --add-to-registry --v=2

I1210 06:42:35.370728 13102 join.go:141] Args and flags: name on-prem, host: on-prem, host-system-namespace: federation-system, registry-namespace: kube-multicluster-public, kubeconfig: , cluster-context: on-prem, secret-name: , limited-scope: false, dry-run: false
I1210 06:42:35.372152 13102 join.go:195] Performing preflight checks.
I1210 06:42:35.887897 13102 join.go:283] Registering cluster: on-prem with the cluster registry.
I1210 06:42:36.077257 13102 join.go:292] Registered cluster: on-prem with the cluster registry.
I1210 06:42:36.077283 13102 join.go:216] Creating federation-system namespace in joining cluster
I1210 06:42:36.275758 13102 join.go:224] Created federation-system namespace in joining cluster
I1210 06:42:36.275779 13102 join.go:227] Creating cluster credentials secret
I1210 06:42:36.275789 13102 join.go:418] Creating service account in joining cluster: on-prem
I1210 06:42:36.465577 13102 join.go:428] Created service account: on-prem-on-prem in joining cluster: on-prem
I1210 06:42:36.465607 13102 join.go:444] Creating cluster role and binding for service account: on-prem-on-prem in joining cluster: on-prem
I1210 06:42:36.894540 13102 join.go:453] Created cluster role and binding for service account: on-prem-on-prem in joining cluster: on-prem
I1210 06:42:36.894561 13102 join.go:457] Creating common cluster role and binding for service account: on-prem-on-prem in joining cluster: on-prem
I1210 06:42:37.277343 13102 join.go:466] Created common cluster role and binding for service account: on-prem-on-prem in joining cluster: on-prem
I1210 06:42:37.277363 13102 join.go:469] Creating secret in host cluster: on-prem
I1210 06:42:37.646818 13102 join.go:696] Using secret named: on-prem-on-prem-token-nls9h
I1210 06:42:37.835650 13102 join.go:728] Created secret in host cluster named: on-prem-xw2hf
I1210 06:42:37.835673 13102 join.go:478] Created secret in host cluster: on-prem
I1210 06:42:37.835683 13102 join.go:237] Cluster credentials secret created
I1210 06:42:37.835690 13102 join.go:239] Creating federated cluster resource
I1210 06:42:38.027830 13102 join.go:248] Created federated cluster resource

root@:~/go/src/github.com/kubernetes-sigs/federation-v2# ./bin/kubefed2 join aws-fed --cluster-context aws-fed --host-cluster-context on-prem --add-to-registry --v=2
......same as above cluster.......

There were no error while joining the cluster.

When i see federation-controller-manager logs:
E1210 07:53:38.554801 1 reflector.go:205] github.com/kubernetes-sigs/federation-v2/pkg/controller/sync/controller.go:271: Failed to list : the server could not find the requested resource (get federatedjobs.core.federation.k8s.io)
E1210 07:53:38.555751 1 reflector.go:205] github.com/kubernetes-sigs/federation-v2/pkg/controller/sync/controller.go:271: Failed to list : the server could not find the requested resource (get federatedservices.core.federation.k8s.io)
E1210 07:53:38.691743 1 clusterclient.go:122] Failed to list nodes while getting zone names: Get https://kubernetes-api/k8s/clusters/xxxx/api/v1/nodes: x509: certificate signed by unknown authority
W1210 07:53:38.691776 1 controller.go:224] Failed to get zones and region for cluster with client {0xc420cd24b0}: Get https://kubernetes-api/k8s/clusters/xxxx/api/v1/nodes: x509: certificate signed by unknown authority

and when I describe federatedclusters It shows me clusters are unreachable.

kubectl -n federation-system describe federatedclusters

Name: aws-fed
Namespace: federation-system
Labels:
Annotations:
API Version: core.federation.k8s.io/v1alpha1
Kind: FederatedCluster
Metadata:
Creation Timestamp: 2018-12-10T06:43:13Z
Generation: 1
Resource Version: 595684
Self Link: /apis/core.federation.k8s.io/v1alpha1/namespaces/federation-system/federatedclusters/aws-fed
UID: xxxxxx
Spec:
Cluster Ref:
Name: aws-fed
Secret Ref:
Name: aws-fed-p5tvx
Status:
Conditions:
Last Probe Time: 2018-12-10T07:20:14Z
Last Transition Time: 2018-12-10T06:43:27Z
Message: cluster is not reachable
Reason: ClusterNotReachable
Status: True
Type: Offline
Events:

Name: on-prem
Namespace: federation-system
Labels:
Annotations:
API Version: core.federation.k8s.io/v1alpha1
Kind: FederatedCluster
Metadata:
Creation Timestamp: 2018-12-10T06:42:37Z
Generation: 1
Resource Version: 595686
Self Link: /apis/core.federation.k8s.io/v1alpha1/namespaces/federation-system/federatedclusters/on-prem
UID: xxxxxx
Spec:
Cluster Ref:
Name: on-prem
Secret Ref:
Name: on-prem-xw2hf
Status:
Conditions:
Last Probe Time: 2018-12-10T07:20:14Z
Last Transition Time: 2018-12-10T06:42:46Z
Message: cluster is not reachable
Reason: ClusterNotReachable
Status: True
Type: Offline
Events:

[gyliu513]

@kc004 do you mean that you installed the federation using the helm chart? But from the controller log as follows, seems you are still using the old federation image but not canary image, can you please show more detail for how did you install the federation controller?

Failed to list : the server could not find the requested resource (get federatedjobs.core.federation.k8s.io)

[marun]

@kc004 I would also appreciate more detail of the method of installation. I would also note that the
v0.0.4 release is now available and encourage you to attempt deployment with that, as there have been considerable changes since v0.0.3 and previous releases will fedv2 remains in alpha are not supportable.

[kc004]

I have tried it using v0.0.4 and it seems the same issue.

Logs of federation-controller-manager

Federation v2 controller-manager version: version.Info{Version:"v0.0.4", GitCommit:"2cdf5d37240d9b8b33e2715deb75fbb7f9e003ad", GitTreeState:"clean", BuildDate:"2018-12-11T01:26:09Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
I1219 08:09:19.628724 1 controller-manager.go:104] Federation will target all namespaces
I1219 08:09:19.629608 1 controller.go:84] Starting cluster controller
I1219 08:09:19.629721 1 controller.go:65] Starting scheduler controller
I1219 08:09:19.730816 1 controller.go:91] Starting ServiceDNS controller
I1219 08:09:19.731629 1 controller.go:82] Starting IngressDNS controller
I1219 08:09:19.731830 1 controller.go:76] Starting FederatedTypeConfig controller
I1219 08:09:19.733025 1 controller.go:112] Starting "service" DNSEndpoint controller
I1219 08:09:19.733629 1 controller.go:112] Starting "ingress" DNSEndpoint controller
I1219 08:09:19.833162 1 controller.go:123] "service" DNSEndpoint controller synced and ready
I1219 08:09:19.833696 1 controller.go:123] "ingress" DNSEndpoint controller synced and ready
I1219 08:09:24.148905 1 controller.go:132] Running reconcile FederatedTypeConfig for "federation-system/clusterroles.rbac.authorization.k8s.io"
I1219 08:09:24.149828 1 controller.go:110] Starting sync controller for "FederatedClusterRole"
I1219 08:09:24.149851 1 controller.go:270] Started sync controller for "FederatedClusterRole"
I1219 08:09:24.151412 1 controller.go:116] Running reconcile FederatedTypeConfig for "federation-system/clusterroles.rbac.authorization.k8s.io"
I1219 08:09:24.233019 1 controller.go:116] Running reconcile FederatedTypeConfig for "federation-system/clusterroles.rbac.authorization.k8s.io"
I1219 08:09:24.233261 1 controller.go:132] Running reconcile FederatedTypeConfig for "federation-system/clusterroles.rbac.authorization.k8s.io"
I1219 08:09:24.236998 1 controller.go:116] Running reconcile FederatedTypeConfig for "federation-system/clusterroles.rbac.authorization.k8s.io"
I1219 08:09:24.237165 1 controller.go:132] Running reconcile FederatedTypeConfig for "federation-system/clusterroles.rbac.authorization.k8s.io"
I1219 08:09:24.576997 1 controller.go:132] Running reconcile FederatedTypeConfig for "federation-system/configmaps"
I1219 08:09:24.577058 1 controller.go:116] Running reconcile FederatedTypeConfig for "federation-system/configmaps"
I1219 08:09:24.578605 1 controller.go:110] Starting sync controller for "FederatedConfigMap"
I1219 08:09:24.578630 1 controller.go:270] Started sync controller for "FederatedConfigMap"
I1219 08:09:24.629504 1 controller.go:116] Running reconcile FederatedTypeConfig for "federation-system/configmaps"
I1219 08:09:24.629607 1 controller.go:132] Running reconcile FederatedTypeConfig for "federation-system/configmaps"
I1219 08:09:24.633566 1 controller.go:132] Running reconcile FederatedTypeConfig for "federation-system/configmaps"
I1219 08:09:24.633866 1 controller.go:116] Running reconcile FederatedTypeConfig for "federation-system/configmaps"
I1219 08:09:25.035433 1 controller.go:132] Running reconcile FederatedTypeConfig for "federation-system/deployments.apps"
I1219 08:09:25.036311 1 controller.go:110] Starting sync controller for "FederatedDeployment"
I1219 08:09:25.036340 1 controller.go:270] Started sync controller for "FederatedDeployment"
I1219 08:09:25.037597 1 controller.go:116] Running reconcile FederatedTypeConfig for "federation-system/deployments.apps"
I1219 08:09:25.039000 1 controller.go:83] Starting replicaschedulingpreferences controller
I1219 08:09:25.039040 1 controller.go:168] Start plugin with kind FederatedDeployment for scheduling type ReplicaSchedulingPreference
I1219 08:09:25.050641 1 controller.go:116] Running reconcile FederatedTypeConfig for "federation-system/deployments.apps"
E1219 08:09:25.050757 1 reflector.go:205] github.com/kubernetes-sigs/federation-v2/pkg/controller/sync/controller.go:273: Failed to list : the server could not find the requested resource (get federateddeploymentoverrides.primitives.federation.k8s.io)
I1219 08:09:25.128478 1 controller.go:132] Running reconcile FederatedTypeConfig for "federation-system/deployments.apps"
E1219 08:09:25.130045 1 reflector.go:205] github.com/kubernetes-sigs/federation-v2/pkg/schedulingtypes/plugin.go:112: Failed to list : the server could not find the requested resource (get federateddeploymentoverrides.primitives.federation.k8s.io)
I1219 08:09:25.132962 1 controller.go:132] Running reconcile FederatedTypeConfig for "federation-system/deployments.apps"
I1219 08:09:25.133299 1 controller.go:116] Running reconcile FederatedTypeConfig for "federation-system/deployments.apps"
I1219 08:09:25.497739 1 controller.go:116] Running reconcile FederatedTypeConfig for "federation-system/ingresses.extensions"
I1219 08:09:25.497740 1 controller.go:132] Running reconcile FederatedTypeConfig for "federation-system/ingresses.extensions"
I1219 08:09:25.498649 1 controller.go:110] Starting sync controller for "FederatedIngress"
I1219 08:09:25.498675 1 controller.go:270] Started sync controller for "FederatedIngress"
I1219 08:09:25.528936 1 controller.go:132] Running reconcile FederatedTypeConfig for "federation-system/ingresses.extensions"
I1219 08:09:25.988478 1 controller.go:132] Running reconcile FederatedTypeConfig for "federation-system/jobs.batch"
I1219 08:09:25.989713 1 controller.go:110] Starting sync controller for "FederatedJob"
I1219 08:09:25.989738 1 controller.go:270] Started sync controller for "FederatedJob"
I1219 08:09:26.000218 1 controller.go:116] Running reconcile FederatedTypeConfig for "federation-system/jobs.batch"
I1219 08:09:26.498064 1 controller.go:116] Running reconcile FederatedTypeConfig for "federation-system/namespaces"
I1219 08:09:26.499129 1 controller.go:110] Starting sync controller for "Namespace"
I1219 08:09:26.499157 1 controller.go:270] Started sync controller for "Namespace"
I1219 08:09:26.504744 1 controller.go:116] Running reconcile FederatedTypeConfig for "federation-system/namespaces"
I1219 08:09:26.504917 1 controller.go:132] Running reconcile FederatedTypeConfig for "federation-system/namespaces"
I1219 08:09:26.517287 1 controller.go:116] Running reconcile FederatedTypeConfig for "federation-system/namespaces"
I1219 08:09:26.519109 1 controller.go:132] Running reconcile FederatedTypeConfig for "federation-system/namespaces"
I1219 08:09:27.004466 1 controller.go:116] Running reconcile FederatedTypeConfig for "federation-system/replicasets.apps"
I1219 08:09:27.004494 1 controller.go:168] Start plugin with kind FederatedReplicaSet for scheduling type ReplicaSchedulingPreference
I1219 08:09:27.017128 1 controller.go:132] Running reconcile FederatedTypeConfig for "federation-system/replicasets.apps"
I1219 08:09:27.028397 1 controller.go:110] Starting sync controller for "FederatedReplicaSet"
I1219 08:09:27.029290 1 controller.go:270] Started sync controller for "FederatedReplicaSet"
E1219 08:09:27.058526 1 reflector.go:205] github.com/kubernetes-sigs/federation-v2/pkg/schedulingtypes/plugin.go:111: Failed to list : the server could not find the requested resource (get federatedreplicasets.primitives.federation.k8s.io)
E1219 08:09:27.058665 1 reflector.go:205] github.com/kubernetes-sigs/federation-v2/pkg/schedulingtypes/plugin.go:113: Failed to list : the server could not find the requested resource (get federatedreplicasetplacements.primitives.federation.k8s.io)
E1219 08:09:27.058847 1 reflector.go:205] github.com/kubernetes-sigs/federation-v2/pkg/controller/sync/placement/resource.go:54: Failed to list : the server could not find the requested resource (get federatedreplicasetplacements.primitives.federation.k8s.io)
E1219 08:09:27.058912 1 reflector.go:205] github.com/kubernetes-sigs/federation-v2/pkg/schedulingtypes/plugin.go:112: Failed to list : the server could not find the requested resource (get federatedreplicasetoverrides.primitives.federation.k8s.io)
E1219 08:09:27.058916 1 reflector.go:205] github.com/kubernetes-sigs/federation-v2/pkg/controller/sync/controller.go:271: Failed to list : the server could not find the requested resource (get federatedreplicasets.primitives.federation.k8s.io)
E1219 08:09:27.058950 1 reflector.go:205] github.com/kubernetes-sigs/federation-v2/pkg/controller/sync/controller.go:273: Failed to list : the server could not find the requested resource (get federatedreplicasetoverrides.primitives.federation.k8s.io)
I1219 08:09:27.076732 1 controller.go:132] Running reconcile FederatedTypeConfig for "federation-system/replicasets.apps"
I1219 08:09:27.528253 1 controller.go:132] Running reconcile FederatedTypeConfig for "federation-system/secrets"
I1219 08:09:27.529189 1 controller.go:110] Starting sync controller for "FederatedSecret"
I1219 08:09:27.529228 1 controller.go:270] Started sync controller for "FederatedSecret"
I1219 08:09:27.539616 1 controller.go:116] Running reconcile FederatedTypeConfig for "federation-system/secrets"
I1219 08:09:28.030791 1 controller.go:116] Running reconcile FederatedTypeConfig for "federation-system/serviceaccounts"
I1219 08:09:28.031645 1 controller.go:110] Starting sync controller for "FederatedServiceAccount"
I1219 08:09:28.031692 1 controller.go:270] Started sync controller for "FederatedServiceAccount"
I1219 08:09:28.129408 1 controller.go:132] Running reconcile FederatedTypeConfig for "federation-system/serviceaccounts"
I1219 08:09:28.528437 1 controller.go:132] Running reconcile FederatedTypeConfig for "federation-system/services"
I1219 08:09:28.529392 1 controller.go:110] Starting sync controller for "FederatedService"
I1219 08:09:28.529433 1 controller.go:270] Started sync controller for "FederatedService"
I1219 08:09:28.534314 1 controller.go:116] Running reconcile FederatedTypeConfig for "federation-system/services"
I1219 08:09:29.018783 1 federated_informer.go:214] Cluster federation-system/aws-fed not added; it is not ready.
I1219 08:09:29.018936 1 federated_informer.go:214] Cluster federation-system/aws-fed not added; it is not ready.
I1219 08:09:59.760982 1 controller.go:200] There is no status stored for cluster: aws-fed before
E1219 08:09:59.769968 1 clusterclient.go:122] Failed to list nodes while getting zone names: Get https://aaa.com/k8s/clusters/c-btjar/api/v1/nodes: x509: certificate signed by unknown authority
W1219 08:09:59.769988 1 controller.go:224] Failed to get zones and region for cluster with client {0xc420628960}: Get https://aaa.com/k8s/clusters/c-btjar/api/v1/nodes: x509: certificate signed by unknown authority
I1219 08:09:59.782458 1 controller.go:200] There is no status stored for cluster: on-prem before
E1219 08:09:59.837128 1 clusterclient.go:122] Failed to list nodes while getting zone names: Get https://aaa.com/k8s/clusters/c-qqtam/api/v1/nodes: x509: certificate signed by unknown authority
W1219 08:09:59.837154 1 controller.go:224] Failed to get zones and region for cluster with client {0xc4206eae10}: Get https://aaa.com/k8s/clusters/c-qqtam/api/v1/nodes: x509: certificate signed by unknown authority

[marun]

It looks like initialization of the primitive crds isn't yet complete when the controller is started, but become initialized eventually such that the Failed to list errors are temporary. Are you still seeing the clusters not become ready?

[marun]

Given the lack of activity on this issue, I'm closing. Feel free to reopen if necessary.