Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

All Masters Never Become Ready #35

Closed
dgoodwin opened this issue Nov 13, 2018 · 8 comments
Closed

All Masters Never Become Ready #35

dgoodwin opened this issue Nov 13, 2018 · 8 comments

Comments

@dgoodwin
Copy link
Contributor

Multiple folks on our team are seeing this today.

Using an image built from: openshift/installer@08018ca

We end up completing install, mostly:

level=debug msg="API not up yet: the server could not find the requested resource"
level=debug msg="API not up yet: Get https://dgoodwin1-api.new-installer.openshift.com:6443/version?timeout=32s: dial tcp 54.82.254.148:6443: connect: connection refused"                                         
level=debug msg="API not up yet: Get https://dgoodwin1-api.new-installer.openshift.com:6443/version?timeout=32s: dial tcp 52.1.146.45:6443: connect: connection refused"                                           
level=debug msg="API not up yet: Get https://dgoodwin1-api.new-installer.openshift.com:6443/version?timeout=32s: dial tcp 54.82.254.148:6443: connect: connection refused"                                         
level=debug msg="API not up yet: Get https://dgoodwin1-api.new-installer.openshift.com:6443/version?timeout=32s: dial tcp 52.3.185.104:6443: connect: connection refused"                                          
level=debug msg="API not up yet: Get https://dgoodwin1-api.new-installer.openshift.com:6443/version?timeout=32s: dial tcp 54.159.140.198:6443: connect: connection refused"                                        
level=debug msg="API not up yet: Get https://dgoodwin1-api.new-installer.openshift.com:6443/version?timeout=32s: dial tcp 34.197.191.119:6443: connect: connection refused"                                        
level=debug msg="API not up yet: Get https://dgoodwin1-api.new-installer.openshift.com:6443/version?timeout=32s: dial tcp 54.158.200.103:6443: connect: connection refused"                                        
level=debug msg="API not up yet: Get https://dgoodwin1-api.new-installer.openshift.com:6443/version?timeout=32s: dial tcp 52.1.146.45:6443: connect: connection refused"                                           
level=debug msg="API not up yet: Get https://dgoodwin1-api.new-installer.openshift.com:6443/version?timeout=32s: dial tcp 54.82.254.148:6443: connect: connection refused"                                         
level=debug msg="API not up yet: Get https://dgoodwin1-api.new-installer.openshift.com:6443/version?timeout=32s: dial tcp 52.3.185.104:6443: connect: connection refused"                                          
level=debug msg="API not up yet: Get https://dgoodwin1-api.new-installer.openshift.com:6443/version?timeout=32s: dial tcp 54.159.140.198:6443: connect: connection refused"                                        
level=info msg="API v1.11.0+d4cacc0 up"
level=debug msg="added kube-scheduler.1566c06fe7813814: ip-10-0-0-76_c653648b-e76c-11e8-8d31-0ec82e7bcc70 became leader"                                                                                           
level=debug msg="added kube-controller-manager.1566c070f2c6967b: ip-10-0-0-76_c6645265-e76c-11e8-860d-0ec82e7bcc70 became leader"                                                                                  
level=warning msg="RetryWatcher - getting event failed! Re-creating the watcher. Last RV: 219"
level=warning msg="Failed to connect events watcher: Get https://dgoodwin1-api.new-installer.openshift.com:6443/api/v1/namespaces/kube-system/events?resourceVersion=219&watch=true: dial tcp 52.1.146.45:6443: connect: connection refused"
level=warning msg="Failed to connect events watcher: Get https://dgoodwin1-api.new-installer.openshift.com:6443/api/v1/namespaces/kube-system/events?resourceVersion=219&watch=true: dial tcp 54.159.140.198:6443: connect: connection refused"
level=warning msg="Failed to connect events watcher: Get https://dgoodwin1-api.new-installer.openshift.com:6443/api/v1/namespaces/kube-system/events?resourceVersion=219&watch=true: dial tcp 52.1.146.45:6443: connect: connection refused"
level=warning msg="Failed to connect events watcher: Get https://dgoodwin1-api.new-installer.openshift.com:6443/api/v1/namespaces/kube-system/events?resourceVersion=219&watch=true: dial tcp 52.3.185.104:6443: connect: connection refused"
level=warning msg="Failed to connect events watcher: Get https://dgoodwin1-api.new-installer.openshift.com:6443/api/v1/namespaces/kube-system/events?resourceVersion=219&watch=true: dial tcp 34.197.191.119:6443: connect: connection refused"
level=warning msg="Failed to connect events watcher: Get https://dgoodwin1-api.new-installer.openshift.com:6443/api/v1/namespaces/kube-system/events?resourceVersion=219&watch=true: dial tcp 54.158.200.103:6443: connect: connection refused"
level=warning msg="Failed to connect events watcher: Get https://dgoodwin1-api.new-installer.openshift.com:6443/api/v1/namespaces/kube-system/events?resourceVersion=219&watch=true: dial tcp 54.82.254.148:6443: connect: connection refused"
level=error msg="waiting for bootstrap-complete: watch closed before UntilWithoutRetry timeout"
level=info msg="Install complete! Run 'export KUBECONFIG=/output/auth/kubeconfig' to manage your cluster."
level=info msg="After exporting your kubeconfig, run 'oc -h' for a list of OpenShift client commands."

Masters are permanently stuck notready:

$ k get nodes
NAME                          STATUS     ROLES     AGE       VERSION
ip-10-0-19-84.ec2.internal    NotReady   master    1h        v1.11.0+d4cacc0
ip-10-0-38-133.ec2.internal   NotReady   master    1h        v1.11.0+d4cacc0
ip-10-0-8-183.ec2.internal    NotReady   master    1h        v1.11.0+d4cacc0

Their status contains a condition like:

    - lastHeartbeatTime: 2018-11-13T18:59:05Z
      lastTransitionTime: 2018-11-13T17:52:16Z       
      message: 'runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady
        message:Network plugin returns error: cni config uninitialized'
      reason: KubeletNotReady            
      status: "False"
      type: Ready

With a similar error in the kubelet systemd logs:

Nov 13 19:00:12 ip-10-0-8-183 hyperkube[841]: E1113 19:00:12.951372     841 kubelet.go:2101] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni config uninitialized

cluster-network-operator pod logs show:

E1113 18:53:06.044313       1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get https://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.12.190:6443: connect: connection refused                      
2018/11/13 18:53:14 Reconciling NetworkConfig /default
@squeed
Copy link
Contributor

squeed commented Nov 13, 2018

Odd - it got a network configuration, but just... hung. It uses the stock operator-sdk. @shawn-hurley, any thoughts how this could happen?

What does kubectl get -n openshift-sdn pods show? Did the operator make any progress at all?

@dgoodwin
Copy link
Contributor Author

No openshift-sdn namespace in this cluster. Is that bad?

@squeed
Copy link
Contributor

squeed commented Nov 13, 2018

The next time this happens, watch the cluster-network-operator to see if its making progress. If not, we need to find out why.

@dgoodwin
Copy link
Contributor Author

dgoodwin commented Nov 14, 2018 •
edited
Loading

Still happening this morning:

$ klf cluster-network-operator-z8bc6                                                                                                                     [12/3643]
2018/11/14 11:38:36 Go Version: go1.10.3                                                                                                                                                                           
2018/11/14 11:38:36 Go OS/Arch: linux/amd64                                                                                                                                                                         
2018/11/14 11:38:36 operator-sdk Version: v0.1.0                                                                                                                                                                   
2018/11/14 11:38:36 overriding kubernetes api to https://dgoodwin1-api.new-installer.openshift.com:6443                                                                                                             
time="2018-11-14T11:38:36Z" level=info msg="trying to become the leader"                                                                                                                                           
time="2018-11-14T11:38:38Z" level=info msg="No pre-existing lock was found."                                                                                                                                        
time="2018-11-14T11:38:38Z" level=info msg="Became the leader."                                                                                                                                                    
2018/11/14 11:38:39 Registering Components.                                                                                                                                                                         
2018/11/14 11:38:39 Starting the Cmd.                                                                                                                                                                               
2018/11/14 11:38:39 Reconciling NetworkConfig /default               
2018/11/14 11:38:40 Reconciling NetworkConfig /default               
2018/11/14 11:38:41 Reconciling NetworkConfig /default               
2018/11/14 11:38:42 Reconciling NetworkConfig /default               
2018/11/14 11:38:43 Reconciling NetworkConfig /default
2018/11/14 11:38:44 Reconciling NetworkConfig /default
2018/11/14 11:38:45 Reconciling NetworkConfig /default
2018/11/14 11:38:46 Reconciling NetworkConfig /default
2018/11/14 11:38:47 Reconciling NetworkConfig /default
2018/11/14 11:38:48 Reconciling NetworkConfig /default
2018/11/14 11:38:51 Reconciling NetworkConfig /default
2018/11/14 11:38:56 Reconciling NetworkConfig /default
2018/11/14 11:39:06 Reconciling NetworkConfig /default
2018/11/14 11:39:27 Reconciling NetworkConfig /default
2018/11/14 11:40:08 Reconciling NetworkConfig /default
2018/11/14 11:41:30 Reconciling NetworkConfig /default
2018/11/14 11:44:14 Reconciling NetworkConfig /default
2018/11/14 11:49:41 Reconciling NetworkConfig /default
E1114 11:57:59.896544       1 reflector.go:322] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to watch *v1.NetworkConfig: Get
https://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?resourceVersion=255&timeoutSeconds=506&watch=true: dial tcp 10.0.1.209:6443: connect: connection refused
E1114 11:58:00.979228       1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.39.68:6443: connect: connection refused
E1114 11:58:01.991688       1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.1.209:6443: connect: connection refused
E1114 11:58:03.004932       1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.39.68:6443: connect: connection refused
E1114 11:58:04.019185       1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.93.45:6443: connect: connection refused
E1114 11:58:05.079807       1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.27.254:6443: connect: connection refused
E1114 11:58:06.178972       1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.60.129:6443: connect: connection refused
E1114 11:58:07.191016       1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.93.45:6443: connect: connection refused
E1114 11:58:08.203615       1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.27.254:6443: connect: connection refused
E1114 11:58:09.216142       1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.60.129:6443: connect: connection refused
E1114 11:58:10.229218       1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.93.45:6443: connect: connection refused
E1114 11:58:11.242174       1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.27.254:6443: connect: connection refused
E1114 11:58:12.254245       1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.60.129:6443: connect: connection refused
E1114 11:58:14.281861       1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.79.187:6443: connect: connection refused
E1114 11:58:15.294857       1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.1.209:6443: connect: connection refused
E1114 11:58:23.908344       1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: netwo
rkconfigs.networkoperator.openshift.io is forbidden: User "system:serviceaccount:openshift-cluster-network-operator:default" cannot list networkconfigs.networkoperator.openshift.io at the cluster scope: no RBAC p
olicy matched
2018/11/14 11:58:24 Reconciling NetworkConfig /default
2018/11/14 12:00:37 Reconciling NetworkConfig /default
E1114 12:18:29.475698       1 reflector.go:322] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to watch *v1.NetworkConfig: Get
https://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?resourceVersion=2949&timeoutSeconds=383&watch=true: dial tcp 10.0.60.129:6443: connect: connection refuse
d
E1114 12:18:30.488786       1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.93.45:6443: connect: connection refused
E1114 12:18:31.501424       1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.27.254:6443: connect: connection refused
E1114 12:18:32.519018       1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.60.129:6443: connect: connection refused
E1114 12:18:33.531669       1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.93.45:6443: connect: connection refused
E1114 12:18:34.568049       1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.27.254:6443: connect: connection refused
E1114 12:18:35.580845       1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.79.187:6443: connect: connection refused
E1114 12:18:36.593299       1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.1.209:6443: connect: connection refused
E1114 12:18:37.677087       1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.39.68:6443: connect: connection refused
E1114 12:18:38.688812       1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.79.187:6443: connect: connection refused
E1114 12:18:39.703323       1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.1.209:6443: connect: connection refused
E1114 12:18:40.715925       1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.39.68:6443: connect: connection refused
E1114 12:18:41.766929       1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.79.187:6443: connect: connection refused
E1114 12:18:42.777297       1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: the s
erver could not find the requested resource (get networkconfigs.networkoperator.openshift.io)
E1114 12:18:43.790260       1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.39.68:6443: connect: connection refused
E1114 12:18:44.803018       1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: Get h
ttps://dgoodwin1-api.new-installer.openshift.com:6443/apis/networkoperator.openshift.io/v1/networkconfigs?limit=500&resourceVersion=0: dial tcp 10.0.60.129:6443: connect: connection refused
E1114 12:18:53.504680       1 reflector.go:205] github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1.NetworkConfig: the s
erver could not find the requested resource (get networkconfigs.networkoperator.openshift.io)

@dgoodwin
Copy link
Contributor Author

May be an issue on our side, we're running openshift-install in pod containers, and we're seeing failure there but success running the container manually with the exact same image. Please hold on any major investigation.

@squeed
Copy link
Contributor

squeed commented Nov 14, 2018 •
edited
Loading

Yeah, the cluster-network-operator is a very special snowflake. It is run before the bootstrap apiserver is torn down. That means it's running on the masters, talking to the boostrap control plane.

In some sense, it's the canary in the coal mine.

@dgoodwin dgoodwin mentioned this issue Nov 14, 2018
Closed
@squeed
Copy link
Contributor

squeed commented Nov 22, 2018

I've not seen any major issues with the openshift-sdn for a while; closing this. If you see more issues please open another bug.

@dgoodwin
Copy link
Contributor Author

Yeah this was on our end. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dgoodwin @squeed