api server not starting

[thomasmckay]

Version

$ bin/openshift-install version
bin/openshift-install v0.7.0-master-54-gac006ae671a645553d58c8a29c676968dfa3d85f-dirty

Platform (aws|libvirt|openstack):

libvirt

What happened?

Instructed to paste logs:

from bootstrap VM

$ sudo crictl ps -a
CONTAINER ID        IMAGE                                                                                                                                           CREATED             STATE               NAME                       ATTEMPT
ca38ddfc54769       registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-20-113525@sha256:38d7c6702c2b09cb9d2bfbfddc9af6dea33045239390edfb1f38f7e39cf4cb20   2 minutes ago       Running             kube-controller-manager    3
22fbd4e14ebbd       registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-20-113525@sha256:e3071632e217ac2a94fdb452b8c9e7a88f431c4a7c7c904e0e2476a33b98f92a   2 minutes ago       Running             kube-apiserver             3
f860ff762c6ef       registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-20-113525@sha256:38d7c6702c2b09cb9d2bfbfddc9af6dea33045239390edfb1f38f7e39cf4cb20   2 minutes ago       Running             kube-scheduler             3
321ac02dda4c9       registry.svc.ci.openshift.org/openshift/origin-release@sha256:1863d9f42adb164a30d004f252c840141c00d5b2855dbc041681750e06fb22ac                  2 minutes ago       Running             cluster-version-operator   3
b0b3c7769ec63       registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-20-113525@sha256:38d7c6702c2b09cb9d2bfbfddc9af6dea33045239390edfb1f38f7e39cf4cb20   22 minutes ago      Exited              kube-scheduler             2
64b3442255997       registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-20-113525@sha256:e3071632e217ac2a94fdb452b8c9e7a88f431c4a7c7c904e0e2476a33b98f92a   22 minutes ago      Exited              kube-apiserver             2
2e9b8d7777b42       registry.svc.ci.openshift.org/openshift/origin-release@sha256:1863d9f42adb164a30d004f252c840141c00d5b2855dbc041681750e06fb22ac                  22 minutes ago      Exited              cluster-version-operator   2
14ec81cf02ed0       registry.svc.ci.openshift.org/openshift/origin-v4.0-2018-12-20-113525@sha256:38d7c6702c2b09cb9d2bfbfddc9af6dea33045239390edfb1f38f7e39cf4cb20   22 minutes ago      Exited              kube-controller-manager    2

on master-0 VM

$ journalctl -f -u kubelet.service 
-- Logs begin at Thu 2018-12-20 18:20:42 UTC. --
Dec 20 19:25:40 quay-master-0 hyperkube[5343]: E1220 19:25:40.122664    5343 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Get https://quay-api.tt.testing:6443/api/v1/nodes?fieldSelector=metadata.name%3Dquay-master-0&limit=500&resourceVersion=0: dial tcp 192.168.126.10:6443: connect: connection refused
Dec 20 19:25:40 quay-master-0 hyperkube[5343]: E1220 19:25:40.122793    5343 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://quay-api.tt.testing:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dquay-master-0&limit=500&resourceVersion=0: dial tcp 192.168.126.11:6443: connect: connection refused
Dec 20 19:25:40 quay-master-0 hyperkube[5343]: E1220 19:25:40.123922    5343 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Get https://quay-api.tt.testing:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.126.11:6443: connect: connection refused
Dec 20 19:25:41 quay-master-0 hyperkube[5343]: E1220 19:25:41.124050    5343 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Get https://quay-api.tt.testing:6443/api/v1/nodes?fieldSelector=metadata.name%3Dquay-master-0&limit=500&resourceVersion=0: dial tcp 192.168.126.10:6443: connect: connection refused
Dec 20 19:25:41 quay-master-0 hyperkube[5343]: E1220 19:25:41.125162    5343 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://quay-api.tt.testing:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dquay-master-0&limit=500&resourceVersion=0: dial tcp 192.168.126.11:6443: connect: connection refused
Dec 20 19:25:41 quay-master-0 hyperkube[5343]: E1220 19:25:41.126065    5343 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Get https://quay-api.tt.testing:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.126.10:6443: connect: connection refused
Dec 20 19:25:42 quay-master-0 hyperkube[5343]: E1220 19:25:42.125311    5343 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Get https://quay-api.tt.testing:6443/api/v1/nodes?fieldSelector=metadata.name%3Dquay-master-0&limit=500&resourceVersion=0: dial tcp 192.168.126.10:6443: connect: connection refused
Dec 20 19:25:42 quay-master-0 hyperkube[5343]: E1220 19:25:42.126575    5343 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://quay-api.tt.testing:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dquay-master-0&limit=500&resourceVersion=0: dial tcp 192.168.126.10:6443: connect: connection refused
Dec 20 19:25:42 quay-master-0 hyperkube[5343]: E1220 19:25:42.127795    5343 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Get https://quay-api.tt.testing:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.126.10:6443: connect: connection refused
Dec 20 19:25:42 quay-master-0 hyperkube[5343]: E1220 19:25:42.221609    5343 kubelet.go:2101] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni config uninitialized
Dec 20 19:25:43 quay-master-0 hyperkube[5343]: E1220 19:25:43.126417    5343 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Get https://quay-api.tt.testing:6443/api/v1/nodes?fieldSelector=metadata.name%3Dquay-master-0&limit=500&resourceVersion=0: dial tcp 192.168.126.11:6443: connect: connection refused
Dec 20 19:25:43 quay-master-0 hyperkube[5343]: E1220 19:25:43.127979    5343 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://quay-api.tt.testing:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dquay-master-0&limit=500&resourceVersion=0: dial tcp 192.168.126.11:6443: connect: connection refused
Dec 20 19:25:43 quay-master-0 hyperkube[5343]: E1220 19:25:43.129079    5343 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Get https://quay-api.tt.testing:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.126.10:6443: connect: connection refused
Dec 20 19:25:44 quay-master-0 hyperkube[5343]: E1220 19:25:44.128034    5343 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Get https://quay-api.tt.testing:6443/api/v1/nodes?fieldSelector=metadata.name%3Dquay-master-0&limit=500&resourceVersion=0: dial tcp 192.168.126.11:6443: connect: connection refused
Dec 20 19:25:44 quay-master-0 hyperkube[5343]: E1220 19:25:44.129607    5343 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://quay-api.tt.testing:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dquay-master-0&limit=500&resourceVersion=0: dial tcp 192.168.126.11:6443: connect: connection refused
Dec 20 19:25:44 quay-master-0 hyperkube[5343]: E1220 19:25:44.130842    5343 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Get https://quay-api.tt.testing:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.126.10:6443: connect: connection refused
Dec 20 19:25:45 quay-master-0 hyperkube[5343]: E1220 19:25:45.129600    5343 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Get https://quay-api.tt.testing:6443/api/v1/nodes?fieldSelector=metadata.name%3Dquay-master-0&limit=500&resourceVersion=0: dial tcp 192.168.126.11:6443: connect: connection refused
Dec 20 19:25:45 quay-master-0 hyperkube[5343]: E1220 19:25:45.130967    5343 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://quay-api.tt.testing:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dquay-master-0&limit=500&resourceVersion=0: dial tcp 192.168.126.10:6443: connect: connection refused
Dec 20 19:25:45 quay-master-0 hyperkube[5343]: E1220 19:25:45.132330    5343 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Get https://quay-api.tt.testing:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.126.10:6443: connect: connection refused
Dec 20 19:25:46 quay-master-0 hyperkube[5343]: E1220 19:25:46.131203    5343 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Get https://quay-api.tt.testing:6443/api/v1/nodes?fieldSelector=metadata.name%3Dquay-master-0&limit=500&resourceVersion=0: dial tcp 192.168.126.11:6443: connect: connection refused
Dec 20 19:25:46 quay-master-0 hyperkube[5343]: E1220 19:25:46.132667    5343 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://quay-api.tt.testing:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dquay-master-0&limit=500&resourceVersion=0: dial tcp 192.168.126.11:6443: connect: connection refused
Dec 20 19:25:46 quay-master-0 hyperkube[5343]: E1220 19:25:46.134166    5343 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Get https://quay-api.tt.testing:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.126.11:6443: connect: connection refused
Dec 20 19:25:46 quay-master-0 hyperkube[5343]: I1220 19:25:46.731764    5343 kubelet_node_status.go:269] Setting node annotation to enable volume controller attach/detach
Dec 20 19:25:46 quay-master-0 hyperkube[5343]: I1220 19:25:46.733739    5343 kubelet_node_status.go:79] Attempting to register node quay-master-0
Dec 20 19:25:46 quay-master-0 hyperkube[5343]: E1220 19:25:46.734865    5343 kubelet_node_status.go:103] Unable to register node "quay-master-0" with API server: Post https://quay-api.tt.testing:6443/api/v1/nodes: dial tcp 192.168.126.11:6443: connect: connection refused
Dec 20 19:25:47 quay-master-0 hyperkube[5343]: E1220 19:25:47.133615    5343 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Get https://quay-api.tt.testing:6443/api/v1/nodes?fieldSelector=metadata.name%3Dquay-master-0&limit=500&resourceVersion=0: dial tcp 192.168.126.11:6443: connect: connection refused
Dec 20 19:25:47 quay-master-0 hyperkube[5343]: E1220 19:25:47.134862    5343 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://quay-api.tt.testing:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dquay-master-0&limit=500&resourceVersion=0: dial tcp 192.168.126.10:6443: connect: connection refused
Dec 20 19:25:47 quay-master-0 hyperkube[5343]: E1220 19:25:47.135792    5343 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Get https://quay-api.tt.testing:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.126.10:6443: connect: connection refused



sudo crictl logs 741a288faeef7
....snip of repeats....
E1220 19:25:31.807286       1 reflector.go:136] github.com/openshift/client-go/security/informers/externalversions/factory.go:101: Failed to list *v1.SecurityContextConstraints: the server could not find the requested resource (get securitycontextconstraints.security.openshift.io)
E1220 19:25:31.808397       1 reflector.go:136] github.com/openshift/origin/pkg/quota/generated/informers/internalversion/factory.go:101: Failed to list *quota.ClusterResourceQuota: the server could not find the requested resource (get clusterresourcequotas.quota.openshift.io)
E1220 19:25:32.404660       1 oauth_apiserver.go:294] the server could not find the requested resource (post oauthclients.oauth.openshift.io)
E1220 19:25:32.806099       1 reflector.go:136] github.com/openshift/client-go/oauth/informers/externalversions/factory.go:101: Failed to list *v1.OAuthClient: the server could not find the requested resource (get oauthclients.oauth.openshift.io)
E1220 19:25:32.806875       1 reflector.go:136] github.com/openshift/client-go/user/informers/externalversions/factory.go:101: Failed to list *v1.Group: the server could not find the requested resource (get groups.user.openshift.io)
E1220 19:25:32.807963       1 reflector.go:136] github.com/openshift/client-go/security/informers/externalversions/factory.go:101: Failed to list *v1.SecurityContextConstraints: the server could not find the requested resource (get securitycontextconstraints.security.openshift.io)
E1220 19:25:32.809151       1 reflector.go:136] github.com/openshift/origin/pkg/quota/generated/informers/internalversion/factory.go:101: Failed to list *quota.ClusterResourceQuota: the server could not find the requested resource (get clusterresourcequotas.quota.openshift.io)
E1220 19:25:33.404232       1 oauth_apiserver.go:294] the server could not find the requested resource (post oauthclients.oauth.openshift.io)
E1220 19:25:33.807349       1 reflector.go:136] github.com/openshift/client-go/oauth/informers/externalversions/factory.go:101: Failed to list *v1.OAuthClient: the server could not find the requested resource (get oauthclients.oauth.openshift.io)
E1220 19:25:33.808097       1 reflector.go:136] github.com/openshift/client-go/user/informers/externalversions/factory.go:101: Failed to list *v1.Group: the server could not find the requested resource (get groups.user.openshift.io)
E1220 19:25:33.809239       1 reflector.go:136] github.com/openshift/client-go/security/informers/externalversions/factory.go:101: Failed to list *v1.SecurityContextConstraints: the server could not find the requested resource (get securitycontextconstraints.security.openshift.io)
E1220 19:25:33.810231       1 reflector.go:136] github.com/openshift/origin/pkg/quota/generated/informers/internalversion/factory.go:101: Failed to list *quota.ClusterResourceQuota: the server could not find the requested resource (get clusterresourcequotas.quota.openshift.io)
E1220 19:25:34.404582       1 oauth_apiserver.go:294] the server could not find the requested resource (post oauthclients.oauth.openshift.io)
I1220 19:25:34.599481       1 controller.go:158] Shutting down kubernetes service endpoint reconciler
E1220 19:25:34.599672       1 clusterquotamapping.go:134] timed out waiting for caches to sync
I1220 19:25:34.599723       1 clusterquotamapping.go:135] Shutting down ClusterQuotaMappingController controller
I1220 19:25:34.600435       1 naming_controller.go:295] Shutting down NamingConditionController
I1220 19:25:34.600584       1 customresource_discovery_controller.go:210] Shutting down DiscoveryController
I1220 19:25:34.600594       1 autoregister_controller.go:160] Shutting down autoregister controller
I1220 19:25:34.600600       1 establishing_controller.go:84] Shutting down EstablishingController
I1220 19:25:34.600606       1 apiservice_controller.go:102] Shutting down APIServiceRegistrationController
I1220 19:25:34.600610       1 crd_finalizer.go:254] Shutting down CRDFinalizer
I1220 19:25:34.600615       1 available_controller.go:350] Shutting down AvailableConditionController
I1220 19:25:34.600620       1 crdregistration_controller.go:143] Shutting down crd-autoregister controller
I1220 19:25:34.600816       1 serve.go:156] Stopped listening on [::]:6443
I1220 19:25:34.600821       1 controller.go:90] Shutting down OpenAPI AggregationController
E1220 19:25:34.601737       1 oauth_apiserver.go:294] the server could not find the requested resource (post oauthclients.oauth.openshift.io)
E1220 19:25:34.605302       1 controller.go:160] no master IPs were listed in storage, refusing to erase all endpoints for the kubernetes service
F1220 19:25:34.605349       1 cmd.go:68] `kube-apiserver [--admission-control-config-file=/tmp/kubeapiserver-admission-config.yaml981904871 --allow-privileged=true --anonymous-auth=false --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=openshift.local.log/audit.log --audit-policy-file=openshift.local.log/audit-policy.yaml --authorization-mode=RBAC --authorization-mode=Node --bind-address=0.0.0.0 --client-ca-file=/etc/kubernetes/secrets/kube-ca.crt --cors-allowed-origins=//127\.0\.0\.1(:|$) --cors-allowed-origins=//localhost(:|$) --enable-admission-plugins=ExternalIPRanger --enable-admission-plugins=openshift.io/RestrictedEndpointsAdmission --enable-logs-handler=false --enable-swagger-ui=true --endpoint-reconciler-type=lease --etcd-cafile=/etc/kubernetes/secrets/etcd-client-ca.crt --etcd-certfile=/etc/kubernetes/secrets/etcd-client.crt --etcd-keyfile=/etc/kubernetes/secrets/etcd-client.key --etcd-prefix=openshift.io --etcd-servers=https://quay-etcd-0.tt.testing:2379 --insecure-port=0 --kubelet-certificate-authority= --kubelet-client-certificate=/etc/kubernetes/secrets/apiserver.crt --kubelet-client-key=/etc/kubernetes/secrets/apiserver.key --kubelet-https=true --kubelet-preferred-address-types=Hostname --kubelet-preferred-address-types=InternalIP --kubelet-preferred-address-types=ExternalIP --kubelet-read-only-port=0 --kubernetes-service-node-port=0 --max-mutating-requests-inflight=600 --max-requests-inflight=1200 --min-request-timeout=3600 --proxy-client-cert-file=/etc/kubernetes/secrets/apiserver-proxy.crt --proxy-client-key-file=/etc/kubernetes/secrets/apiserver-proxy.key --requestheader-allowed-names=kube-apiserver-proxy --requestheader-allowed-names=system:kube-apiserver-proxy --requestheader-allowed-names=system:openshift-aggregator --requestheader-client-ca-file=/etc/kubernetes/secrets/aggregator-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-cluster-ip-range=172.30.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --storage-media-type=application/vnd.kubernetes.protobuf --tls-cert-file=/etc/kubernetes/secrets/apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305 --tls-cipher-suites=TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA --tls-cipher-suites=TLS_RSA_WITH_AES_128_GCM_SHA256 --tls-cipher-suites=TLS_RSA_WITH_AES_256_GCM_SHA384 --tls-cipher-suites=TLS_RSA_WITH_AES_128_CBC_SHA --tls-cipher-suites=TLS_RSA_WITH_AES_256_CBC_SHA --tls-min-version=VersionTLS12 --tls-private-key-file=/etc/kubernetes/secrets/apiserver.key]` exited
 
 
sudo crictl logs 64b3442255997
...snip...
E1220 19:34:35.289032       1 oauth_apiserver.go:294] the server could not find the requested resource (post oauthclients.oauth.openshift.io)
E1220 19:34:35.946040       1 reflector.go:136] github.com/openshift/client-go/security/informers/externalversions/factory.go:101: Failed to list *v1.SecurityContextConstraints: the server could not find the requested resource (get securitycontextconstraints.security.openshift.io)
E1220 19:34:35.946926       1 reflector.go:136] github.com/openshift/client-go/oauth/informers/externalversions/factory.go:101: Failed to list *v1.OAuthClient: the server could not find the requested resource (get oauthclients.oauth.openshift.io)
E1220 19:34:35.948356       1 reflector.go:136] github.com/openshift/origin/pkg/quota/generated/informers/internalversion/factory.go:101: Failed to list *quota.ClusterResourceQuota: the server could not find the requested resource (get clusterresourcequotas.quota.openshift.io)
E1220 19:34:35.949843       1 reflector.go:136] github.com/openshift/client-go/user/informers/externalversions/factory.go:101: Failed to list *v1.Group: the server could not find the requested resource (get groups.user.openshift.io)
E1220 19:34:36.289235       1 oauth_apiserver.go:294] the server could not find the requested resource (post oauthclients.oauth.openshift.io)
E1220 19:34:36.947093       1 reflector.go:136] github.com/openshift/client-go/security/informers/externalversions/factory.go:101: Failed to list *v1.SecurityContextConstraints: the server could not find the requested resource (get securitycontextconstraints.security.openshift.io)
E1220 19:34:36.947888       1 reflector.go:136] github.com/openshift/client-go/oauth/informers/externalversions/factory.go:101: Failed to list *v1.OAuthClient: the server could not find the requested resource (get oauthclients.oauth.openshift.io)
E1220 19:34:36.949034       1 reflector.go:136] github.com/openshift/origin/pkg/quota/generated/informers/internalversion/factory.go:101: Failed to list *quota.ClusterResourceQuota: the server could not find the requested resource (get clusterresourcequotas.quota.openshift.io)
E1220 19:34:36.950594       1 reflector.go:136] github.com/openshift/client-go/user/informers/externalversions/factory.go:101: Failed to list *v1.Group: the server could not find the requested resource (get groups.user.openshift.io)
E1220 19:34:37.289458       1 oauth_apiserver.go:294] the server could not find the requested resource (post oauthclients.oauth.openshift.io)
E1220 19:34:37.459056       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate has expired or is not yet valid, x509: certificate has expired or is not yet valid]
E1220 19:34:37.948247       1 reflector.go:136] github.com/openshift/client-go/security/informers/externalversions/factory.go:101: Failed to list *v1.SecurityContextConstraints: the server could not find the requested resource (get securitycontextconstraints.security.openshift.io)
E1220 19:34:37.948949       1 reflector.go:136] github.com/openshift/client-go/oauth/informers/externalversions/factory.go:101: Failed to list *v1.OAuthClient: the server could not find the requested resource (get oauthclients.oauth.openshift.io)
E1220 19:34:37.949941       1 reflector.go:136] github.com/openshift/origin/pkg/quota/generated/informers/internalversion/factory.go:101: Failed to list *quota.ClusterResourceQuota: the server could not find the requested resource (get clusterresourcequotas.quota.openshift.io)
E1220 19:34:37.951330       1 reflector.go:136] github.com/openshift/client-go/user/informers/externalversions/factory.go:101: Failed to list *v1.Group: the server could not find the requested resource (get groups.user.openshift.io)
E1220 19:34:38.289251       1 oauth_apiserver.go:294] the server could not find the requested resource (post oauthclients.oauth.openshift.io)
E1220 19:34:38.949351       1 reflector.go:136] github.com/openshift/client-go/security/informers/externalversions/factory.go:101: Failed to list *v1.SecurityContextConstraints: the server could not find the requested resource (get securitycontextconstraints.security.openshift.io)
E1220 19:34:38.950267       1 reflector.go:136] github.com/openshift/client-go/oauth/informers/externalversions/factory.go:101: Failed to list *v1.OAuthClient: the server could not find the requested resource (get oauthclients.oauth.openshift.io)
E1220 19:34:38.952615       1 reflector.go:136] github.com/openshift/origin/pkg/quota/generated/informers/internalversion/factory.go:101: Failed to list *quota.ClusterResourceQuota: the server could not find the requested resource (get clusterresourcequotas.quota.openshift.io)
E1220 19:34:38.952683       1 reflector.go:136] github.com/openshift/client-go/user/informers/externalversions/factory.go:101: Failed to list *v1.Group: the server could not find the requested resource (get groups.user.openshift.io)
E1220 19:34:39.289037       1 oauth_apiserver.go:294] the server could not find the requested resource (post oauthclients.oauth.openshift.io)
E1220 19:34:39.672180       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate has expired or is not yet valid, x509: certificate has expired or is not yet valid]
E1220 19:34:39.950133       1 reflector.go:136] github.com/openshift/client-go/security/informers/externalversions/factory.go:101: Failed to list *v1.SecurityContextConstraints: the server could not find the requested resource (get securitycontextconstraints.security.openshift.io)
E1220 19:34:39.951056       1 reflector.go:136] github.com/openshift/client-go/oauth/informers/externalversions/factory.go:101: Failed to list *v1.OAuthClient: the server could not find the requested resource (get oauthclients.oauth.openshift.io)
E1220 19:34:39.953179       1 reflector.go:136] github.com/openshift/origin/pkg/quota/generated/informers/internalversion/factory.go:101: Failed to list *quota.ClusterResourceQuota: the server could not find the requested resource (get clusterresourcequotas.quota.openshift.io)
E1220 19:34:39.954218       1 reflector.go:136] github.com/openshift/client-go/user/informers/externalversions/factory.go:101: Failed to list *v1.Group: the server could not find the requested resource (get groups.user.openshift.io)
E1220 19:34:40.289297       1 oauth_apiserver.go:294] the server could not find the requested resource (post oauthclients.oauth.openshift.io)
E1220 19:34:40.951120       1 reflector.go:136] github.com/openshift/client-go/security/informers/externalversions/factory.go:101: Failed to list *v1.SecurityContextConstraints: the server could not find the requested resource (get securitycontextconstraints.security.openshift.io)
E1220 19:34:40.952314       1 reflector.go:136] github.com/openshift/client-go/oauth/informers/externalversions/factory.go:101: Failed to list *v1.OAuthClient: the server could not find the requested resource (get oauthclients.oauth.openshift.io)
E1220 19:34:40.953937       1 reflector.go:136] github.com/openshift/origin/pkg/quota/generated/informers/internalversion/factory.go:101: Failed to list *quota.ClusterResourceQuota: the server could not find the requested resource (get clusterresourcequotas.quota.openshift.io)
E1220 19:34:40.954849       1 reflector.go:136] github.com/openshift/client-go/user/informers/externalversions/factory.go:101: Failed to list *v1.Group: the server could not find the requested resource (get groups.user.openshift.io)
E1220 19:34:41.289056       1 oauth_apiserver.go:294] the server could not find the requested resource (post oauthclients.oauth.openshift.io)
E1220 19:34:41.952454       1 reflector.go:136] github.com/openshift/client-go/security/informers/externalversions/factory.go:101: Failed to list *v1.SecurityContextConstraints: the server could not find the requested resource (get securitycontextconstraints.security.openshift.io)
E1220 19:34:41.953179       1 reflector.go:136] github.com/openshift/client-go/oauth/informers/externalversions/factory.go:101: Failed to list *v1.OAuthClient: the server could not find the requested resource (get oauthclients.oauth.openshift.io)
E1220 19:34:41.954782       1 reflector.go:136] github.com/openshift/origin/pkg/quota/generated/informers/internalversion/factory.go:101: Failed to list *quota.ClusterResourceQuota: the server could not find the requested resource (get clusterresourcequotas.quota.openshift.io)
E1220 19:34:41.955595       1 reflector.go:136] github.com/openshift/client-go/user/informers/externalversions/factory.go:101: Failed to list *v1.Group: the server could not find the requested resource (get groups.user.openshift.io)
E1220 19:34:42.289559       1 oauth_apiserver.go:294] the server could not find the requested resource (post oauthclients.oauth.openshift.io)
E1220 19:34:42.953322       1 reflector.go:136] github.com/openshift/client-go/security/informers/externalversions/factory.go:101: Failed to list *v1.SecurityContextConstraints: the server could not find the requested resource (get securitycontextconstraints.security.openshift.io)
E1220 19:34:42.954139       1 reflector.go:136] github.com/openshift/client-go/oauth/informers/externalversions/factory.go:101: Failed to list *v1.OAuthClient: the server could not find the requested resource (get oauthclients.oauth.openshift.io)
E1220 19:34:42.955527       1 reflector.go:136] github.com/openshift/origin/pkg/quota/generated/informers/internalversion/factory.go:101: Failed to list *quota.ClusterResourceQuota: the server could not find the requested resource (get clusterresourcequotas.quota.openshift.io)
E1220 19:34:42.956640       1 reflector.go:136] github.com/openshift/client-go/user/informers/externalversions/factory.go:101: Failed to list *v1.Group: the server could not find the requested resource (get groups.user.openshift.io)
E1220 19:34:43.289087       1 oauth_apiserver.go:294] the server could not find the requested resource (post oauthclients.oauth.openshift.io)

What you expected to happen?

Successful startup

How to reproduce it (as minimally and precisely as possible)?

$ bin/openshift-install create cluster

Anything else we need to know?

Worked once but failed numerous time before and after the one success.

References

• enter text here.

[sttts]

Please attach more logs, especially of the terminated kube-apiserver on the bootstrap node (64b3442255997 in the upper example). The provided logs only show non-critical reflector logs.

[steven-ellis]

Could this be connected to

• libvirt: 1st worker needs more memory #1041

[steven-ellis]

Also related to

• failed to initialize the cluster: a lot off restarting pods - CrashLoopBackOff #1198

[steven-ellis]

I'm going to follow this up under #1041

[eparis]

Not enough information is present to determine the root cause. I'm going to close this issue. If we find a reproducer unable to start the Masters please file bugs in bugzilla against the "master" component.