Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"Cluster health check failed: cluster agent is not ready" error seen during cluster provisioning after which it proceeds to succeed. #28836

Closed
sowmyav27 opened this issue Sep 10, 2020 · 25 comments
Assignees
Labels
kind/bug-qa Issues that have not yet hit a real release. Bugs introduced by a new feature or enhancement
Milestone

Comments

@sowmyav27
Copy link
Contributor

What kind of request is this (question/bug/enhancement/feature request): bug

Steps to reproduce (least amount of steps as possible):

  • Deploy a 3 nodes custom cluster - 1 etcd, 1 control and 1 worker nodes
  • sometimes the cluster is seen in an error state Cluster health check failed: cluster agent is not ready after it has finished provisioning. Nodes will be seen Active in this case. But the cluster is seen in an error state.
  • After about2 to 3 minutes (approx) the cluster comes to Active state.
  • Seen happening quite often on master-head but NOT everytime.

Screen Shot 2020-09-09 at 4 23 55 PM

Other details that may be helpful:

Environment information

  • Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI): master-head - c63f8fda
  • Installation option (single install/HA): HA

Cluster information

  • Cluster type (Hosted/Infrastructure Provider/Custom/Imported): custom cluster
  • Kubernetes version (use kubectl version):
1.18
@dramich
Copy link
Contributor

dramich commented Sep 17, 2020

This is not unexpected behavior when first setting up a cluster. When doing a custom cluster this will just run the base agent image to get it connected into rancher. After the cluster connects rancher will determine if the current agent has all the required features/settings/etc. and if not will redeploy the agent with the desired configuration. Trace logging can be enabled and the difference between the configs can be viewed to see what rancher is changing in the config for the agent.

@sowmyav27
Copy link
Contributor Author

@dramich This issue seems to be happening recently on master and happens quite often. This was not seen before. Reopening the issue to see why would that be the case.

@dramich
Copy link
Contributor

dramich commented Sep 18, 2020

01490ab
It is a newer change which is an additional check to the cluster health sync process. If the agent deployment is not ready the check fails and we try again in ~15 seconds. See previous comment for why the agent is deployed twice.

@sowmyav27
Copy link
Contributor Author

Closing the issue based on comments #28836 (comment) and #28836 (comment)

@sangeethah sangeethah changed the title "Cluster health check failed: cluster agent is not ready" error seen intermittently during cluster provisioning "Cluster health check failed: cluster agent is not ready" error seen during cluster provisioning after which it proceeds to succeed. Sep 24, 2020
@slash1387
Copy link

slash1387 commented Oct 10, 2020

@sowmyav27 @izaac please re-open

  1. fresh install
docker run -d --restart=unless-stopped --name=rancher2.5.1 -p 80:80 -p 443:443 --privileged rancher/rancher:v2.5.1
  1. Global -> Add Cluster -> Existing Nodes -> Choose all 3 roles (etcd, worker, control-plane)
  • all default values, except no authorization endpoint...but tried with both aynway same result.
  1. exec docker run on 2 or 1 nodes, they register and start to setup...
sudo docker run -d --privileged --restart=unless-stopped --net=host -v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run rancher/rancher-agent:v2.5.1 --server https://10.3.2.19 --token slrznggf27dbr42tfpwxqfzldkjwbbrrhjft4bpn67p289f7hftz9n --ca-checksum 52cfb3fb1446db991cb01d707e254b0f13048a40e2416d5a9c1facfa66699587 --etcd --controlplane --worker
  1. wait...it will stop with ERROR: Cluster health check failed: cluster agent is not ready

image

not working! also after waiting 15min

Test Node docker ps:

# docker ps
CONTAINER ID        IMAGE                                  COMMAND                  CREATED             STATUS              PORTS               NAMES
416b38af77b4        b5af743e5984                           "/server"                4 minutes ago       Up 4 minutes                            k8s_default-http-backend_default-http-backend-65dd5949d9-dc96j_ingress-nginx_5e33f3c3-aac0-4e82-a6a6-86f7ec7aeb32_0
d6acc3084ef6        141c7125f5a2                           "run.sh"                 4 minutes ago       Up 4 minutes                            k8s_agent_cattle-node-agent-txn4r_cattle-system_6cdd8d8d-c654-4390-b966-bde360a094b8_0
d6aac9ad6a2f        rancher/pause:3.2                      "/pause"                 4 minutes ago       Up 4 minutes                            k8s_POD_cattle-node-agent-txn4r_cattle-system_6cdd8d8d-c654-4390-b966-bde360a094b8_0
424f2433e8e7        rancher/pause:3.2                      "/pause"                 4 minutes ago       Up 4 minutes                            k8s_POD_cattle-cluster-agent-66585596d4-9v2wh_cattle-system_42936ba1-6adb-4808-b20b-7d8073f16ffe_0
6f33ed2c0531        9afe4570cb30                           "/usr/bin/dumb-init …"   4 minutes ago       Up 4 minutes                            k8s_nginx-ingress-controller_nginx-ingress-controller-lzccz_ingress-nginx_7df658fc-76e7-4db9-a293-d66302673066_0
ade21a6dc414        rancher/pause:3.2                      "/pause"                 4 minutes ago       Up 4 minutes                            k8s_POD_default-http-backend-65dd5949d9-dc96j_ingress-nginx_5e33f3c3-aac0-4e82-a6a6-86f7ec7aeb32_0
f314e3f84cf1        rancher/pause:3.2                      "/pause"                 4 minutes ago       Up 4 minutes                            k8s_POD_nginx-ingress-controller-lzccz_ingress-nginx_7df658fc-76e7-4db9-a293-d66302673066_0
58670dfe32a7        rancher/metrics-server                 "/metrics-server --k…"   5 minutes ago       Up 5 minutes                            k8s_metrics-server_metrics-server-8449844bf-dwzx5_kube-system_2d9aa486-ea16-47b3-b81d-cea366c3f237_0
18350db5fd5f        rancher/pause:3.2                      "/pause"                 5 minutes ago       Up 5 minutes                            k8s_POD_metrics-server-8449844bf-dwzx5_kube-system_2d9aa486-ea16-47b3-b81d-cea366c3f237_0
a3e55fb7e190        4f1064cf7caf                           "/cluster-proportion…"   5 minutes ago       Up 5 minutes                            k8s_autoscaler_coredns-autoscaler-79599b9dc6-hqpss_kube-system_5fdd5321-9e6e-4863-9d8a-0ec71ef67852_0
6a98c985c7b8        bfe3a36ebd25                           "/coredns -conf /etc…"   5 minutes ago       Up 5 minutes                            k8s_coredns_coredns-6f85d5fb88-jcd4f_kube-system_8715ecc1-02a6-4092-9477-ea83f41e2bc8_0
fccd37796804        rancher/pause:3.2                      "/pause"                 5 minutes ago       Up 5 minutes                            k8s_POD_coredns-autoscaler-79599b9dc6-hqpss_kube-system_5fdd5321-9e6e-4863-9d8a-0ec71ef67852_0
0e10da014a77        03feeb39a75a                           "/usr/bin/kube-contr…"   5 minutes ago       Up 5 minutes                            k8s_calico-kube-controllers_calico-kube-controllers-649b7b795b-q9hx7_kube-system_fc3eae97-c561-439c-9e6c-49e09279971b_0
509c63a3a47b        rancher/pause:3.2                      "/pause"                 5 minutes ago       Up 5 minutes                            k8s_POD_coredns-6f85d5fb88-jcd4f_kube-system_8715ecc1-02a6-4092-9477-ea83f41e2bc8_0
7a1da9b4f259        rancher/pause:3.2                      "/pause"                 5 minutes ago       Up 5 minutes                            k8s_POD_calico-kube-controllers-649b7b795b-q9hx7_kube-system_fc3eae97-c561-439c-9e6c-49e09279971b_0
658b02a3a9f8        0bfefe9f649b                           "/opt/bin/flanneld -…"   5 minutes ago       Up 5 minutes                            k8s_kube-flannel_canal-qxqmr_kube-system_8b0e721f-a222-401d-bb1a-bca34d75755c_0
febb003a768e        0f351f210d5e                           "start_runit"            5 minutes ago       Up 5 minutes                            k8s_calico-node_canal-qxqmr_kube-system_8b0e721f-a222-401d-bb1a-bca34d75755c_0
13145f665765        rancher/pause:3.2                      "/pause"                 5 minutes ago       Up 5 minutes                            k8s_POD_canal-qxqmr_kube-system_8b0e721f-a222-401d-bb1a-bca34d75755c_0
18c5a41141af        rancher/hyperkube:v1.19.2-rancher1     "/opt/rke-tools/entr…"   5 minutes ago       Up 5 minutes                            kube-proxy
238853871c11        rancher/hyperkube:v1.19.2-rancher1     "/opt/rke-tools/entr…"   5 minutes ago       Up 5 minutes                            kubelet
780acf3f9be3        rancher/hyperkube:v1.19.2-rancher1     "/opt/rke-tools/entr…"   6 minutes ago       Up 5 minutes                            kube-scheduler
64312146f9cf        rancher/hyperkube:v1.19.2-rancher1     "/opt/rke-tools/entr…"   6 minutes ago       Up 6 minutes                            kube-controller-manager
1626ead60e83        rancher/hyperkube:v1.19.2-rancher1     "/opt/rke-tools/entr…"   6 minutes ago       Up 6 minutes                            kube-apiserver
e5e2a65fccb8        rancher/coreos-etcd:v3.4.13-rancher1   "/usr/local/bin/etcd…"   6 minutes ago       Up 6 minutes                            etcd
docker logs d6acc3084ef6
INFO: Environment: CATTLE_ADDRESS=10.3.2.30 CATTLE_AGENT_CONNECT=true CATTLE_CA_CHECKSUM=52cfb3fb1446db991cb01d707e254b0f13048a40e2416d5a9c1facfa66699587 CATTLE_CLUSTER=false CATTLE_INTERNAL_ADDRESS= CATTLE_K8S_MANAGED=true CATTLE_NODE_NAME=test-1 CATTLE_SERVER=https://10.3.2.19
INFO: Using resolv.conf: nameserver 10.3.0.22 nameserver 10.3.0.24 search ******.net
INFO: https://10.3.2.19/ping is accessible
INFO: Value from https://10.3.2.19/v3/settings/cacerts is an x509 certificate
time="2020-10-10T08:17:26Z" level=info msg="Listening on /tmp/log.sock"
time="2020-10-10T08:17:26Z" level=info msg="Rancher agent version v2.5.1 is starting"
time="2020-10-10T08:17:26Z" level=info msg="Option customConfig=map[address:10.3.2.30 internalAddress: label:map[] roles:[] taints:[]]"
time="2020-10-10T08:17:26Z" level=info msg="Option etcd=false"
time="2020-10-10T08:17:26Z" level=info msg="Option controlPlane=false"
time="2020-10-10T08:17:26Z" level=info msg="Option worker=false"
time="2020-10-10T08:17:26Z" level=info msg="Option requestedHostname=test-1"
time="2020-10-10T08:17:26Z" level=info msg="Connecting to wss://10.3.2.19/v3/connect with token gd4q8gsxgzqj5gd4blmxt9xvvlcs28rt949qxn5dp7b4kqgwl8jmhq"
time="2020-10-10T08:17:26Z" level=info msg="Connecting to proxy" url="wss://10.3.2.19/v3/connect"
time="2020-10-10T08:17:27Z" level=info msg="Starting plan monitor, checking every 120 seconds"
time="2020-10-10T08:17:42Z" level=info msg="Removing unmanaged agent /hopeful_margulis(2c97d0a10c990faadcc676ef9162dd86680de144c2327d7a97f1fd799299085f)"
time="2020-10-10T08:17:42Z" level=info msg="Removing unmanaged agent /tender_haibt(02a8b444d7a0be47b0587ad98e97da025d5c42758977c143dcb20d3d61275f41)"

Rancher Logs

W1010 08:18:32.593197       7 warnings.go:67] v1 ComponentStatus is deprecated in v1.19+
W1010 08:18:37.662692       7 warnings.go:67] v1 ComponentStatus is deprecated in v1.19+
2020/10/10 08:18:40 [ERROR] error syncing 'system-default-registry': handler copy-settings: the server could not find the requested resource, requeuing
2020/10/10 08:18:40 [ERROR] error syncing 'install-uuid': handler copy-settings: the server could not find the requested resource, requeuing
W1010 08:18:42.716823       7 warnings.go:67] v1 ComponentStatus is deprecated in v1.19+
2020/10/10 08:18:46 [ERROR] error syncing 'system-library': handler system-image-upgrade-catalog-controller: upgrade cluster c-tg8nj system service logging failed: cluster c-tg8nj not ready, requeuing
W1010 08:18:47.558799       7 warnings.go:67] v1 ComponentStatus is deprecated in v1.19+
2020/10/10 08:18:48 [ERROR] error syncing 'c-tg8nj/p-v7brm': handler system-image-upgrade-controller: upgrade cluster c-tg8nj system service logging failed: cluster c-tg8nj not ready, requeuing
W1010 08:18:52.629379       7 warnings.go:67] v1 ComponentStatus is deprecated in v1.19+
W1010 08:18:57.687910       7 warnings.go:67] v1 ComponentStatus is deprecated in v1.19+
W1010 08:19:02.566794       7 warnings.go:67] v1 ComponentStatus is deprecated in v1.19+
W1010 08:19:07.630732       7 warnings.go:67] v1 ComponentStatus is deprecated in v1.19+
W1010 08:19:12.690350       7 warnings.go:67] v1 ComponentStatus is deprecated in v1.19+
W1010 08:19:17.564625       7 warnings.go:67] v1 ComponentStatus is deprecated in v1.19+
W1010 08:19:22.624527       7 warnings.go:67] v1 ComponentStatus is deprecated in v1.19+
W1010 08:19:27.710653       7 warnings.go:67] v1 ComponentStatus is deprecated in v1.19+
W1010 08:19:32.586652       7 warnings.go:67] v1 ComponentStatus is deprecated in v1.19+
W1010 08:19:37.662635       7 warnings.go:67] v1 ComponentStatus is deprecated in v1.19+
W1010 08:19:42.724826       7 warnings.go:67] v1 ComponentStatus is deprecated in v1.19+
W1010 08:19:47.558410       7 warnings.go:67] v1 ComponentStatus is deprecated in v1.19+
2020-10-10 08:19:51.814627 I | mvcc: store.index: compact 268875
2020-10-10 08:19:51.911637 I | mvcc: finished scheduled compaction at 268875 (took 92.416171ms)
W1010 08:19:52.612026       7 warnings.go:67] v1 ComponentStatus is deprecated in v1.19+
W1010 08:19:57.672658       7 warnings.go:67] v1 ComponentStatus is deprecated in v1.19+
W1010 08:20:02.580673       7 warnings.go:67] v1 ComponentStatus is deprecated in v1.19+
2020/10/10 08:20:02 [ERROR] error syncing 'system-default-registry': handler copy-settings: the server could not find the requested resource, requeuing
2020/10/10 08:20:02 [ERROR] error syncing 'install-uuid': handler copy-settings: the server could not find the requested resource, requeuing
W1010 08:20:07.652478       7 warnings.go:67] v1 ComponentStatus is deprecated in v1.19+
2020/10/10 08:20:08 [ERROR] error syncing 'system-library': handler system-image-upgrade-catalog-controller: upgrade cluster c-tg8nj system service logging failed: cluster c-tg8nj not ready, requeuing
2020/10/10 08:20:09 [ERROR] error syncing 'c-tg8nj/p-969zw': handler system-image-upgrade-controller: upgrade cluster c-tg8nj system service logging failed: cluster c-tg8nj not ready, requeuing
W1010 08:20:12.712819       7 warnings.go:67] v1 ComponentStatus is deprecated in v1.19+
W1010 08:20:14.428951      33 watcher.go:199] watch chan error: etcdserver: mvcc: required revision has been compacted
W1010 08:20:17.556641       7 warnings.go:67] v1 ComponentStatus is deprecated in v1.19+
W1010 08:20:22.617235       7 warnings.go:67] v1 ComponentStatus is deprecated in v1.19+
W1010 08:20:27.690820       7 warnings.go:67] v1 ComponentStatus is deprecated in v1.19+
W1010 08:20:32.562017       7 warnings.go:67] v1 ComponentStatus is deprecated in v1.19+
W1010 08:20:37.617024       7 warnings.go:67] v1 ComponentStatus is deprecated in v1.19+
W1010 08:20:42.674255       7 warnings.go:67] v1 ComponentStatus is deprecated in v1.19+
W1010 08:20:47.558463       7 warnings.go:67] v1 ComponentStatus is deprecated in v1.19+
W1010 08:20:52.612394       7 warnings.go:67] v1 ComponentStatus is deprecated in v1.19+

kube-apiserver

I1010 08:16:47.066177       1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.3.2.30:2379  <nil> 0 <nil>}]
E1010 08:16:47.180643       1 customresource_handler.go:668] error building openapi models for globalnetworkpolicies.crd.projectcalico.org: ERROR $root.definitions.org.projectcalico.crd.v1.GlobalNetworkPolicy.properties.spec.properties.egress.items.<array>.properties.destination.properties.notPorts.items.<array> has invalid property: anyOf
ERROR $root.definitions.org.projectcalico.crd.v1.GlobalNetworkPolicy.properties.spec.properties.egress.items.<array>.properties.destination.properties.ports.items.<array> has invalid property: anyOf
ERROR $root.definitions.org.projectcalico.crd.v1.GlobalNetworkPolicy.properties.spec.properties.egress.items.<array>.properties.notProtocol has invalid property: anyOf
ERROR $root.definitions.org.projectcalico.crd.v1.GlobalNetworkPolicy.properties.spec.properties.egress.items.<array>.properties.protocol has invalid property: anyOf
ERROR $root.definitions.org.projectcalico.crd.v1.GlobalNetworkPolicy.properties.spec.properties.egress.items.<array>.properties.source.properties.notPorts.items.<array> has invalid property: anyOf
ERROR $root.definitions.org.projectcalico.crd.v1.GlobalNetworkPolicy.properties.spec.properties.egress.items.<array>.properties.source.properties.ports.items.<array> has invalid property: anyOf
ERROR $root.definitions.org.projectcalico.crd.v1.GlobalNetworkPolicy.properties.spec.properties.ingress.items.<array>.properties.destination.properties.notPorts.items.<array> has invalid property: anyOf
ERROR $root.definitions.org.projectcalico.crd.v1.GlobalNetworkPolicy.properties.spec.properties.ingress.items.<array>.properties.destination.properties.ports.items.<array> has invalid property: anyOf
ERROR $root.definitions.org.projectcalico.crd.v1.GlobalNetworkPolicy.properties.spec.properties.ingress.items.<array>.properties.notProtocol has invalid property: anyOf
ERROR $root.definitions.org.projectcalico.crd.v1.GlobalNetworkPolicy.properties.spec.properties.ingress.items.<array>.properties.protocol has invalid property: anyOf
ERROR $root.definitions.org.projectcalico.crd.v1.GlobalNetworkPolicy.properties.spec.properties.ingress.items.<array>.properties.source.properties.notPorts.items.<array> has invalid property: anyOf
ERROR $root.definitions.org.projectcalico.crd.v1.GlobalNetworkPolicy.properties.spec.properties.ingress.items.<array>.properties.source.properties.ports.items.<array> has invalid property: anyOf
I1010 08:16:47.196654       1 client.go:360] parsed scheme: "endpoint"
I1010 08:16:47.196851       1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.3.2.30:2379  <nil> 0 <nil>}]
I1010 08:16:47.364774       1 client.go:360] parsed scheme: "endpoint"
I1010 08:16:47.364914       1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.3.2.30:2379  <nil> 0 <nil>}]
I1010 08:16:47.507046       1 client.go:360] parsed scheme: "endpoint"
I1010 08:16:47.507181       1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.3.2.30:2379  <nil> 0 <nil>}]
I1010 08:16:47.604191       1 client.go:360] parsed scheme: "endpoint"
I1010 08:16:47.604361       1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.3.2.30:2379  <nil> 0 <nil>}]
I1010 08:16:47.707196       1 client.go:360] parsed scheme: "endpoint"
I1010 08:16:47.707299       1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.3.2.30:2379  <nil> 0 <nil>}]
I1010 08:16:47.816388       1 client.go:360] parsed scheme: "endpoint"
I1010 08:16:47.816541       1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.3.2.30:2379  <nil> 0 <nil>}]
I1010 08:16:47.933727       1 client.go:360] parsed scheme: "endpoint"
I1010 08:16:47.933829       1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.3.2.30:2379  <nil> 0 <nil>}]
E1010 08:16:59.396521       1 available_controller.go:437] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
I1010 08:17:02.936635       1 client.go:360] parsed scheme: "passthrough"
I1010 08:17:02.936925       1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://10.3.2.30:2379  <nil> 0 <nil>}] <nil> <nil>}
I1010 08:17:02.937045       1 clientconn.go:948] ClientConn switching balancer to "pick_first"
W1010 08:17:03.105207       1 handler_proxy.go:102] no RequestInfo found in the context
E1010 08:17:03.105442       1 controller.go:116] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I1010 08:17:03.105506       1 controller.go:129] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E1010 08:17:06.046538       1 available_controller.go:437] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.43.135.217:443/apis/metrics.k8s.io/v1beta1: Get "https://10.43.135.217:443/apis/metrics.k8s.io/v1beta1": dial tcp 10.43.135.217:443: connect: connection refused
E1010 08:17:06.047756       1 available_controller.go:437] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.43.135.217:443/apis/metrics.k8s.io/v1beta1: Get "https://10.43.135.217:443/apis/metrics.k8s.io/v1beta1": dial tcp 10.43.135.217:443: connect: connection refused
E1010 08:17:06.052483       1 available_controller.go:437] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.43.135.217:443/apis/metrics.k8s.io/v1beta1: Get "https://10.43.135.217:443/apis/metrics.k8s.io/v1beta1": dial tcp 10.43.135.217:443: connect: connection refused
E1010 08:17:06.073621       1 available_controller.go:437] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.43.135.217:443/apis/metrics.k8s.io/v1beta1: Get "https://10.43.135.217:443/apis/metrics.k8s.io/v1beta1": dial tcp 10.43.135.217:443: connect: connection refused
E1010 08:17:06.114637       1 available_controller.go:437] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.43.135.217:443/apis/metrics.k8s.io/v1beta1: Get "https://10.43.135.217:443/apis/metrics.k8s.io/v1beta1": dial tcp 10.43.135.217:443: connect: connection refused
E1010 08:17:06.196002       1 available_controller.go:437] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.43.135.217:443/apis/metrics.k8s.io/v1beta1: Get "https://10.43.135.217:443/apis/metrics.k8s.io/v1beta1": dial tcp 10.43.135.217:443: connect: connection refused
E1010 08:17:06.358061       1 available_controller.go:437] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.43.135.217:443/apis/metrics.k8s.io/v1beta1: Get "https://10.43.135.217:443/apis/metrics.k8s.io/v1beta1": dial tcp 10.43.135.217:443: connect: connection refused
E1010 08:17:06.679233       1 available_controller.go:437] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.43.135.217:443/apis/metrics.k8s.io/v1beta1: Get "https://10.43.135.217:443/apis/metrics.k8s.io/v1beta1": dial tcp 10.43.135.217:443: connect: connection refused
W1010 08:17:07.047019       1 handler_proxy.go:102] no RequestInfo found in the context
E1010 08:17:07.047171       1 controller.go:116] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I1010 08:17:07.047206       1 controller.go:129] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E1010 08:17:07.321203       1 available_controller.go:437] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.43.135.217:443/apis/metrics.k8s.io/v1beta1: Get "https://10.43.135.217:443/apis/metrics.k8s.io/v1beta1": dial tcp 10.43.135.217:443: connect: connection refused
E1010 08:17:08.605645       1 available_controller.go:437] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.43.135.217:443/apis/metrics.k8s.io/v1beta1: Get "https://10.43.135.217:443/apis/metrics.k8s.io/v1beta1": dial tcp 10.43.135.217:443: connect: connection refused
I1010 08:17:18.499650       1 controller.go:606] quota admission added evaluator for: namespaces
I1010 08:17:41.012792       1 client.go:360] parsed scheme: "passthrough"
I1010 08:17:41.013224       1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://10.3.2.30:2379  <nil> 0 <nil>}] <nil> <nil>}
I1010 08:17:41.013356       1 clientconn.go:948] ClientConn switching balancer to "pick_first"
I1010 08:17:45.972161       1 trace.go:205] Trace[240014665]: "cacher list" type:*apiregistration.APIService (10-Oct-2020 08:17:42.971) (total time: 3000ms):
Trace[240014665]: [3.000331838s] [3.000331838s] END
I1010 08:17:45.973008       1 trace.go:205] Trace[406069070]: "List" url:/apis/apiregistration.k8s.io/v1/apiservices,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:42.971) (total time: 3001ms):
Trace[406069070]: [3.001395854s] [3.001395854s] END
I1010 08:17:45.982504       1 trace.go:205] Trace[1549864598]: "cacher list" type:*batch.CronJob (10-Oct-2020 08:17:42.982) (total time: 3000ms):
Trace[1549864598]: [3.000270688s] [3.000270688s] END
I1010 08:17:45.983261       1 trace.go:205] Trace[1202945246]: "List" url:/apis/batch/v1beta1/cronjobs,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:42.982) (total time: 3001ms):
Trace[1202945246]: [3.001070586s] [3.001070586s] END
I1010 08:17:46.039176       1 trace.go:205] Trace[1633962375]: "cacher list" type:*core.LimitRange (10-Oct-2020 08:17:43.038) (total time: 3000ms):
Trace[1633962375]: [3.000503802s] [3.000503802s] END
I1010 08:17:46.040088       1 trace.go:205] Trace[1930040326]: "List" url:/api/v1/limitranges,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:43.038) (total time: 3001ms):
Trace[1930040326]: [3.001640795s] [3.001640795s] END
I1010 08:17:46.074821       1 trace.go:205] Trace[1069487470]: "cacher list" type:*networking.Ingress (10-Oct-2020 08:17:43.074) (total time: 3000ms):
Trace[1069487470]: [3.000401013s] [3.000401013s] END
I1010 08:17:46.075531       1 trace.go:205] Trace[528487566]: "List" url:/apis/extensions/v1beta1/ingresses,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:43.074) (total time: 3001ms):
Trace[528487566]: [3.001241242s] [3.001241242s] END
I1010 08:17:46.099518       1 trace.go:205] Trace[2029159466]: "cacher list" type:*rbac.Role (10-Oct-2020 08:17:43.099) (total time: 3000ms):
Trace[2029159466]: [3.000345902s] [3.000345902s] END
I1010 08:17:46.100164       1 trace.go:205] Trace[997429371]: "List" url:/apis/rbac.authorization.k8s.io/v1/roles,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:43.098) (total time: 3001ms):
Trace[997429371]: [3.001160185s] [3.001160185s] END
I1010 08:17:46.166306       1 trace.go:205] Trace[657625798]: "cacher list" type:*apps.StatefulSet (10-Oct-2020 08:17:43.165) (total time: 3000ms):
Trace[657625798]: [3.000420125s] [3.000420125s] END
I1010 08:17:46.167350       1 trace.go:205] Trace[1687139835]: "List" url:/apis/apps/v1/statefulsets,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:43.165) (total time: 3001ms):
Trace[1687139835]: [3.001600575s] [3.001600575s] END
I1010 08:17:46.206815       1 trace.go:205] Trace[838238294]: "cacher list" type:*core.ResourceQuota (10-Oct-2020 08:17:43.206) (total time: 3000ms):
Trace[838238294]: [3.000398072s] [3.000398072s] END
I1010 08:17:46.207679       1 trace.go:205] Trace[2146476840]: "List" url:/api/v1/resourcequotas,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:43.206) (total time: 3001ms):
Trace[2146476840]: [3.001384615s] [3.001384615s] END
I1010 08:17:46.322295       1 trace.go:205] Trace[892844732]: "cacher list" type:*policy.PodSecurityPolicy (10-Oct-2020 08:17:43.321) (total time: 3000ms):
Trace[892844732]: [3.000319401s] [3.000319401s] END
I1010 08:17:46.324335       1 trace.go:205] Trace[686513893]: "List" url:/apis/policy/v1beta1/podsecuritypolicies,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:43.321) (total time: 3002ms):
Trace[686513893]: [3.002535715s] [3.002535715s] END
I1010 08:17:46.409682       1 trace.go:205] Trace[86227139]: "cacher list" type:*core.Node (10-Oct-2020 08:17:43.409) (total time: 3000ms):
Trace[86227139]: [3.000335035s] [3.000335035s] END
I1010 08:17:46.410616       1 trace.go:205] Trace[1583300760]: "List" url:/api/v1/nodes,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:43.409) (total time: 3001ms):
Trace[1583300760]: [3.001440851s] [3.001440851s] END
I1010 08:17:46.473816       1 trace.go:205] Trace[955474244]: "cacher list" type:*batch.Job (10-Oct-2020 08:17:43.472) (total time: 3001ms):
Trace[955474244]: [3.001280405s] [3.001280405s] END
I1010 08:17:46.474445       1 trace.go:205] Trace[1856187428]: "List" url:/apis/batch/v1/jobs,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:43.472) (total time: 3001ms):
Trace[1856187428]: [3.001963404s] [3.001963404s] END
I1010 08:17:46.626224       1 trace.go:205] Trace[1066237289]: "cacher list" type:*core.ReplicationController (10-Oct-2020 08:17:43.625) (total time: 3000ms):
Trace[1066237289]: [3.000429132s] [3.000429132s] END
I1010 08:17:46.626848       1 trace.go:205] Trace[1857733247]: "List" url:/api/v1/replicationcontrollers,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:43.625) (total time: 3001ms):
Trace[1857733247]: [3.001126462s] [3.001126462s] END
I1010 08:17:46.648008       1 trace.go:205] Trace[584253046]: "cacher list" type:*networking.NetworkPolicy (10-Oct-2020 08:17:43.647) (total time: 3000ms):
Trace[584253046]: [3.000181596s] [3.000181596s] END
I1010 08:17:46.648682       1 trace.go:205] Trace[1164788071]: "List" url:/apis/networking.k8s.io/v1/networkpolicies,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:43.647) (total time: 3000ms):
Trace[1164788071]: [3.000989728s] [3.000989728s] END
I1010 08:17:46.667356       1 trace.go:205] Trace[1020316343]: "cacher list" type:*core.Service (10-Oct-2020 08:17:43.666) (total time: 3000ms):
Trace[1020316343]: [3.000631668s] [3.000631668s] END
I1010 08:17:46.668114       1 trace.go:205] Trace[570078856]: "List" url:/api/v1/services,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:43.666) (total time: 3001ms):
Trace[570078856]: [3.001490205s] [3.001490205s] END
I1010 08:17:49.979684       1 trace.go:205] Trace[1132676804]: "cacher list" type:*apiregistration.APIService (10-Oct-2020 08:17:46.979) (total time: 3000ms):
Trace[1132676804]: [3.000355529s] [3.000355529s] END
I1010 08:17:49.980372       1 trace.go:205] Trace[458580704]: "List" url:/apis/apiregistration.k8s.io/v1/apiservices,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:46.979) (total time: 3001ms):
Trace[458580704]: [3.001237532s] [3.001237532s] END
I1010 08:17:49.988152       1 trace.go:205] Trace[1307920244]: "cacher list" type:*batch.CronJob (10-Oct-2020 08:17:46.987) (total time: 3000ms):
Trace[1307920244]: [3.00023408s] [3.00023408s] END
I1010 08:17:49.988757       1 trace.go:205] Trace[819084270]: "List" url:/apis/batch/v1beta1/cronjobs,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:46.987) (total time: 3000ms):
Trace[819084270]: [3.000910583s] [3.000910583s] END
I1010 08:17:50.045883       1 trace.go:205] Trace[1631466515]: "cacher list" type:*core.LimitRange (10-Oct-2020 08:17:47.045) (total time: 3000ms):
Trace[1631466515]: [3.000474484s] [3.000474484s] END
I1010 08:17:50.046504       1 trace.go:205] Trace[420013027]: "List" url:/api/v1/limitranges,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:47.045) (total time: 3001ms):
Trace[420013027]: [3.001227972s] [3.001227972s] END
I1010 08:17:50.080269       1 trace.go:205] Trace[573198140]: "cacher list" type:*networking.Ingress (10-Oct-2020 08:17:47.079) (total time: 3000ms):
Trace[573198140]: [3.000298628s] [3.000298628s] END
I1010 08:17:50.081083       1 trace.go:205] Trace[385639674]: "List" url:/apis/extensions/v1beta1/ingresses,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:47.079) (total time: 3001ms):
Trace[385639674]: [3.001122595s] [3.001122595s] END
I1010 08:17:50.104851       1 trace.go:205] Trace[2063829735]: "cacher list" type:*rbac.Role (10-Oct-2020 08:17:47.104) (total time: 3000ms):
Trace[2063829735]: [3.000232625s] [3.000232625s] END
I1010 08:17:50.105830       1 trace.go:205] Trace[244301395]: "List" url:/apis/rbac.authorization.k8s.io/v1/roles,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:47.104) (total time: 3001ms):
Trace[244301395]: [3.001339442s] [3.001339442s] END
I1010 08:17:50.172480       1 trace.go:205] Trace[1915601798]: "cacher list" type:*apps.StatefulSet (10-Oct-2020 08:17:47.172) (total time: 3000ms):
Trace[1915601798]: [3.000249622s] [3.000249622s] END
I1010 08:17:50.173304       1 trace.go:205] Trace[1722204060]: "List" url:/apis/apps/v1/statefulsets,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:47.172) (total time: 3001ms):
Trace[1722204060]: [3.001119455s] [3.001119455s] END
I1010 08:17:50.213367       1 trace.go:205] Trace[1641689336]: "cacher list" type:*core.ResourceQuota (10-Oct-2020 08:17:47.212) (total time: 3000ms):
Trace[1641689336]: [3.000445681s] [3.000445681s] END
I1010 08:17:50.214323       1 trace.go:205] Trace[267610312]: "List" url:/api/v1/resourcequotas,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:47.212) (total time: 3001ms):
Trace[267610312]: [3.001561749s] [3.001561749s] END
I1010 08:17:50.330300       1 trace.go:205] Trace[1157173049]: "cacher list" type:*policy.PodSecurityPolicy (10-Oct-2020 08:17:47.329) (total time: 3000ms):
Trace[1157173049]: [3.000402296s] [3.000402296s] END
I1010 08:17:50.330966       1 trace.go:205] Trace[951932460]: "List" url:/apis/policy/v1beta1/podsecuritypolicies,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:47.329) (total time: 3001ms):
Trace[951932460]: [3.00112471s] [3.00112471s] END
I1010 08:17:50.416609       1 trace.go:205] Trace[2147293587]: "cacher list" type:*core.Node (10-Oct-2020 08:17:47.416) (total time: 3000ms):
Trace[2147293587]: [3.000319485s] [3.000319485s] END
I1010 08:17:50.417276       1 trace.go:205] Trace[367228671]: "List" url:/api/v1/nodes,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:47.416) (total time: 3001ms):
Trace[367228671]: [3.001037288s] [3.001037288s] END
I1010 08:17:50.479891       1 trace.go:205] Trace[1425726791]: "cacher list" type:*batch.Job (10-Oct-2020 08:17:47.479) (total time: 3000ms):
Trace[1425726791]: [3.000420132s] [3.000420132s] END
I1010 08:17:50.480557       1 trace.go:205] Trace[683409321]: "List" url:/apis/batch/v1/jobs,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:47.479) (total time: 3001ms):
Trace[683409321]: [3.001160514s] [3.001160514s] END
I1010 08:17:50.632907       1 trace.go:205] Trace[1716450529]: "cacher list" type:*core.ReplicationController (10-Oct-2020 08:17:47.632) (total time: 3000ms):
Trace[1716450529]: [3.000229926s] [3.000229926s] END
I1010 08:17:50.633786       1 trace.go:205] Trace[332681322]: "List" url:/api/v1/replicationcontrollers,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:47.632) (total time: 3001ms):
Trace[332681322]: [3.001183055s] [3.001183055s] END
I1010 08:17:50.653691       1 trace.go:205] Trace[1461063619]: "cacher list" type:*networking.NetworkPolicy (10-Oct-2020 08:17:47.653) (total time: 3000ms):
Trace[1461063619]: [3.000417054s] [3.000417054s] END
I1010 08:17:50.654493       1 trace.go:205] Trace[752756956]: "List" url:/apis/networking.k8s.io/v1/networkpolicies,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:47.653) (total time: 3001ms):
Trace[752756956]: [3.001308698s] [3.001308698s] END
I1010 08:17:50.672828       1 trace.go:205] Trace[1944386825]: "cacher list" type:*core.Service (10-Oct-2020 08:17:47.672) (total time: 3000ms):
Trace[1944386825]: [3.000293112s] [3.000293112s] END
I1010 08:17:50.673424       1 trace.go:205] Trace[1554054849]: "List" url:/api/v1/services,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:47.672) (total time: 3000ms):
Trace[1554054849]: [3.000937039s] [3.000937039s] END
I1010 08:17:53.988613       1 trace.go:205] Trace[2089959381]: "cacher list" type:*apiregistration.APIService (10-Oct-2020 08:17:50.988) (total time: 3000ms):
Trace[2089959381]: [3.000145152s] [3.000145152s] END
I1010 08:17:53.989193       1 trace.go:205] Trace[1839875814]: "List" url:/apis/apiregistration.k8s.io/v1/apiservices,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:50.988) (total time: 3000ms):
Trace[1839875814]: [3.000857089s] [3.000857089s] END
I1010 08:17:53.992945       1 trace.go:205] Trace[827564713]: "cacher list" type:*batch.CronJob (10-Oct-2020 08:17:50.992) (total time: 3000ms):
Trace[827564713]: [3.000370672s] [3.000370672s] END
I1010 08:17:53.993431       1 trace.go:205] Trace[1977449194]: "List" url:/apis/batch/v1beta1/cronjobs,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:50.992) (total time: 3000ms):
Trace[1977449194]: [3.000899888s] [3.000899888s] END
I1010 08:17:54.052195       1 trace.go:205] Trace[967922745]: "cacher list" type:*core.LimitRange (10-Oct-2020 08:17:51.051) (total time: 3000ms):
Trace[967922745]: [3.000333693s] [3.000333693s] END
I1010 08:17:54.052744       1 trace.go:205] Trace[621375191]: "List" url:/api/v1/limitranges,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:51.051) (total time: 3001ms):
Trace[621375191]: [3.001016555s] [3.001016555s] END
I1010 08:17:54.086397       1 trace.go:205] Trace[1632253011]: "cacher list" type:*networking.Ingress (10-Oct-2020 08:17:51.085) (total time: 3000ms):
Trace[1632253011]: [3.000272672s] [3.000272672s] END
I1010 08:17:54.088165       1 trace.go:205] Trace[36981596]: "List" url:/apis/extensions/v1beta1/ingresses,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:51.085) (total time: 3002ms):
Trace[36981596]: [3.002223438s] [3.002223438s] END
I1010 08:17:54.110793       1 trace.go:205] Trace[832364865]: "cacher list" type:*rbac.Role (10-Oct-2020 08:17:51.110) (total time: 3000ms):
Trace[832364865]: [3.000346398s] [3.000346398s] END
I1010 08:17:54.111832       1 trace.go:205] Trace[1748741072]: "List" url:/apis/rbac.authorization.k8s.io/v1/roles,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:51.110) (total time: 3001ms):
Trace[1748741072]: [3.001449125s] [3.001449125s] END
I1010 08:17:54.179137       1 trace.go:205] Trace[268908314]: "cacher list" type:*apps.StatefulSet (10-Oct-2020 08:17:51.178) (total time: 3000ms):
Trace[268908314]: [3.000434506s] [3.000434506s] END
I1010 08:17:54.179840       1 trace.go:205] Trace[795103738]: "List" url:/apis/apps/v1/statefulsets,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:51.178) (total time: 3001ms):
Trace[795103738]: [3.001341475s] [3.001341475s] END
I1010 08:17:54.219835       1 trace.go:205] Trace[1784139899]: "cacher list" type:*core.ResourceQuota (10-Oct-2020 08:17:51.219) (total time: 3000ms):
Trace[1784139899]: [3.000446298s] [3.000446298s] END
I1010 08:17:54.220589       1 trace.go:205] Trace[695987470]: "List" url:/api/v1/resourcequotas,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:51.219) (total time: 3001ms):
Trace[695987470]: [3.001304801s] [3.001304801s] END
I1010 08:17:54.341319       1 trace.go:205] Trace[1286310745]: "cacher list" type:*policy.PodSecurityPolicy (10-Oct-2020 08:17:51.341) (total time: 3000ms):
Trace[1286310745]: [3.000221872s] [3.000221872s] END
I1010 08:17:54.342046       1 trace.go:205] Trace[862296813]: "List" url:/apis/policy/v1beta1/podsecuritypolicies,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:51.340) (total time: 3001ms):
Trace[862296813]: [3.001016915s] [3.001016915s] END
I1010 08:17:54.421465       1 trace.go:205] Trace[745779087]: "cacher list" type:*core.Node (10-Oct-2020 08:17:51.421) (total time: 3000ms):
Trace[745779087]: [3.000143952s] [3.000143952s] END
I1010 08:17:54.422113       1 trace.go:205] Trace[109993728]: "List" url:/api/v1/nodes,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:51.421) (total time: 3000ms):
Trace[109993728]: [3.000744612s] [3.000744612s] END
I1010 08:17:54.485821       1 trace.go:205] Trace[919365989]: "cacher list" type:*batch.Job (10-Oct-2020 08:17:51.485) (total time: 3000ms):
Trace[919365989]: [3.000454402s] [3.000454402s] END
I1010 08:17:54.487725       1 trace.go:205] Trace[1006518318]: "List" url:/apis/batch/v1/jobs,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:51.485) (total time: 3002ms):
Trace[1006518318]: [3.002414491s] [3.002414491s] END
I1010 08:17:54.640063       1 trace.go:205] Trace[650966462]: "cacher list" type:*core.ReplicationController (10-Oct-2020 08:17:51.639) (total time: 3000ms):
Trace[650966462]: [3.000522606s] [3.000522606s] END
I1010 08:17:54.641445       1 trace.go:205] Trace[1799115141]: "List" url:/api/v1/replicationcontrollers,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:51.639) (total time: 3002ms):
Trace[1799115141]: [3.002084412s] [3.002084412s] END
I1010 08:17:54.659253       1 trace.go:205] Trace[934200725]: "cacher list" type:*networking.NetworkPolicy (10-Oct-2020 08:17:51.658) (total time: 3000ms):
Trace[934200725]: [3.000586705s] [3.000586705s] END
I1010 08:17:54.660164       1 trace.go:205] Trace[2053254858]: "List" url:/apis/networking.k8s.io/v1/networkpolicies,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:51.658) (total time: 3001ms):
Trace[2053254858]: [3.001574631s] [3.001574631s] END
I1010 08:17:54.678163       1 trace.go:205] Trace[1179129148]: "cacher list" type:*core.Service (10-Oct-2020 08:17:51.676) (total time: 3000ms):
Trace[1179129148]: [3.000980367s] [3.000980367s] END
I1010 08:17:54.679109       1 trace.go:205] Trace[1677226952]: "List" url:/api/v1/services,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:51.676) (total time: 3002ms):
Trace[1677226952]: [3.002043524s] [3.002043524s] END
I1010 08:17:57.996609       1 trace.go:205] Trace[1328523915]: "cacher list" type:*apiregistration.APIService (10-Oct-2020 08:17:54.996) (total time: 3000ms):
Trace[1328523915]: [3.000384805s] [3.000384805s] END
I1010 08:17:57.997407       1 trace.go:205] Trace[1663519738]: "List" url:/apis/apiregistration.k8s.io/v1/apiservices,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:54.995) (total time: 3001ms):
Trace[1663519738]: [3.001449178s] [3.001449178s] END
I1010 08:17:57.997809       1 trace.go:205] Trace[1375637893]: "cacher list" type:*batch.CronJob (10-Oct-2020 08:17:54.997) (total time: 3000ms):
Trace[1375637893]: [3.000226785s] [3.000226785s] END
I1010 08:17:57.998549       1 trace.go:205] Trace[1902952137]: "List" url:/apis/batch/v1beta1/cronjobs,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:54.997) (total time: 3000ms):
Trace[1902952137]: [3.000993105s] [3.000993105s] END
I1010 08:17:58.058451       1 trace.go:205] Trace[1051154100]: "cacher list" type:*core.LimitRange (10-Oct-2020 08:17:55.057) (total time: 3000ms):
Trace[1051154100]: [3.000406192s] [3.000406192s] END
I1010 08:17:58.059261       1 trace.go:205] Trace[1036621984]: "List" url:/api/v1/limitranges,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:55.057) (total time: 3001ms):
Trace[1036621984]: [3.001318118s] [3.001318118s] END
I1010 08:17:58.094286       1 trace.go:205] Trace[408876391]: "cacher list" type:*networking.Ingress (10-Oct-2020 08:17:55.093) (total time: 3000ms):
Trace[408876391]: [3.000912555s] [3.000912555s] END
I1010 08:17:58.095147       1 trace.go:205] Trace[1808439140]: "List" url:/apis/extensions/v1beta1/ingresses,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:55.093) (total time: 3001ms):
Trace[1808439140]: [3.001740981s] [3.001740981s] END
I1010 08:17:58.116942       1 trace.go:205] Trace[1113505073]: "cacher list" type:*rbac.Role (10-Oct-2020 08:17:55.116) (total time: 3000ms):
Trace[1113505073]: [3.000361598s] [3.000361598s] END
I1010 08:17:58.117498       1 trace.go:205] Trace[1824344327]: "List" url:/apis/rbac.authorization.k8s.io/v1/roles,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:55.116) (total time: 3000ms):
Trace[1824344327]: [3.000992458s] [3.000992458s] END
I1010 08:17:58.186478       1 trace.go:205] Trace[1875783897]: "cacher list" type:*apps.StatefulSet (10-Oct-2020 08:17:55.185) (total time: 3000ms):
Trace[1875783897]: [3.000396995s] [3.000396995s] END
I1010 08:17:58.187280       1 trace.go:205] Trace[851438758]: "List" url:/apis/apps/v1/statefulsets,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:55.185) (total time: 3001ms):
Trace[851438758]: [3.001333274s] [3.001333274s] END
I1010 08:17:58.226799       1 trace.go:205] Trace[928417027]: "cacher list" type:*core.ResourceQuota (10-Oct-2020 08:17:55.226) (total time: 3000ms):
Trace[928417027]: [3.000269291s] [3.000269291s] END
I1010 08:17:58.227857       1 trace.go:205] Trace[1651169903]: "List" url:/api/v1/resourcequotas,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:55.226) (total time: 3001ms):
Trace[1651169903]: [3.001434991s] [3.001434991s] END
I1010 08:17:58.348793       1 trace.go:205] Trace[1244618722]: "cacher list" type:*policy.PodSecurityPolicy (10-Oct-2020 08:17:55.348) (total time: 3000ms):
Trace[1244618722]: [3.000203636s] [3.000203636s] END
I1010 08:17:58.349606       1 trace.go:205] Trace[1437617880]: "List" url:/apis/policy/v1beta1/podsecuritypolicies,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:55.348) (total time: 3001ms):
Trace[1437617880]: [3.001142639s] [3.001142639s] END
I1010 08:17:58.428836       1 trace.go:205] Trace[1112285649]: "cacher list" type:*core.Node (10-Oct-2020 08:17:55.428) (total time: 3000ms):
Trace[1112285649]: [3.000246957s] [3.000246957s] END
I1010 08:17:58.429673       1 trace.go:205] Trace[1757160434]: "List" url:/api/v1/nodes,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:55.428) (total time: 3001ms):
Trace[1757160434]: [3.001182335s] [3.001182335s] END
I1010 08:17:58.495164       1 trace.go:205] Trace[2116897238]: "cacher list" type:*batch.Job (10-Oct-2020 08:17:55.493) (total time: 3001ms):
Trace[2116897238]: [3.001283509s] [3.001283509s] END
I1010 08:17:58.496109       1 trace.go:205] Trace[1001061286]: "List" url:/apis/batch/v1/jobs,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:55.493) (total time: 3002ms):
Trace[1001061286]: [3.00229356s] [3.00229356s] END
I1010 08:17:58.648105       1 trace.go:205] Trace[520838484]: "cacher list" type:*core.ReplicationController (10-Oct-2020 08:17:55.647) (total time: 3000ms):
Trace[520838484]: [3.000404456s] [3.000404456s] END
I1010 08:17:58.649263       1 trace.go:205] Trace[1835561728]: "List" url:/api/v1/replicationcontrollers,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:55.647) (total time: 3001ms):
Trace[1835561728]: [3.001708165s] [3.001708165s] END
I1010 08:17:58.666326       1 trace.go:205] Trace[957209477]: "cacher list" type:*networking.NetworkPolicy (10-Oct-2020 08:17:55.665) (total time: 3000ms):
Trace[957209477]: [3.000739285s] [3.000739285s] END
I1010 08:17:58.667621       1 trace.go:205] Trace[2103261847]: "List" url:/apis/networking.k8s.io/v1/networkpolicies,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:55.665) (total time: 3002ms):
Trace[2103261847]: [3.002159035s] [3.002159035s] END
I1010 08:17:58.685578       1 trace.go:205] Trace[897573751]: "cacher list" type:*core.Service (10-Oct-2020 08:17:55.685) (total time: 3000ms):
Trace[897573751]: [3.000373449s] [3.000373449s] END
I1010 08:17:58.686477       1 trace.go:205] Trace[1653657986]: "List" url:/api/v1/services,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:55.685) (total time: 3001ms):
Trace[1653657986]: [3.001311662s] [3.001311662s] END
I1010 08:18:02.004156       1 trace.go:205] Trace[886973031]: "cacher list" type:*apiregistration.APIService (10-Oct-2020 08:17:59.003) (total time: 3000ms):
Trace[886973031]: [3.000431466s] [3.000431466s] END
I1010 08:18:02.004728       1 trace.go:205] Trace[284402780]: "List" url:/apis/apiregistration.k8s.io/v1/apiservices,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:59.003) (total time: 3001ms):
Trace[284402780]: [3.001110205s] [3.001110205s] END
I1010 08:18:02.004830       1 trace.go:205] Trace[850625601]: "cacher list" type:*batch.CronJob (10-Oct-2020 08:17:59.004) (total time: 3000ms):
Trace[850625601]: [3.000409351s] [3.000409351s] END
I1010 08:18:02.005575       1 trace.go:205] Trace[1086675835]: "List" url:/apis/batch/v1beta1/cronjobs,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:59.004) (total time: 3001ms):
Trace[1086675835]: [3.001197754s] [3.001197754s] END
I1010 08:18:02.069075       1 trace.go:205] Trace[66630505]: "cacher list" type:*core.LimitRange (10-Oct-2020 08:17:59.068) (total time: 3000ms):
Trace[66630505]: [3.000256245s] [3.000256245s] END
I1010 08:18:02.069584       1 trace.go:205] Trace[223577697]: "List" url:/api/v1/limitranges,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:59.068) (total time: 3000ms):
Trace[223577697]: [3.000915425s] [3.000915425s] END
I1010 08:18:02.101828       1 trace.go:205] Trace[1066794709]: "cacher list" type:*networking.Ingress (10-Oct-2020 08:17:59.101) (total time: 3000ms):
Trace[1066794709]: [3.000422845s] [3.000422845s] END
I1010 08:18:02.102857       1 trace.go:205] Trace[2137987534]: "List" url:/apis/extensions/v1beta1/ingresses,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:59.101) (total time: 3001ms):
Trace[2137987534]: [3.001612205s] [3.001612205s] END
I1010 08:18:02.123909       1 trace.go:205] Trace[368677953]: "cacher list" type:*rbac.Role (10-Oct-2020 08:17:59.123) (total time: 3000ms):
Trace[368677953]: [3.000488878s] [3.000488878s] END
I1010 08:18:02.125066       1 trace.go:205] Trace[675733020]: "List" url:/apis/rbac.authorization.k8s.io/v1/roles,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:59.123) (total time: 3001ms):
Trace[675733020]: [3.001674782s] [3.001674782s] END
I1010 08:18:02.193074       1 trace.go:205] Trace[159572834]: "cacher list" type:*apps.StatefulSet (10-Oct-2020 08:17:59.192) (total time: 3000ms):
Trace[159572834]: [3.000368783s] [3.000368783s] END
I1010 08:18:02.194144       1 trace.go:205] Trace[1644599574]: "List" url:/apis/apps/v1/statefulsets,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:59.192) (total time: 3001ms):
Trace[1644599574]: [3.001593072s] [3.001593072s] END
I1010 08:18:02.234428       1 trace.go:205] Trace[128498444]: "cacher list" type:*core.ResourceQuota (10-Oct-2020 08:17:59.233) (total time: 3000ms):
Trace[128498444]: [3.000454673s] [3.000454673s] END
I1010 08:18:02.235259       1 trace.go:205] Trace[2096037500]: "List" url:/api/v1/resourcequotas,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:59.233) (total time: 3001ms):
Trace[2096037500]: [3.001312169s] [3.001312169s] END
I1010 08:18:02.358383       1 trace.go:205] Trace[1762357194]: "cacher list" type:*policy.PodSecurityPolicy (10-Oct-2020 08:17:59.355) (total time: 3002ms):
Trace[1762357194]: [3.002412742s] [3.002412742s] END
I1010 08:18:02.359300       1 trace.go:205] Trace[227350770]: "List" url:/apis/policy/v1beta1/podsecuritypolicies,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:59.355) (total time: 3003ms):
Trace[227350770]: [3.003405291s] [3.003405291s] END
I1010 08:18:02.436860       1 trace.go:205] Trace[2027830353]: "cacher list" type:*core.Node (10-Oct-2020 08:17:59.436) (total time: 3000ms):
Trace[2027830353]: [3.000447621s] [3.000447621s] END
I1010 08:18:02.437728       1 trace.go:205] Trace[1581928758]: "List" url:/api/v1/nodes,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:59.436) (total time: 3001ms):
Trace[1581928758]: [3.001377978s] [3.001377978s] END
I1010 08:18:02.503639       1 trace.go:205] Trace[2076396710]: "cacher list" type:*batch.Job (10-Oct-2020 08:17:59.502) (total time: 3000ms):
Trace[2076396710]: [3.000920279s] [3.000920279s] END
I1010 08:18:02.504513       1 trace.go:205] Trace[639284593]: "List" url:/apis/batch/v1/jobs,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:59.502) (total time: 3001ms):
Trace[639284593]: [3.001947191s] [3.001947191s] END
I1010 08:18:02.655826       1 trace.go:205] Trace[1353300418]: "cacher list" type:*core.ReplicationController (10-Oct-2020 08:17:59.655) (total time: 3000ms):
Trace[1353300418]: [3.000390168s] [3.000390168s] END
I1010 08:18:02.666276       1 trace.go:205] Trace[723168381]: "List" url:/api/v1/replicationcontrollers,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:59.655) (total time: 3010ms):
Trace[723168381]: [3.010841552s] [3.010841552s] END
I1010 08:18:02.680916       1 trace.go:205] Trace[495304490]: "cacher list" type:*networking.NetworkPolicy (10-Oct-2020 08:17:59.680) (total time: 3000ms):
Trace[495304490]: [3.000499545s] [3.000499545s] END
I1010 08:18:02.681919       1 trace.go:205] Trace[1660988225]: "List" url:/apis/networking.k8s.io/v1/networkpolicies,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:59.680) (total time: 3001ms):
Trace[1660988225]: [3.001592868s] [3.001592868s] END
I1010 08:18:02.691379       1 trace.go:205] Trace[1471827964]: "cacher list" type:*core.Service (10-Oct-2020 08:17:59.690) (total time: 3000ms):
Trace[1471827964]: [3.000458149s] [3.000458149s] END
I1010 08:18:02.692381       1 trace.go:205] Trace[1332039337]: "List" url:/api/v1/services,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:17:59.690) (total time: 3001ms):
Trace[1332039337]: [3.001612945s] [3.001612945s] END
I1010 08:18:06.011208       1 trace.go:205] Trace[953076618]: "cacher list" type:*apiregistration.APIService (10-Oct-2020 08:18:03.010) (total time: 3000ms):
Trace[953076618]: [3.000332754s] [3.000332754s] END
I1010 08:18:06.011643       1 trace.go:205] Trace[543290573]: "List" url:/apis/apiregistration.k8s.io/v1/apiservices,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:03.010) (total time: 3001ms):
Trace[543290573]: [3.001024099s] [3.001024099s] END
I1010 08:18:06.011891       1 trace.go:205] Trace[1878520674]: "cacher list" type:*batch.CronJob (10-Oct-2020 08:18:03.011) (total time: 3000ms):
Trace[1878520674]: [3.000206337s] [3.000206337s] END
I1010 08:18:06.012240       1 trace.go:205] Trace[34074021]: "List" url:/apis/batch/v1beta1/cronjobs,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:03.011) (total time: 3000ms):
Trace[34074021]: [3.000659211s] [3.000659211s] END
I1010 08:18:06.074486       1 trace.go:205] Trace[895099789]: "cacher list" type:*core.LimitRange (10-Oct-2020 08:18:03.074) (total time: 3000ms):
Trace[895099789]: [3.000259891s] [3.000259891s] END
I1010 08:18:06.074936       1 trace.go:205] Trace[149570670]: "List" url:/api/v1/limitranges,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:03.074) (total time: 3000ms):
Trace[149570670]: [3.000850714s] [3.000850714s] END
I1010 08:18:06.108199       1 trace.go:205] Trace[2026021762]: "cacher list" type:*networking.Ingress (10-Oct-2020 08:18:03.107) (total time: 3000ms):
Trace[2026021762]: [3.000334414s] [3.000334414s] END
I1010 08:18:06.108614       1 trace.go:205] Trace[1473883171]: "List" url:/apis/extensions/v1beta1/ingresses,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:03.107) (total time: 3000ms):
Trace[1473883171]: [3.000818035s] [3.000818035s] END
I1010 08:18:06.129583       1 trace.go:205] Trace[1258384451]: "cacher list" type:*rbac.Role (10-Oct-2020 08:18:03.129) (total time: 3000ms):
Trace[1258384451]: [3.000289389s] [3.000289389s] END
I1010 08:18:06.129884       1 trace.go:205] Trace[120269750]: "List" url:/apis/rbac.authorization.k8s.io/v1/roles,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:03.129) (total time: 3000ms):
Trace[120269750]: [3.000657071s] [3.000657071s] END
I1010 08:18:06.199897       1 trace.go:205] Trace[598255177]: "cacher list" type:*apps.StatefulSet (10-Oct-2020 08:18:03.199) (total time: 3000ms):
Trace[598255177]: [3.00030803s] [3.00030803s] END
I1010 08:18:06.200451       1 trace.go:205] Trace[614798405]: "List" url:/apis/apps/v1/statefulsets,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:03.199) (total time: 3000ms):
Trace[614798405]: [3.000968687s] [3.000968687s] END
I1010 08:18:06.241192       1 trace.go:205] Trace[653292742]: "cacher list" type:*core.ResourceQuota (10-Oct-2020 08:18:03.240) (total time: 3000ms):
Trace[653292742]: [3.000272697s] [3.000272697s] END
I1010 08:18:06.241746       1 trace.go:205] Trace[1683866198]: "List" url:/api/v1/resourcequotas,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:03.240) (total time: 3000ms):
Trace[1683866198]: [3.000900221s] [3.000900221s] END
I1010 08:18:06.365465       1 trace.go:205] Trace[1881169959]: "cacher list" type:*policy.PodSecurityPolicy (10-Oct-2020 08:18:03.365) (total time: 3000ms):
Trace[1881169959]: [3.000239249s] [3.000239249s] END
I1010 08:18:06.365891       1 trace.go:205] Trace[2109715768]: "List" url:/apis/policy/v1beta1/podsecuritypolicies,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:03.365) (total time: 3000ms):
Trace[2109715768]: [3.000812825s] [3.000812825s] END
I1010 08:18:06.444046       1 trace.go:205] Trace[66425231]: "cacher list" type:*core.Node (10-Oct-2020 08:18:03.443) (total time: 3000ms):
Trace[66425231]: [3.000601299s] [3.000601299s] END
I1010 08:18:06.445100       1 trace.go:205] Trace[295512843]: "List" url:/api/v1/nodes,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:03.443) (total time: 3001ms):
Trace[295512843]: [3.001814724s] [3.001814724s] END
I1010 08:18:06.510885       1 trace.go:205] Trace[1055279857]: "cacher list" type:*batch.Job (10-Oct-2020 08:18:03.510) (total time: 3000ms):
Trace[1055279857]: [3.000492817s] [3.000492817s] END
I1010 08:18:06.511878       1 trace.go:205] Trace[80725673]: "List" url:/apis/batch/v1/jobs,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:03.510) (total time: 3001ms):
Trace[80725673]: [3.001599808s] [3.001599808s] END
I1010 08:18:06.674140       1 trace.go:205] Trace[897330714]: "cacher list" type:*core.ReplicationController (10-Oct-2020 08:18:03.673) (total time: 3000ms):
Trace[897330714]: [3.000588895s] [3.000588895s] END
I1010 08:18:06.675094       1 trace.go:205] Trace[1774610041]: "List" url:/api/v1/replicationcontrollers,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:03.673) (total time: 3001ms):
Trace[1774610041]: [3.001537008s] [3.001537008s] END
I1010 08:18:06.688381       1 trace.go:205] Trace[578946252]: "cacher list" type:*networking.NetworkPolicy (10-Oct-2020 08:18:03.687) (total time: 3000ms):
Trace[578946252]: [3.000357942s] [3.000357942s] END
I1010 08:18:06.688944       1 trace.go:205] Trace[304475251]: "List" url:/apis/networking.k8s.io/v1/networkpolicies,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:03.687) (total time: 3001ms):
Trace[304475251]: [3.001046405s] [3.001046405s] END
I1010 08:18:06.697199       1 trace.go:205] Trace[2146425141]: "cacher list" type:*core.Service (10-Oct-2020 08:18:03.696) (total time: 3000ms):
Trace[2146425141]: [3.000534768s] [3.000534768s] END
I1010 08:18:06.698234       1 trace.go:205] Trace[976582191]: "List" url:/api/v1/services,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:03.696) (total time: 3001ms):
Trace[976582191]: [3.001649634s] [3.001649634s] END
I1010 08:18:10.016823       1 trace.go:205] Trace[1429065649]: "cacher list" type:*apiregistration.APIService (10-Oct-2020 08:18:07.016) (total time: 3000ms):
Trace[1429065649]: [3.000318352s] [3.000318352s] END
I1010 08:18:10.017116       1 trace.go:205] Trace[154413807]: "cacher list" type:*batch.CronJob (10-Oct-2020 08:18:07.016) (total time: 3000ms):
Trace[154413807]: [3.000156886s] [3.000156886s] END
I1010 08:18:10.017834       1 trace.go:205] Trace[1411504899]: "List" url:/apis/apiregistration.k8s.io/v1/apiservices,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:07.016) (total time: 3001ms):
Trace[1411504899]: [3.001452062s] [3.001452062s] END
I1010 08:18:10.018031       1 trace.go:205] Trace[10897390]: "List" url:/apis/batch/v1beta1/cronjobs,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:07.016) (total time: 3001ms):
Trace[10897390]: [3.001080452s] [3.001080452s] END
I1010 08:18:10.079631       1 trace.go:205] Trace[346211744]: "cacher list" type:*core.LimitRange (10-Oct-2020 08:18:07.079) (total time: 3000ms):
Trace[346211744]: [3.000288149s] [3.000288149s] END
I1010 08:18:10.080382       1 trace.go:205] Trace[1843082867]: "List" url:/api/v1/limitranges,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:07.079) (total time: 3001ms):
Trace[1843082867]: [3.001113275s] [3.001113275s] END
I1010 08:18:10.112693       1 trace.go:205] Trace[16172383]: "cacher list" type:*networking.Ingress (10-Oct-2020 08:18:07.112) (total time: 3000ms):
Trace[16172383]: [3.000411769s] [3.000411769s] END
I1010 08:18:10.113432       1 trace.go:205] Trace[1557632568]: "List" url:/apis/extensions/v1beta1/ingresses,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:07.112) (total time: 3001ms):
Trace[1557632568]: [3.001248282s] [3.001248282s] END
I1010 08:18:10.133782       1 trace.go:205] Trace[2017819181]: "cacher list" type:*rbac.Role (10-Oct-2020 08:18:07.133) (total time: 3000ms):
Trace[2017819181]: [3.000450617s] [3.000450617s] END
I1010 08:18:10.134536       1 trace.go:205] Trace[1752171671]: "List" url:/apis/rbac.authorization.k8s.io/v1/roles,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:07.133) (total time: 3001ms):
Trace[1752171671]: [3.001252165s] [3.001252165s] END
I1010 08:18:10.205810       1 trace.go:205] Trace[1020779499]: "cacher list" type:*apps.StatefulSet (10-Oct-2020 08:18:07.205) (total time: 3000ms):
Trace[1020779499]: [3.000381409s] [3.000381409s] END
I1010 08:18:10.206537       1 trace.go:205] Trace[1356228075]: "List" url:/apis/apps/v1/statefulsets,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:07.205) (total time: 3001ms):
Trace[1356228075]: [3.001207108s] [3.001207108s] END
I1010 08:18:10.246498       1 trace.go:205] Trace[792632191]: "cacher list" type:*core.ResourceQuota (10-Oct-2020 08:18:07.246) (total time: 3000ms):
Trace[792632191]: [3.000403861s] [3.000403861s] END
I1010 08:18:10.247336       1 trace.go:205] Trace[932376898]: "List" url:/api/v1/resourcequotas,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:07.245) (total time: 3001ms):
Trace[932376898]: [3.001323902s] [3.001323902s] END
I1010 08:18:10.372820       1 trace.go:205] Trace[1557480572]: "cacher list" type:*policy.PodSecurityPolicy (10-Oct-2020 08:18:07.372) (total time: 3000ms):
Trace[1557480572]: [3.000328072s] [3.000328072s] END
I1010 08:18:10.373803       1 trace.go:205] Trace[2101048846]: "List" url:/apis/policy/v1beta1/podsecuritypolicies,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:07.372) (total time: 3001ms):
Trace[2101048846]: [3.001388498s] [3.001388498s] END
I1010 08:18:10.451530       1 trace.go:205] Trace[849776008]: "cacher list" type:*core.Node (10-Oct-2020 08:18:07.450) (total time: 3000ms):
Trace[849776008]: [3.000510006s] [3.000510006s] END
I1010 08:18:10.452414       1 trace.go:205] Trace[186852217]: "List" url:/api/v1/nodes,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:07.450) (total time: 3001ms):
Trace[186852217]: [3.001463962s] [3.001463962s] END
I1010 08:18:10.517815       1 trace.go:205] Trace[789746751]: "cacher list" type:*batch.Job (10-Oct-2020 08:18:07.517) (total time: 3000ms):
Trace[789746751]: [3.000269576s] [3.000269576s] END
I1010 08:18:10.518892       1 trace.go:205] Trace[2033815780]: "List" url:/apis/batch/v1/jobs,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:07.517) (total time: 3001ms):
Trace[2033815780]: [3.001494914s] [3.001494914s] END
I1010 08:18:10.681481       1 trace.go:205] Trace[1213241051]: "cacher list" type:*core.ReplicationController (10-Oct-2020 08:18:07.680) (total time: 3000ms):
Trace[1213241051]: [3.000425629s] [3.000425629s] END
I1010 08:18:10.682433       1 trace.go:205] Trace[1601333875]: "List" url:/api/v1/replicationcontrollers,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:07.680) (total time: 3001ms):
Trace[1601333875]: [3.001505768s] [3.001505768s] END
I1010 08:18:10.695786       1 trace.go:205] Trace[1641894991]: "cacher list" type:*networking.NetworkPolicy (10-Oct-2020 08:18:07.695) (total time: 3000ms):
Trace[1641894991]: [3.000325886s] [3.000325886s] END
I1010 08:18:10.696538       1 trace.go:205] Trace[1096122671]: "List" url:/apis/networking.k8s.io/v1/networkpolicies,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:07.695) (total time: 3001ms):
Trace[1096122671]: [3.001122486s] [3.001122486s] END
I1010 08:18:10.702650       1 trace.go:205] Trace[1379572102]: "cacher list" type:*core.Service (10-Oct-2020 08:18:07.702) (total time: 3000ms):
Trace[1379572102]: [3.000346632s] [3.000346632s] END
I1010 08:18:10.703548       1 trace.go:205] Trace[1922666932]: "List" url:/api/v1/services,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:07.702) (total time: 3001ms):
Trace[1922666932]: [3.001261094s] [3.001261094s] END
I1010 08:18:14.023910       1 trace.go:205] Trace[651104675]: "cacher list" type:*apiregistration.APIService (10-Oct-2020 08:18:11.023) (total time: 3000ms):
Trace[651104675]: [3.000207385s] [3.000207385s] END
I1010 08:18:14.024880       1 trace.go:205] Trace[1555580661]: "cacher list" type:*batch.CronJob (10-Oct-2020 08:18:11.023) (total time: 3001ms):
Trace[1555580661]: [3.001001908s] [3.001001908s] END
I1010 08:18:14.024933       1 trace.go:205] Trace[884474506]: "List" url:/apis/apiregistration.k8s.io/v1/apiservices,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:11.023) (total time: 3001ms):
Trace[884474506]: [3.001404531s] [3.001404531s] END
I1010 08:18:14.025780       1 trace.go:205] Trace[1609466440]: "List" url:/apis/batch/v1beta1/cronjobs,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:11.023) (total time: 3001ms):
Trace[1609466440]: [3.001991808s] [3.001991808s] END
I1010 08:18:14.085898       1 trace.go:205] Trace[1565270856]: "cacher list" type:*core.LimitRange (10-Oct-2020 08:18:11.085) (total time: 3000ms):
Trace[1565270856]: [3.000411722s] [3.000411722s] END
I1010 08:18:14.086727       1 trace.go:205] Trace[1168933692]: "List" url:/api/v1/limitranges,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:11.085) (total time: 3001ms):
Trace[1168933692]: [3.001250665s] [3.001250665s] END
I1010 08:18:14.118512       1 trace.go:205] Trace[1569922493]: "cacher list" type:*networking.Ingress (10-Oct-2020 08:18:11.118) (total time: 3000ms):
Trace[1569922493]: [3.000378856s] [3.000378856s] END
I1010 08:18:14.119483       1 trace.go:205] Trace[1293737163]: "List" url:/apis/extensions/v1beta1/ingresses,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:11.117) (total time: 3001ms):
Trace[1293737163]: [3.001424768s] [3.001424768s] END
I1010 08:18:14.139340       1 trace.go:205] Trace[1085057639]: "cacher list" type:*rbac.Role (10-Oct-2020 08:18:11.139) (total time: 3000ms):
Trace[1085057639]: [3.000183999s] [3.000183999s] END
I1010 08:18:14.140295       1 trace.go:205] Trace[224089146]: "List" url:/apis/rbac.authorization.k8s.io/v1/roles,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:11.139) (total time: 3001ms):
Trace[224089146]: [3.001175222s] [3.001175222s] END
I1010 08:18:14.212350       1 trace.go:205] Trace[1115505767]: "cacher list" type:*apps.StatefulSet (10-Oct-2020 08:18:11.211) (total time: 3000ms):
Trace[1115505767]: [3.000399645s] [3.000399645s] END
I1010 08:18:14.213363       1 trace.go:205] Trace[1957863707]: "List" url:/apis/apps/v1/statefulsets,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:11.211) (total time: 3001ms):
Trace[1957863707]: [3.001429222s] [3.001429222s] END
I1010 08:18:14.253059       1 trace.go:205] Trace[916749989]: "cacher list" type:*core.ResourceQuota (10-Oct-2020 08:18:11.252) (total time: 3000ms):
Trace[916749989]: [3.000443805s] [3.000443805s] END
I1010 08:18:14.253815       1 trace.go:205] Trace[2057853057]: "List" url:/api/v1/resourcequotas,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:11.252) (total time: 3001ms):
Trace[2057853057]: [3.001333696s] [3.001333696s] END
I1010 08:18:14.392058       1 trace.go:205] Trace[363834564]: "cacher list" type:*policy.PodSecurityPolicy (10-Oct-2020 08:18:11.391) (total time: 3000ms):
Trace[363834564]: [3.000452137s] [3.000452137s] END
I1010 08:18:14.393345       1 trace.go:205] Trace[1790876985]: "List" url:/apis/policy/v1beta1/podsecuritypolicies,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:11.391) (total time: 3001ms):
Trace[1790876985]: [3.001882921s] [3.001882921s] END
I1010 08:18:14.458637       1 trace.go:205] Trace[731457897]: "cacher list" type:*core.Node (10-Oct-2020 08:18:11.458) (total time: 3000ms):
Trace[731457897]: [3.000464499s] [3.000464499s] END
I1010 08:18:14.459526       1 trace.go:205] Trace[1114988864]: "List" url:/api/v1/nodes,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:11.457) (total time: 3001ms):
Trace[1114988864]: [3.001494159s] [3.001494159s] END
I1010 08:18:14.525538       1 trace.go:205] Trace[1841125835]: "cacher list" type:*batch.Job (10-Oct-2020 08:18:11.525) (total time: 3000ms):
Trace[1841125835]: [3.000391679s] [3.000391679s] END
I1010 08:18:14.526900       1 trace.go:205] Trace[163358665]: "List" url:/apis/batch/v1/jobs,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:11.524) (total time: 3001ms):
Trace[163358665]: [3.001812031s] [3.001812031s] END
I1010 08:18:14.689044       1 trace.go:205] Trace[1946543514]: "cacher list" type:*core.ReplicationController (10-Oct-2020 08:18:11.688) (total time: 3000ms):
Trace[1946543514]: [3.000369535s] [3.000369535s] END
I1010 08:18:14.690149       1 trace.go:205] Trace[2102595015]: "List" url:/api/v1/replicationcontrollers,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:11.688) (total time: 3001ms):
Trace[2102595015]: [3.001647227s] [3.001647227s] END
I1010 08:18:14.701139       1 trace.go:205] Trace[1900489069]: "cacher list" type:*networking.NetworkPolicy (10-Oct-2020 08:18:11.700) (total time: 3000ms):
Trace[1900489069]: [3.000460615s] [3.000460615s] END
I1010 08:18:14.701970       1 trace.go:205] Trace[1107089982]: "List" url:/apis/networking.k8s.io/v1/networkpolicies,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:11.700) (total time: 3001ms):
Trace[1107089982]: [3.001315949s] [3.001315949s] END
I1010 08:18:14.708253       1 trace.go:205] Trace[365481466]: "cacher list" type:*core.Service (10-Oct-2020 08:18:11.707) (total time: 3000ms):
Trace[365481466]: [3.000396684s] [3.000396684s] END
I1010 08:18:14.708940       1 trace.go:205] Trace[1867893538]: "List" url:/api/v1/services,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:11.707) (total time: 3001ms):
Trace[1867893538]: [3.001222812s] [3.001222812s] END
I1010 08:18:16.360439       1 client.go:360] parsed scheme: "passthrough"
I1010 08:18:16.360648       1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://10.3.2.30:2379  <nil> 0 <nil>}] <nil> <nil>}
I1010 08:18:16.360794       1 clientconn.go:948] ClientConn switching balancer to "pick_first"
I1010 08:18:18.031130       1 trace.go:205] Trace[401817598]: "cacher list" type:*batch.CronJob (10-Oct-2020 08:18:15.030) (total time: 3000ms):
Trace[401817598]: [3.000660162s] [3.000660162s] END
I1010 08:18:18.032297       1 trace.go:205] Trace[1051985302]: "List" url:/apis/batch/v1beta1/cronjobs,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:15.030) (total time: 3001ms):
Trace[1051985302]: [3.001882931s] [3.001882931s] END
I1010 08:18:18.032532       1 trace.go:205] Trace[613357335]: "cacher list" type:*apiregistration.APIService (10-Oct-2020 08:18:15.031) (total time: 3001ms):
Trace[613357335]: [3.001274525s] [3.001274525s] END
I1010 08:18:18.033962       1 trace.go:205] Trace[141454822]: "List" url:/apis/apiregistration.k8s.io/v1/apiservices,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:15.030) (total time: 3002ms):
Trace[141454822]: [3.002910258s] [3.002910258s] END
I1010 08:18:18.092206       1 trace.go:205] Trace[1797787920]: "cacher list" type:*core.LimitRange (10-Oct-2020 08:18:15.091) (total time: 3000ms):
Trace[1797787920]: [3.000211083s] [3.000211083s] END
I1010 08:18:18.093091       1 trace.go:205] Trace[666164662]: "List" url:/api/v1/limitranges,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:15.091) (total time: 3001ms):
Trace[666164662]: [3.001050722s] [3.001050722s] END
I1010 08:18:18.124679       1 trace.go:205] Trace[832883822]: "cacher list" type:*networking.Ingress (10-Oct-2020 08:18:15.124) (total time: 3000ms):
Trace[832883822]: [3.000347606s] [3.000347606s] END
I1010 08:18:18.125473       1 trace.go:205] Trace[1447237874]: "List" url:/apis/extensions/v1beta1/ingresses,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:15.124) (total time: 3001ms):
Trace[1447237874]: [3.001211408s] [3.001211408s] END
I1010 08:18:18.145243       1 trace.go:205] Trace[1807925171]: "cacher list" type:*rbac.Role (10-Oct-2020 08:18:15.144) (total time: 3000ms):
Trace[1807925171]: [3.000285226s] [3.000285226s] END
I1010 08:18:18.146253       1 trace.go:205] Trace[1316736347]: "List" url:/apis/rbac.authorization.k8s.io/v1/roles,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:15.144) (total time: 3001ms):
Trace[1316736347]: [3.001359112s] [3.001359112s] END
I1010 08:18:18.218859       1 trace.go:205] Trace[1201003027]: "cacher list" type:*apps.StatefulSet (10-Oct-2020 08:18:15.218) (total time: 3000ms):
Trace[1201003027]: [3.000370439s] [3.000370439s] END
I1010 08:18:18.219683       1 trace.go:205] Trace[1776891054]: "List" url:/apis/apps/v1/statefulsets,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:15.218) (total time: 3001ms):
Trace[1776891054]: [3.001248936s] [3.001248936s] END
I1010 08:18:18.259969       1 trace.go:205] Trace[55578172]: "cacher list" type:*core.ResourceQuota (10-Oct-2020 08:18:15.259) (total time: 3000ms):
Trace[55578172]: [3.000419732s] [3.000419732s] END
I1010 08:18:18.260785       1 trace.go:205] Trace[2088535381]: "List" url:/api/v1/resourcequotas,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:15.259) (total time: 3001ms):
Trace[2088535381]: [3.001449538s] [3.001449538s] END
I1010 08:18:18.400080       1 trace.go:205] Trace[286984805]: "cacher list" type:*policy.PodSecurityPolicy (10-Oct-2020 08:18:15.399) (total time: 3000ms):
Trace[286984805]: [3.000360149s] [3.000360149s] END
I1010 08:18:18.400929       1 trace.go:205] Trace[1588941273]: "List" url:/apis/policy/v1beta1/podsecuritypolicies,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:15.399) (total time: 3001ms):
Trace[1588941273]: [3.001389502s] [3.001389502s] END
I1010 08:18:18.466137       1 trace.go:205] Trace[367406855]: "cacher list" type:*core.Node (10-Oct-2020 08:18:15.465) (total time: 3000ms):
Trace[367406855]: [3.000820135s] [3.000820135s] END
I1010 08:18:18.466904       1 trace.go:205] Trace[40803725]: "List" url:/api/v1/nodes,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:15.465) (total time: 3001ms):
Trace[40803725]: [3.001758165s] [3.001758165s] END
I1010 08:18:18.533288       1 trace.go:205] Trace[671059406]: "cacher list" type:*batch.Job (10-Oct-2020 08:18:15.532) (total time: 3000ms):
Trace[671059406]: [3.000370592s] [3.000370592s] END
I1010 08:18:18.534485       1 trace.go:205] Trace[1684130095]: "List" url:/apis/batch/v1/jobs,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:15.532) (total time: 3001ms):
Trace[1684130095]: [3.001715396s] [3.001715396s] END
I1010 08:18:18.696725       1 trace.go:205] Trace[65352147]: "cacher list" type:*core.ReplicationController (10-Oct-2020 08:18:15.696) (total time: 3000ms):
Trace[65352147]: [3.000306983s] [3.000306983s] END
I1010 08:18:18.697567       1 trace.go:205] Trace[908390708]: "List" url:/api/v1/replicationcontrollers,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:15.696) (total time: 3001ms):
Trace[908390708]: [3.00124398s] [3.00124398s] END
I1010 08:18:18.706193       1 trace.go:205] Trace[209399047]: "cacher list" type:*networking.NetworkPolicy (10-Oct-2020 08:18:15.705) (total time: 3000ms):
Trace[209399047]: [3.000304429s] [3.000304429s] END
I1010 08:18:18.707092       1 trace.go:205] Trace[91873839]: "List" url:/apis/networking.k8s.io/v1/networkpolicies,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:15.705) (total time: 3001ms):
Trace[91873839]: [3.001142512s] [3.001142512s] END
I1010 08:18:18.713665       1 trace.go:205] Trace[1727861456]: "cacher list" type:*core.Service (10-Oct-2020 08:18:15.713) (total time: 3000ms):
Trace[1727861456]: [3.000472608s] [3.000472608s] END
I1010 08:18:18.714856       1 trace.go:205] Trace[530452474]: "List" url:/api/v1/services,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:15.713) (total time: 3001ms):
Trace[530452474]: [3.001755984s] [3.001755984s] END
I1010 08:18:22.053312       1 trace.go:205] Trace[1626657639]: "cacher list" type:*apiregistration.APIService (10-Oct-2020 08:18:19.052) (total time: 3000ms):
Trace[1626657639]: [3.000368872s] [3.000368872s] END
I1010 08:18:22.054376       1 trace.go:205] Trace[608885293]: "List" url:/apis/apiregistration.k8s.io/v1/apiservices,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:19.051) (total time: 3002ms):
Trace[608885293]: [3.002388771s] [3.002388771s] END
I1010 08:18:22.054425       1 trace.go:205] Trace[794860036]: "cacher list" type:*batch.CronJob (10-Oct-2020 08:18:19.053) (total time: 3000ms):
Trace[794860036]: [3.000682597s] [3.000682597s] END
I1010 08:18:22.055225       1 trace.go:205] Trace[167020185]: "List" url:/apis/batch/v1beta1/cronjobs,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:19.053) (total time: 3001ms):
Trace[167020185]: [3.001524555s] [3.001524555s] END
I1010 08:18:22.098446       1 trace.go:205] Trace[1326299676]: "cacher list" type:*core.LimitRange (10-Oct-2020 08:18:19.097) (total time: 3000ms):
Trace[1326299676]: [3.000408289s] [3.000408289s] END
I1010 08:18:22.099343       1 trace.go:205] Trace[1953463376]: "List" url:/api/v1/limitranges,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:19.097) (total time: 3001ms):
Trace[1953463376]: [3.001477368s] [3.001477368s] END
I1010 08:18:22.130838       1 trace.go:205] Trace[391287652]: "cacher list" type:*networking.Ingress (10-Oct-2020 08:18:19.130) (total time: 3000ms):
Trace[391287652]: [3.000353869s] [3.000353869s] END
I1010 08:18:22.131650       1 trace.go:205] Trace[690332514]: "List" url:/apis/extensions/v1beta1/ingresses,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:19.130) (total time: 3001ms):
Trace[690332514]: [3.001258022s] [3.001258022s] END
I1010 08:18:22.169955       1 trace.go:205] Trace[341557415]: "cacher list" type:*rbac.Role (10-Oct-2020 08:18:19.169) (total time: 3000ms):
Trace[341557415]: [3.000179324s] [3.000179324s] END
I1010 08:18:22.170929       1 trace.go:205] Trace[1088944675]: "List" url:/apis/rbac.authorization.k8s.io/v1/roles,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:19.169) (total time: 3001ms):
Trace[1088944675]: [3.001241869s] [3.001241869s] END
I1010 08:18:22.225693       1 trace.go:205] Trace[855430652]: "cacher list" type:*apps.StatefulSet (10-Oct-2020 08:18:19.225) (total time: 3000ms):
Trace[855430652]: [3.000469643s] [3.000469643s] END
I1010 08:18:22.226551       1 trace.go:205] Trace[1348513179]: "List" url:/apis/apps/v1/statefulsets,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:19.225) (total time: 3001ms):
Trace[1348513179]: [3.001401948s] [3.001401948s] END
I1010 08:18:22.267944       1 trace.go:205] Trace[393126552]: "cacher list" type:*core.ResourceQuota (10-Oct-2020 08:18:19.267) (total time: 3000ms):
Trace[393126552]: [3.000309966s] [3.000309966s] END
I1010 08:18:22.268798       1 trace.go:205] Trace[2052633673]: "List" url:/api/v1/resourcequotas,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:19.267) (total time: 3001ms):
Trace[2052633673]: [3.001288245s] [3.001288245s] END
I1010 08:18:22.407089       1 trace.go:205] Trace[1114875039]: "cacher list" type:*policy.PodSecurityPolicy (10-Oct-2020 08:18:19.406) (total time: 3000ms):
Trace[1114875039]: [3.000430218s] [3.000430218s] END
I1010 08:18:22.408286       1 trace.go:205] Trace[1294263875]: "List" url:/apis/policy/v1beta1/podsecuritypolicies,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:19.406) (total time: 3001ms):
Trace[1294263875]: [3.001646139s] [3.001646139s] END
I1010 08:18:22.473221       1 trace.go:205] Trace[1456270033]: "cacher list" type:*core.Node (10-Oct-2020 08:18:19.472) (total time: 3000ms):
Trace[1456270033]: [3.000501465s] [3.000501465s] END
I1010 08:18:22.474579       1 trace.go:205] Trace[2025035728]: "List" url:/api/v1/nodes,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:19.472) (total time: 3001ms):
Trace[2025035728]: [3.001929687s] [3.001929687s] END
I1010 08:18:22.540655       1 trace.go:205] Trace[96283934]: "cacher list" type:*batch.Job (10-Oct-2020 08:18:19.540) (total time: 3000ms):
Trace[96283934]: [3.000438768s] [3.000438768s] END
I1010 08:18:22.541799       1 trace.go:205] Trace[1043726116]: "List" url:/apis/batch/v1/jobs,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:19.540) (total time: 3001ms):
Trace[1043726116]: [3.001623821s] [3.001623821s] END
I1010 08:18:22.705348       1 trace.go:205] Trace[520363627]: "cacher list" type:*core.ReplicationController (10-Oct-2020 08:18:19.704) (total time: 3000ms):
Trace[520363627]: [3.000698283s] [3.000698283s] END
I1010 08:18:22.706304       1 trace.go:205] Trace[88603150]: "List" url:/api/v1/replicationcontrollers,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:19.704) (total time: 3001ms):
Trace[88603150]: [3.001713515s] [3.001713515s] END
I1010 08:18:22.711779       1 trace.go:205] Trace[581436437]: "cacher list" type:*networking.NetworkPolicy (10-Oct-2020 08:18:19.711) (total time: 3000ms):
Trace[581436437]: [3.000417399s] [3.000417399s] END
I1010 08:18:22.712525       1 trace.go:205] Trace[1140267459]: "List" url:/apis/networking.k8s.io/v1/networkpolicies,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:19.711) (total time: 3001ms):
Trace[1140267459]: [3.001209328s] [3.001209328s] END
I1010 08:18:22.719510       1 trace.go:205] Trace[1271342841]: "cacher list" type:*core.Service (10-Oct-2020 08:18:19.719) (total time: 3000ms):
Trace[1271342841]: [3.000358153s] [3.000358153s] END
I1010 08:18:22.720304       1 trace.go:205] Trace[457546719]: "List" url:/api/v1/services,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:19.718) (total time: 3001ms):
Trace[457546719]: [3.001303885s] [3.001303885s] END
I1010 08:18:26.061577       1 trace.go:205] Trace[370960936]: "cacher list" type:*apiregistration.APIService (10-Oct-2020 08:18:23.061) (total time: 3000ms):
Trace[370960936]: [3.000209916s] [3.000209916s] END
I1010 08:18:26.062513       1 trace.go:205] Trace[1221914970]: "cacher list" type:*batch.CronJob (10-Oct-2020 08:18:23.061) (total time: 3001ms):
Trace[1221914970]: [3.001102342s] [3.001102342s] END
I1010 08:18:26.062836       1 trace.go:205] Trace[2035453588]: "List" url:/apis/apiregistration.k8s.io/v1/apiservices,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:23.061) (total time: 3001ms):
Trace[2035453588]: [3.001733399s] [3.001733399s] END
I1010 08:18:26.063246       1 trace.go:205] Trace[1743254839]: "List" url:/apis/batch/v1beta1/cronjobs,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:23.061) (total time: 3002ms):
Trace[1743254839]: [3.002036852s] [3.002036852s] END
I1010 08:18:26.105046       1 trace.go:205] Trace[1637135937]: "cacher list" type:*core.LimitRange (10-Oct-2020 08:18:23.104) (total time: 3000ms):
Trace[1637135937]: [3.000433443s] [3.000433443s] END
I1010 08:18:26.106013       1 trace.go:205] Trace[2122907320]: "List" url:/api/v1/limitranges,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:23.104) (total time: 3001ms):
Trace[2122907320]: [3.001522998s] [3.001522998s] END
I1010 08:18:26.153269       1 trace.go:205] Trace[1346582090]: "cacher list" type:*networking.Ingress (10-Oct-2020 08:18:23.152) (total time: 3000ms):
Trace[1346582090]: [3.000494459s] [3.000494459s] END
I1010 08:18:26.154116       1 trace.go:205] Trace[621980133]: "List" url:/apis/extensions/v1beta1/ingresses,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:23.152) (total time: 3001ms):
Trace[621980133]: [3.001643362s] [3.001643362s] END
I1010 08:18:26.176148       1 trace.go:205] Trace[282840038]: "cacher list" type:*rbac.Role (10-Oct-2020 08:18:23.175) (total time: 3000ms):
Trace[282840038]: [3.000344026s] [3.000344026s] END
I1010 08:18:26.176863       1 trace.go:205] Trace[987973920]: "List" url:/apis/rbac.authorization.k8s.io/v1/roles,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:23.175) (total time: 3001ms):
Trace[987973920]: [3.001183448s] [3.001183448s] END
I1010 08:18:26.232554       1 trace.go:205] Trace[317155411]: "cacher list" type:*apps.StatefulSet (10-Oct-2020 08:18:23.232) (total time: 3000ms):
Trace[317155411]: [3.000376532s] [3.000376532s] END
I1010 08:18:26.233723       1 trace.go:205] Trace[859744590]: "List" url:/apis/apps/v1/statefulsets,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:23.231) (total time: 3001ms):
Trace[859744590]: [3.001636695s] [3.001636695s] END
I1010 08:18:26.274191       1 trace.go:205] Trace[1801502677]: "cacher list" type:*core.ResourceQuota (10-Oct-2020 08:18:23.273) (total time: 3000ms):
Trace[1801502677]: [3.000796734s] [3.000796734s] END
I1010 08:18:26.274868       1 trace.go:205] Trace[1212864065]: "List" url:/api/v1/resourcequotas,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:23.273) (total time: 3001ms):
Trace[1212864065]: [3.001541267s] [3.001541267s] END
I1010 08:18:26.414369       1 trace.go:205] Trace[1659112737]: "cacher list" type:*policy.PodSecurityPolicy (10-Oct-2020 08:18:23.414) (total time: 3000ms):
Trace[1659112737]: [3.000229632s] [3.000229632s] END
I1010 08:18:26.415252       1 trace.go:205] Trace[1244872078]: "List" url:/apis/policy/v1beta1/podsecuritypolicies,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:23.413) (total time: 3001ms):
Trace[1244872078]: [3.001256552s] [3.001256552s] END
I1010 08:18:26.480737       1 trace.go:205] Trace[924685880]: "cacher list" type:*core.Node (10-Oct-2020 08:18:23.480) (total time: 3000ms):
Trace[924685880]: [3.000275408s] [3.000275408s] END
I1010 08:18:26.481757       1 trace.go:205] Trace[1374682132]: "List" url:/api/v1/nodes,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:23.480) (total time: 3001ms):
Trace[1374682132]: [3.001330725s] [3.001330725s] END
I1010 08:18:26.548171       1 trace.go:205] Trace[224736911]: "cacher list" type:*batch.Job (10-Oct-2020 08:18:23.547) (total time: 3000ms):
Trace[224736911]: [3.000584182s] [3.000584182s] END
I1010 08:18:26.549480       1 trace.go:205] Trace[583540957]: "List" url:/apis/batch/v1/jobs,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:23.547) (total time: 3002ms):
Trace[583540957]: [3.002018596s] [3.002018596s] END
I1010 08:18:26.712856       1 trace.go:205] Trace[1106768081]: "cacher list" type:*core.ReplicationController (10-Oct-2020 08:18:23.712) (total time: 3000ms):
Trace[1106768081]: [3.000389389s] [3.000389389s] END
I1010 08:18:26.713766       1 trace.go:205] Trace[784818554]: "List" url:/api/v1/replicationcontrollers,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:23.712) (total time: 3001ms):
Trace[784818554]: [3.001346578s] [3.001346578s] END
I1010 08:18:26.717515       1 trace.go:205] Trace[1683063015]: "cacher list" type:*networking.NetworkPolicy (10-Oct-2020 08:18:23.717) (total time: 3000ms):
Trace[1683063015]: [3.000365781s] [3.000365781s] END
I1010 08:18:26.718674       1 trace.go:205] Trace[1947651936]: "List" url:/apis/networking.k8s.io/v1/networkpolicies,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:23.716) (total time: 3001ms):
Trace[1947651936]: [3.001616984s] [3.001616984s] END
I1010 08:18:26.725154       1 trace.go:205] Trace[286114399]: "cacher list" type:*core.Service (10-Oct-2020 08:18:23.724) (total time: 3000ms):
Trace[286114399]: [3.000360276s] [3.000360276s] END
I1010 08:18:26.726161       1 trace.go:205] Trace[814655251]: "List" url:/api/v1/services,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format cluster c-tg8nj,client:127.0.0.1 (10-Oct-2020 08:18:23.724) (total time: 3001ms):
Trace[814655251]: [3.001484155s] [3.001484155s] END
I1010 08:18:55.701899       1 client.go:360] parsed scheme: "passthrough"
I1010 08:18:55.702395       1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://10.3.2.30:2379  <nil> 0 <nil>}] <nil> <nil>}
I1010 08:18:55.702548       1 clientconn.go:948] ClientConn switching balancer to "pick_first"
I1010 08:19:40.648696       1 client.go:360] parsed scheme: "passthrough"
I1010 08:19:40.649094       1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://10.3.2.30:2379  <nil> 0 <nil>}] <nil> <nil>}
I1010 08:19:40.649240       1 clientconn.go:948] ClientConn switching balancer to "pick_first"
I1010 08:20:13.938750       1 client.go:360] parsed scheme: "passthrough"
I1010 08:20:13.939011       1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://10.3.2.30:2379  <nil> 0 <nil>}] <nil> <nil>}
I1010 08:20:13.939069       1 clientconn.go:948] ClientConn switching balancer to "pick_first"
I1010 08:20:47.803514       1 client.go:360] parsed scheme: "passthrough"
I1010 08:20:47.803821       1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://10.3.2.30:2379  <nil> 0 <nil>}] <nil> <nil>}
I1010 08:20:47.803923       1 clientconn.go:948] ClientConn switching balancer to "pick_first"
I1010 08:21:18.708335       1 client.go:360] parsed scheme: "passthrough"
I1010 08:21:18.708514       1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://10.3.2.30:2379  <nil> 0 <nil>}] <nil> <nil>}
I1010 08:21:18.708564       1 clientconn.go:948] ClientConn switching balancer to "pick_first"
I1010 08:22:00.565654       1 client.go:360] parsed scheme: "passthrough"
I1010 08:22:00.566022       1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://10.3.2.30:2379  <nil> 0 <nil>}] <nil> <nil>}
I1010 08:22:00.566176       1 clientconn.go:948] ClientConn switching balancer to "pick_first"
I1010 08:22:31.852137       1 client.go:360] parsed scheme: "passthrough"
I1010 08:22:31.852505       1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://10.3.2.30:2379  <nil> 0 <nil>}] <nil> <nil>}
I1010 08:22:31.852594       1 clientconn.go:948] ClientConn switching balancer to "pick_first"
I1010 08:23:08.373594       1 client.go:360] parsed scheme: "passthrough"
I1010 08:23:08.373896       1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://10.3.2.30:2379  <nil> 0 <nil>}] <nil> <nil>}
I1010 08:23:08.374022       1 clientconn.go:948] ClientConn switching balancer to "pick_first"
I1010 08:23:46.866416       1 client.go:360] parsed scheme: "passthrough"
I1010 08:23:46.866806       1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://10.3.2.30:2379  <nil> 0 <nil>}] <nil> <nil>}
I1010 08:23:46.866882       1 clientconn.go:948] ClientConn switching balancer to "pick_first"
I1010 08:24:19.937433       1 client.go:360] parsed scheme: "passthrough"
I1010 08:24:19.937765       1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://10.3.2.30:2379  <nil> 0 <nil>}] <nil> <nil>}
I1010 08:24:19.937871       1 clientconn.go:948] ClientConn switching balancer to "pick_first"
I1010 08:24:55.652339       1 client.go:360] parsed scheme: "passthrough"
I1010 08:24:55.652651       1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://10.3.2.30:2379  <nil> 0 <nil>}] <nil> <nil>}
I1010 08:24:55.652747       1 clientconn.go:948] ClientConn switching balancer to "pick_first"
I1010 08:25:36.315698       1 client.go:360] parsed scheme: "passthrough"
I1010 08:25:36.316065       1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://10.3.2.30:2379  <nil> 0 <nil>}] <nil> <nil>}
I1010 08:25:36.316149       1 clientconn.go:948] ClientConn switching balancer to "pick_first"
I1010 08:26:15.805057       1 client.go:360] parsed scheme: "passthrough"
I1010 08:26:15.805380       1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://10.3.2.30:2379  <nil> 0 <nil>}] <nil> <nil>}
I1010 08:26:15.805457       1 clientconn.go:948] ClientConn switching balancer to "pick_first"
I1010 08:26:57.632858       1 client.go:360] parsed scheme: "passthrough"
I1010 08:26:57.633151       1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://10.3.2.30:2379  <nil> 0 <nil>}] <nil> <nil>}
I1010 08:26:57.633206       1 clientconn.go:948] ClientConn switching balancer to "pick_first"

@yceoshda
Copy link

I have the same kind of issue when provisionning a new custom cluster. I guess since I had the problem described here #16454 It's the same kind of issue (cluster's DNS cannot resolv my Rancher's hostname since it's not public). The added trouble being that unlike in 2.4.8 where I could edit the cattle-cluster-agent deployment to change the dnsConfig ... this time I can't ... rendering the provisionned cluster useless.

@kwims
Copy link

kwims commented Oct 19, 2020

Same issue here for the newer version 2.5.1. worked find for version 2.4.8

@sowmyav27
Copy link
Contributor Author

@slash1387 Did the cluster recover after the error ? Because this error mentioned in the original issue was an intermittent error (always reproducible and cluster would recover after about 5 minutes). If the error persisted, you could log a new issue to track it.

@kwims
Copy link

kwims commented Oct 20, 2020

@sowmyav27 like I said I got the exact same problem and after several hour ( like 6h) It still did not recover. So in my oppinion the bug reported by @slash1387 still exists

@sowmyav27
Copy link
Contributor Author

@kwims The error mentioned in the original issue was an intermittent error (always reproducible and cluster would recover after about 5 minutes). If the error persisted, could you please log a new issue to track it.

@deniseschannon
Copy link

As this is a transient messgae, this is expected.

@sowmyav27
Copy link
Contributor Author

After an offline conversation with @deniseschannon , changing it to 2.5.7 for it to be looked at

@sowmyav27 sowmyav27 added this to the v2.5.7 milestone Jan 14, 2021
@zube zube bot removed this from the v2.5.7 milestone Jan 14, 2021
@janeczku
Copy link
Contributor

janeczku commented Jan 25, 2021

An average user seeing this error lasting for more than a minute would not interpret this as a transient message but rather assume that the cluster provisioning has failed.
If it's a transient message it should not be presented as an error ?

@luthermonson luthermonson self-assigned this Jan 28, 2021
@luthermonson
Copy link
Contributor

#30427

@sowmyav27
Copy link
Contributor Author

2.5 PR: #31039

@jiaqiluo
Copy link
Member

The bug fix is validated on the following two rancher versions:

  • rancher/rancher:v2.5-head 2ae4c
  • rancher/rancher:master-head d160b

Steps:

  • run rancher single-install
  • provision a clsuer with EC2 node-driver, the cluster contains 1 etcd, 1 control plan, and 3 workers
  • Monitor the changes of the cluster's state during the whole provisioning process

Results:

  • the state or cluster never become Error
  • the following error never show up: Cluster health check failed: cluster agent is not ready
  • the cluster is provisioned successfully

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/bug-qa Issues that have not yet hit a real release. Bugs introduced by a new feature or enhancement
Projects
None yet
Development

No branches or pull requests