Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Spire integration with Istio, Istio proxy readiness probe failed: HTTP probe failed with statuscode: 503 #43268

Closed
RunlinXiao opened this issue Feb 10, 2023 · 12 comments
Closed

Spire integration with Istio, Istio proxy readiness probe failed: HTTP probe failed with statuscode: 503 #43268

RunlinXiao opened this issue Feb 10, 2023 · 12 comments
Labels
lifecycle/automatically-closed Indicates a PR or issue that has been closed automatically. lifecycle/stale Indicates a PR or issue hasn't been manipulated by an Istio team member for a while

Comments

@RunlinXiao
Copy link

RunlinXiao commented Feb 10, 2023 •
edited
Loading

Hi, I'm following the Istio/SPIRE doc to integrate the Istio with Spire, but the logs of Istiod shows Istio proxy readiness probe keep failing.. Even all the pods are showing running. Any idea? Thanks!

Screen Shot 2023-02-09 at 6 36 23 PM

logs of Istiod pod(fail)

Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Normal   Scheduled  20m   default-scheduler  Successfully assigned istio-system/istiod-7f8c8bb8c8-vqdlh to gke-k8saas-k8s-bvzvb-default-pool--v0-e23daf9d-dx76
  Normal   Pulling    20m   kubelet            Pulling image "docker.io/istio/pilot:1.16.1"
  Normal   Pulled     19m   kubelet            Successfully pulled image "docker.io/istio/pilot:1.16.1" in 4.739937871s
  Normal   Created    19m   kubelet            Created container discovery
  Normal   Started    19m   kubelet            Started container discovery
  Warning  Unhealthy  19m   kubelet            Readiness probe failed: HTTP probe failed with statuscode: 503

logs of Istio-ingressgateway(normal)

Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  26m   default-scheduler  Successfully assigned istio-system/istio-ingressgateway-7b4455498-l5bdz to gke-k8saas-k8s-bvzvb-default-pool--v0-e23daf9d-dx76
  Normal  Pulling    26m   kubelet            Pulling image "docker.io/istio/proxyv2:1.16.1"
  Normal  Pulled     26m   kubelet            Successfully pulled image "docker.io/istio/proxyv2:1.16.1" in 2.281250512s
  Normal  Created    26m   kubelet            Created container istio-proxy
  Normal  Started    26m   kubelet            Started container istio-proxy

Steps to reproduce:

  1. Download Istio release 1.16.1
  2. Change to dir istio-1.16.1/samples/security/spire
  3. Run kubectl apply -f spire-quickstart.yaml
  4. Run kubectl wait pod --for=condition=ready -n spire -l app=spire-agent
  5. Run istioctl install -f istio-spire-config.yaml
  6. kubectl get all -n istio-system
@keithmattix
Copy link
Contributor

/cc @jaellio who's been working on updating some of these docs

@howardjohn
Copy link
Member

the kubectl logs will show more than events

@RunlinXiao
Copy link
Author

RunlinXiao commented Feb 10, 2023 •
edited
Loading

@howardjohn Hi John, below are the full logs of the istiod pod. Also when other service (e.g. druid service)created with istio-injection=enabled, the logs of the service (druid) also show the istio proxy probe failed:

  Normal   Started    25m                   kubelet            Started container istio-proxy
  Normal   Created    25m                   kubelet            Created container istio-proxy
  Warning  Unhealthy  25m (x3 over 25m)     kubelet            Readiness probe failed: Get "http://10.111.197.38:15021/healthz/ready": dial tcp 10.111.197.38:15021: connect: connection refused

❯ kb logs -f pod/istiod-7f8c8bb8c8-vqdlh -n istio-system

2023-02-10T01:13:44.880569Z	info	FLAG: --caCertFile=""
2023-02-10T01:13:44.880605Z	info	FLAG: --clusterAliases="[]"
2023-02-10T01:13:44.880610Z	info	FLAG: --clusterID="Kubernetes"
2023-02-10T01:13:44.880616Z	info	FLAG: --clusterRegistriesNamespace="istio-system"
2023-02-10T01:13:44.880620Z	info	FLAG: --configDir=""
2023-02-10T01:13:44.880624Z	info	FLAG: --ctrlz_address="localhost"
2023-02-10T01:13:44.880628Z	info	FLAG: --ctrlz_port="9876"
2023-02-10T01:13:44.880633Z	info	FLAG: --domain="cluster.local"
2023-02-10T01:13:44.880639Z	info	FLAG: --grpcAddr=":15010"
2023-02-10T01:13:44.880645Z	info	FLAG: --help="false"
2023-02-10T01:13:44.880650Z	info	FLAG: --httpAddr=":8080"
2023-02-10T01:13:44.880656Z	info	FLAG: --httpsAddr=":15017"
2023-02-10T01:13:44.880664Z	info	FLAG: --keepaliveInterval="30s"
2023-02-10T01:13:44.880669Z	info	FLAG: --keepaliveMaxServerConnectionAge="30m0s"
2023-02-10T01:13:44.880674Z	info	FLAG: --keepaliveTimeout="10s"
2023-02-10T01:13:44.880679Z	info	FLAG: --kubeconfig=""
2023-02-10T01:13:44.880685Z	info	FLAG: --kubernetesApiBurst="160"
2023-02-10T01:13:44.880693Z	info	FLAG: --kubernetesApiQPS="80"
2023-02-10T01:13:44.880698Z	info	FLAG: --log_as_json="false"
2023-02-10T01:13:44.880704Z	info	FLAG: --log_caller=""
2023-02-10T01:13:44.880710Z	info	FLAG: --log_output_level="default:info"
2023-02-10T01:13:44.880715Z	info	FLAG: --log_rotate=""
2023-02-10T01:13:44.880720Z	info	FLAG: --log_rotate_max_age="30"
2023-02-10T01:13:44.880756Z	info	FLAG: --log_rotate_max_backups="1000"
2023-02-10T01:13:44.880762Z	info	FLAG: --log_rotate_max_size="104857600"
2023-02-10T01:13:44.880769Z	info	FLAG: --log_stacktrace_level="default:none"
2023-02-10T01:13:44.880781Z	info	FLAG: --log_target="[stdout]"
2023-02-10T01:13:44.880792Z	info	FLAG: --meshConfig="./etc/istio/config/mesh"
2023-02-10T01:13:44.880797Z	info	FLAG: --monitoringAddr=":15014"
2023-02-10T01:13:44.880802Z	info	FLAG: --namespace="istio-system"
2023-02-10T01:13:44.880807Z	info	FLAG: --networksConfig="./etc/istio/config/meshNetworks"
2023-02-10T01:13:44.880812Z	info	FLAG: --profile="true"
2023-02-10T01:13:44.880832Z	info	FLAG: --registries="[Kubernetes]"
2023-02-10T01:13:44.880842Z	info	FLAG: --secureGRPCAddr=":15012"
2023-02-10T01:13:44.880847Z	info	FLAG: --shutdownDuration="10s"
2023-02-10T01:13:44.880856Z	info	FLAG: --tls-cipher-suites="[]"
2023-02-10T01:13:44.880861Z	info	FLAG: --tlsCertFile=""
2023-02-10T01:13:44.880865Z	info	FLAG: --tlsKeyFile=""
2023-02-10T01:13:44.880877Z	info	FLAG: --vklog="0"
2023-02-10T01:13:44.899642Z	info	initializing Istiod admin server
2023-02-10T01:13:44.900133Z	info	starting HTTP service at [::]:8080
2023-02-10T01:13:44.900268Z	info	klog	Config not found: /var/run/secrets/remote/config
2023-02-10T01:13:44.909473Z	info	initializing mesh configuration ./etc/istio/config/mesh
2023-02-10T01:13:44.916360Z	info	controllers	starting	controller=configmap istio
2023-02-10T01:13:44.916752Z	info	Loaded MeshNetworks config from Kubernetes API server.
2023-02-10T01:13:44.916783Z	info	mesh networks configuration updated to: {

}
2023-02-10T01:13:44.923510Z	info	Loaded MeshConfig config from Kubernetes API server.
2023-02-10T01:13:44.924246Z	info	mesh configuration updated to: {
    "proxyListenPort": 15001,
    "connectTimeout": "10s",
    "protocolDetectionTimeout": "0s",
    "ingressClass": "istio",
    "ingressService": "istio-ingressgateway",
    "ingressControllerMode": "STRICT",
    "enableTracing": true,
    "accessLogFile": "/dev/stdout",
    "defaultConfig": {
        "configPath": "./etc/istio/proxy",
        "binaryPath": "/usr/local/bin/envoy",
        "serviceCluster": "istio-proxy",
        "drainDuration": "45s",
        "parentShutdownDuration": "60s",
        "discoveryAddress": "istiod.istio-system.svc:15012",
        "proxyAdminPort": 15000,
        "controlPlaneAuthPolicy": "MUTUAL_TLS",
        "statNameLength": 189,
        "concurrency": 2,
        "tracing": {
            "zipkin": {
                "address": "zipkin.istio-system:9411"
            }
        },
        "statusPort": 15020,
        "terminationDrainDuration": "5s"
    },
    "outboundTrafficPolicy": {
        "mode": "ALLOW_ANY"
    },
    "enableAutoMtls": true,
    "trustDomain": "example.org",
    "defaultServiceExportTo": [
        "*"
    ],
    "defaultVirtualServiceExportTo": [
        "*"
    ],
    "defaultDestinationRuleExportTo": [
        "*"
    ],
    "rootNamespace": "istio-system",
    "localityLbSetting": {
        "enabled": true
    },
    "dnsRefreshRate": "60s",
    "thriftConfig": {

    },
    "enablePrometheusMerge": true,
    "extensionProviders": [
        {
            "name": "prometheus",
            "prometheus": {

            }
        },
        {
            "name": "stackdriver",
            "stackdriver": {

            }
        },
        {
            "name": "envoy",
            "envoyFileAccessLog": {
                "path": "/dev/stdout"
            }
        }
    ],
    "defaultProviders": {

    }
}
2023-02-10T01:13:44.924304Z	info	initializing mesh networks from mesh config watcher
2023-02-10T01:13:44.924503Z	info	mesh configuration: {
    "proxyListenPort": 15001,
    "connectTimeout": "10s",
    "protocolDetectionTimeout": "0s",
    "ingressClass": "istio",
    "ingressService": "istio-ingressgateway",
    "ingressControllerMode": "STRICT",
    "enableTracing": true,
    "accessLogFile": "/dev/stdout",
    "defaultConfig": {
        "configPath": "./etc/istio/proxy",
        "binaryPath": "/usr/local/bin/envoy",
        "serviceCluster": "istio-proxy",
        "drainDuration": "45s",
        "parentShutdownDuration": "60s",
        "discoveryAddress": "istiod.istio-system.svc:15012",
        "proxyAdminPort": 15000,
        "controlPlaneAuthPolicy": "MUTUAL_TLS",
        "statNameLength": 189,
        "concurrency": 2,
        "tracing": {
            "zipkin": {
                "address": "zipkin.istio-system:9411"
            }
        },
        "statusPort": 15020,
        "terminationDrainDuration": "5s"
    },
    "outboundTrafficPolicy": {
        "mode": "ALLOW_ANY"
    },
    "enableAutoMtls": true,
    "trustDomain": "example.org",
    "defaultServiceExportTo": [
        "*"
    ],
    "defaultVirtualServiceExportTo": [
        "*"
    ],
    "defaultDestinationRuleExportTo": [
        "*"
    ],
    "rootNamespace": "istio-system",
    "localityLbSetting": {
        "enabled": true
    },
    "dnsRefreshRate": "60s",
    "thriftConfig": {

    },
    "enablePrometheusMerge": true,
    "extensionProviders": [
        {
            "name": "prometheus",
            "prometheus": {

            }
        },
        {
            "name": "stackdriver",
            "stackdriver": {

            }
        },
        {
            "name": "envoy",
            "envoyFileAccessLog": {
                "path": "/dev/stdout"
            }
        }
    ],
    "defaultProviders": {

    }
}
2023-02-10T01:13:44.924520Z	info	version: 1.16.1-f6d7bf648e571a6a523210d97bde8b489250354b-Clean
2023-02-10T01:13:44.924841Z	info	flags: {
   "ServerOptions": {
      "HTTPAddr": ":8080",
      "HTTPSAddr": ":15017",
      "GRPCAddr": ":15010",
      "MonitoringAddr": ":15014",
      "EnableProfiling": true,
      "TLSOptions": {
         "CaCertFile": "",
         "CertFile": "",
         "KeyFile": "",
         "TLSCipherSuites": null,
         "CipherSuits": null
      },
      "SecureGRPCAddr": ":15012"
   },
   "InjectionOptions": {
      "InjectionDirectory": "./var/lib/istio/inject"
   },
   "PodName": "istiod-7f8c8bb8c8-vqdlh",
   "Namespace": "istio-system",
   "Revision": "default",
   "MeshConfigFile": "./etc/istio/config/mesh",
   "NetworksConfigFile": "./etc/istio/config/meshNetworks",
   "RegistryOptions": {
      "FileDir": "",
      "Registries": [
         "Kubernetes"
      ],
      "KubeOptions": {
         "SystemNamespace": "",
         "MeshServiceController": null,
         "DomainSuffix": "cluster.local",
         "ClusterID": "Kubernetes",
         "ClusterAliases": {},
         "Metrics": null,
         "XDSUpdater": null,
         "NetworksWatcher": null,
         "MeshWatcher": null,
         "EndpointMode": 1,
         "KubernetesAPIQPS": 80,
         "KubernetesAPIBurst": 160,
         "SyncTimeout": 0,
         "DiscoveryNamespacesFilter": null
      },
      "ClusterRegistriesNamespace": "istio-system",
      "KubeConfig": "",
      "DistributionCacheRetention": 60000000000,
      "DistributionTrackingEnabled": false
   },
   "CtrlZOptions": {
      "Port": 9876,
      "Address": "localhost"
   },
   "KeepaliveOptions": {
      "Time": 30000000000,
      "Timeout": 10000000000,
      "MaxServerConnectionAge": 1800000000000,
      "MaxServerConnectionAgeGrace": 10000000000
   },
   "ShutdownDuration": 10000000000,
   "JwtRule": ""
}
2023-02-10T01:13:44.924858Z	info	initializing mesh handlers
2023-02-10T01:13:44.924928Z	info	model	reloading network gateways
2023-02-10T01:13:44.924953Z	info	creating CA and initializing public key
2023-02-10T01:13:44.924976Z	info	Using istiod file format for signing ca files
2023-02-10T01:13:44.925001Z	info	Use self-signed certificate as the CA certificate
2023-02-10T01:13:44.928369Z	info	pkica	Failed to get secret (error: secrets "istio-ca-secret" not found), will create one
2023-02-10T01:13:45.275072Z	info	pkica	Using self-generated public key: -----BEGIN CERTIFICATE-----
MIIC+DCCAeCgAwIBAgIQQ8+bOfFFaYUMmDVw88tlWzANBgkqhkiG9w0BAQsFADAW
MRQwEgYDVQQKEwtleGFtcGxlLm9yZzAeFw0yMzAyMTAwMTEzNDVaFw0zMzAyMDcw
MTEzNDVaMBYxFDASBgNVBAoTC2V4YW1wbGUub3JnMIIBIjANBgkqhkiG9w0BAQEF
AAOCAQ8AMIIBCgKCAQEAwWwFK1fBNrGwtVLkhV8XZ6RNHo0iP5aRknCpyt6I3Ytd
61FMzLOXBoOYZWSYILSEegE0THcUTp5NwSIAPehww+JG0PspNgfjodNbvTD8zgDI
dYS6AWfq3N1dEq2E4s/H9nqwJWhHu04KOOh2Fukv2tqFlCGCQ6cYYnsAqzPaOrsr
OEcBauilnJkTNXnJZyFtp4uqXyXmjYjlGTLQpbOPsnSLSS4vvMJ+z1zgnmepcMgJ
wYpusOEhIYupuD0bVgJhi356Zd9Yxof11cUFwwZxAEGRO2ryNwDiRQB/idRGO5w1
ZSjYYlaCKCEV3doiCbh2ACFeVVovoHiWdkE8AZxUcQIDAQABo0IwQDAOBgNVHQ8B
Af8EBAMCAgQwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUSZIx6YnlJzZsyPUp
bR2YxbX19TAwDQYJKoZIhvcNAQELBQADggEBAJtPCWPHk5Iu9ojS36v2sTTsf/1z
N9YY7nUZeWjWVcUMleKb5uVfgSSJi1jIf5zEl73uuDA4U2yKhXx9WQfbelemX8Z9
KexFWQcaaSrOAXQ+5xXkVuWzWPywcZiwWCexDnDPImxn0VxOrxwr8UdD7u9WaLT4
IlmKTJ4pzDmYsQflFeNG/kQW9wRMdBVDCDCTkyJWNHkguWK7bv7ny2F0LWtyYZKr
N8xtg8oR6s8Qj+VJ/AqnyOn6VWEY1M4Zcf/eVb0I/KvGureOJSS4G+WsR+CBkYBB
QDBLWPuYNzjSEDn71yGy6926sfI3tVMlMGjGd80WcAb/EXGa8SvJ8HtdDaU=
-----END CERTIFICATE-----

2023-02-10T01:13:45.275115Z	info	rootcertrotator	Set up back off time 31m29s to start rotator.
2023-02-10T01:13:45.275146Z	info	initializing controllers
2023-02-10T01:13:45.275199Z	info	rootcertrotator	Jitter is enabled, wait 31m29s before starting root cert rotator.
2023-02-10T01:13:45.275223Z	info	No certificates specified, skipping K8S DNS certificate controller
2023-02-10T01:13:45.370982Z	warn	kube	Skipping CRD gateway.networking.k8s.io/v1alpha2/ReferenceGrant as it is not present	controller=crd-controller
2023-02-10T01:13:45.371016Z	warn	kube	Skipping CRD gateway.networking.k8s.io/v1alpha2/ReferencePolicy as it is not present	controller=crd-controller
2023-02-10T01:13:45.371022Z	warn	kube	Skipping CRD gateway.networking.k8s.io/v1alpha2/TCPRoute as it is not present	controller=crd-controller
2023-02-10T01:13:45.371028Z	warn	kube	Skipping CRD gateway.networking.k8s.io/v1alpha2/TLSRoute as it is not present	controller=crd-controller
2023-02-10T01:13:45.371032Z	warn	kube	Skipping CRD gateway.networking.k8s.io/v1beta1/GatewayClass as it is not present	controller=crd-controller
2023-02-10T01:13:45.371038Z	warn	kube	Skipping CRD gateway.networking.k8s.io/v1beta1/Gateway as it is not present	controller=crd-controller
2023-02-10T01:13:45.371044Z	warn	kube	Skipping CRD gateway.networking.k8s.io/v1beta1/HTTPRoute as it is not present	controller=crd-controller
2023-02-10T01:13:45.371407Z	info	Adding Kubernetes registry adapter
2023-02-10T01:13:45.371484Z	info	initializing Istiod DNS certificates host: istiod.istio-system.svc, custom host:
2023-02-10T01:13:45.568892Z	warn	discovery is not ready: <nil>
2023-02-10T01:13:45.588905Z	info	Generating istiod-signed cert for [istiod.istio-system.svc istiod-remote.istio-system.svc istio-pilot.istio-system.svc]:
 -----BEGIN CERTIFICATE-----
MIIDZzCCAk+gAwIBAgIQQfwhPIuEc9R20h0rWQyb2zANBgkqhkiG9w0BAQsFADAW
MRQwEgYDVQQKEwtleGFtcGxlLm9yZzAeFw0yMzAyMTAwMTExNDVaFw0zMzAyMDcw
MTEzNDVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDG38fEfiSY
yzye7Exra3Iru3Vf/wGjZho80hUTpuEfUHWHsWuPu3TuVHmiDFJ4iJcPIOo/OaAn
LxLSu74r7aIJIhPvalUm7tt2L82fjgceInWI0mzwDaSx5s8iQXeCduOwPKFNt/FY
dwijZnnPXMig3GqeSQi6NnbseainA5VVKXDgEpTVRusr4MPG4NO9Q8WSgFdYEcj2
WymVD+zzraFeNSM7QD4SwvGsQRvjAAd1kizl8w2uSUD0p4jOQ1+21L6bvC8XmAcC
IwFOl2q+/XirGEX2dUbIUBzrtSys16c6MownSnP6tH21ZcJ/qr3zKYRfaQW+AX6t
/5Yos4CzLoIpAgMBAAGjgcYwgcMwDgYDVR0PAQH/BAQDAgWgMB0GA1UdJQQWMBQG
CCsGAQUFBwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFEmS
MemJ5Sc2bMj1KW0dmMW19fUwMGMGA1UdEQEB/wRZMFeCF2lzdGlvZC5pc3Rpby1z
eXN0ZW0uc3Zjgh5pc3Rpb2QtcmVtb3RlLmlzdGlvLXN5c3RlbS5zdmOCHGlzdGlv
LXBpbG90LmlzdGlvLXN5c3RlbS5zdmMwDQYJKoZIhvcNAQELBQADggEBACoPh1dN
BKhPeJcvoMfRLPJ5KrFh6XP9ONIEC6eDhKnaQtNWuxQQYTXraTKZU41cqlip5jfR
1mxurDHzic5U/aAGm2zOQl4JzLC/DjUFqylTUBAc8sGsRzXPLQSy+MfRaMWSZXQg
y2hJi22GgIx/oST9LhHIRXC4jb7KDEQj34ecxqvMvjIcaMNFpTVyBZ7EhhNjM7wu
ndup/DcMFlff0dY/+3JVRgqQULd8S88ZSY7w/LmSAfBQlGPLmqTauoQpcURnYBVq
mpyQGu0z6oRrryNNPRnwJVy+Ajkd7uSkLkVUWZr/ULudDFM1FMAD2Pof91NOECTc
XST9TTvOu4YwkR8=
-----END CERTIFICATE-----

2023-02-10T01:13:45.588973Z	info	Using istiod file format for signing ca files
2023-02-10T01:13:45.588991Z	info	No plugged-in cert at etc/cacerts/ca-key.pem; self-signed cert is used
2023-02-10T01:13:45.589189Z	info	x509 cert - Issuer: "O=example.org", Subject: "", SN: 41fc213c8b8473d476d21d2b590c9bdb, NotBefore: "2023-02-10T01:11:45Z", NotAfter: "2033-02-07T01:13:45Z"
2023-02-10T01:13:45.589209Z	info	Istiod certificates are reloaded
2023-02-10T01:13:45.589270Z	info	spiffe	Added 1 certs to trust domain example.org in peer cert verifier
2023-02-10T01:13:45.589306Z	info	initializing secure discovery service
2023-02-10T01:13:45.589368Z	info	initializing secure webhook server for istiod webhooks
2023-02-10T01:13:45.631536Z	info	initializing sidecar injector
2023-02-10T01:13:45.666017Z	info	initializing config validator
2023-02-10T01:13:45.666057Z	info	initializing registry event handlers
2023-02-10T01:13:45.666117Z	info	starting discovery service
2023-02-10T01:13:45.670967Z	info	Starting Istiod Server with primary cluster Kubernetes
2023-02-10T01:13:45.671052Z	info	ControlZ available at 127.0.0.1:9876
2023-02-10T01:13:45.671240Z	info	kube	Initializing Kubernetes service registry "Kubernetes"
2023-02-10T01:13:45.671409Z	info	kube	should join leader-election for cluster Kubernetes: false
2023-02-10T01:13:45.671425Z	info	initializing Kubernetes credential reader for cluster Kubernetes
2023-02-10T01:13:45.672137Z	info	status	Starting status manager
2023-02-10T01:13:45.672143Z	info	Starting multicluster remote secrets controller
2023-02-10T01:13:45.672220Z	info	Starting validation controller
2023-02-10T01:13:45.672250Z	info	Starting ADS server
2023-02-10T01:13:45.672265Z	info	Starting IstioD CA
2023-02-10T01:13:45.672271Z	info	JWT policy is third-party-jwt
2023-02-10T01:13:45.672323Z	info	klog	attempting to acquire leader lease istio-system/istio-leader...
2023-02-10T01:13:45.672449Z	info	Istiod CA has started
2023-02-10T01:13:45.672487Z	info	klog	attempting to acquire leader lease istio-system/istio-gateway-status-leader...
2023-02-10T01:13:45.672574Z	info	controllers	starting	controller=ingress
2023-02-10T01:13:45.672635Z	info	klog	attempting to acquire leader lease istio-system/istio-gateway-deployment-leader...
2023-02-10T01:13:45.672676Z	info	controllers	starting	controller=webhook patcher
2023-02-10T01:13:45.673511Z	info	controllers	starting	controller=healthcheck
2023-02-10T01:13:45.673867Z	info	controllers	starting	controller=unregister_workloadentry
2023-02-10T01:13:45.674306Z	info	kube	Starting Pilot K8S CRD controller	controller=crd-controller
2023-02-10T01:13:45.674351Z	info	kube	controller "networking.istio.io/v1alpha3/DestinationRule" is syncing...	controller=crd-controller
2023-02-10T01:13:45.676802Z	info	kube	controller "networking.istio.io/v1alpha3/VirtualService" is syncing...	controller=crd-controller
2023-02-10T01:13:45.680880Z	info	kube	controller "extensions.istio.io/v1alpha1/WasmPlugin" is syncing...	controller=crd-controller
2023-02-10T01:13:45.690203Z	info	kube	controller "networking.istio.io/v1alpha3/EnvoyFilter" is syncing...	controller=crd-controller
2023-02-10T01:13:45.690232Z	info	controllers	starting	controller=configmap istio-sidecar-injector
2023-02-10T01:13:45.690241Z	info	controllers	starting	controller=default revision
2023-02-10T01:13:45.690282Z	info	controllers	starting	controller=default revision
2023-02-10T01:13:45.690320Z	info	controllers	starting	controller=default revision
2023-02-10T01:13:45.706312Z	info	kube	controller "networking.istio.io/v1alpha3/VirtualService" is syncing...	controller=crd-controller
2023-02-10T01:13:45.738565Z	info	kube	controller "security.istio.io/v1beta1/AuthorizationPolicy" is syncing...	controller=crd-controller
2023-02-10T01:13:45.739870Z	info	multicluster remote secrets controller cache synced in 67.73267ms
2023-02-10T01:13:45.739907Z	info	controllers	starting	controller=multicluster secret
2023-02-10T01:13:45.802948Z	info	kube	Pilot K8S CRD controller synced in 128.640809ms	controller=crd-controller
2023-02-10T01:13:45.842261Z	info	klog	successfully acquired lease istio-system/istio-gateway-status-leader
2023-02-10T01:13:45.842383Z	info	leader election lock obtained: istio-gateway-status-leader
2023-02-10T01:13:45.842456Z	info	Starting gateway status writer
2023-02-10T01:13:45.844165Z	info	klog	successfully acquired lease istio-system/istio-gateway-deployment-leader
2023-02-10T01:13:45.844306Z	info	klog	successfully acquired lease istio-system/istio-leader
2023-02-10T01:13:45.844292Z	info	leader election lock obtained: istio-gateway-deployment-leader
2023-02-10T01:13:45.844350Z	info	leader election lock obtained: istio-leader
2023-02-10T01:13:45.888638Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid config not rejected
2023-02-10T01:13:45.944584Z	info	Starting ingress controller
2023-02-10T01:13:45.944660Z	warn	Missing ingress, skip status updates
2023-02-10T01:13:45.961002Z	info	validationController	Successfully updated validatingwebhookconfiguration istio-validator-istio-system (failurePolicy=Ignore,resourceVersion=213754)
2023-02-10T01:13:45.965907Z	info	ads	Push debounce stable[1] 185 for reason config and 184 more reasons: 100.343446ms since last change, 162.930451ms since last push, full=true
2023-02-10T01:13:45.971364Z	info	ads	XDS: Pushing:2023-02-10T01:13:45Z/1 Services:0 ConnectedEndpoints:0 Version:2023-02-10T01:13:45Z/1
2023-02-10T01:13:45.974558Z	info	Waiting for caches to be synced
2023-02-10T01:13:45.974602Z	info	kube	joining leader-election for istio-namespace-controller-election in istio-system on cluster Kubernetes
2023-02-10T01:13:45.974704Z	info	klog	attempting to acquire leader lease istio-system/istio-namespace-controller-election...
2023-02-10T01:13:45.974857Z	info	controllers	starting	controller=default revision
2023-02-10T01:13:46.001253Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid config not rejected
2023-02-10T01:13:46.001285Z	info	validationController	validatingwebhookconfiguration istio-validator-istio-system (failurePolicy=Ignore, resourceVersion=213754) is up-to-date. No change required.
2023-02-10T01:13:46.006004Z	info	ads	Full push, new service kube-state-metrics/kube-state-metrics.kube-state-metrics.svc.cluster.local
2023-02-10T01:13:46.006447Z	info	ads	Full push, new service postgres/postgresql.postgres.svc.cluster.local
2023-02-10T01:13:46.006499Z	info	ads	Incremental push, service istiod.istio-system.svc.cluster.local at shard Kubernetes/Kubernetes has no endpoints
2023-02-10T01:13:46.006536Z	info	ads	Full push, new service kube-system/default-http-backend.kube-system.svc.cluster.local
2023-02-10T01:13:46.006876Z	info	ads	Full push, new service default/kubernetes.default.svc.cluster.local
2023-02-10T01:13:46.007151Z	info	ads	Full push, new service redis/redis-headless.redis.svc.cluster.local
2023-02-10T01:13:46.007290Z	info	ads	Full push, new service logging/kibana-kibana.logging.svc.cluster.local
2023-02-10T01:13:46.007590Z	info	ads	Full push, new service spire/spire-server.spire.svc.cluster.local
2023-02-10T01:13:46.007622Z	info	ads	Full push, new service self-service/ratelimit.self-service.svc.cluster.local
2023-02-10T01:13:46.007652Z	info	ads	Full push, new service kube-system/metrics-server.kube-system.svc.cluster.local
2023-02-10T01:13:46.007719Z	info	ads	Full push, new service kube-system/kube-dns.kube-system.svc.cluster.local
2023-02-10T01:13:46.007999Z	info	ads	Full push, new service external-dns/external-dns.external-dns.svc.cluster.local
2023-02-10T01:13:46.008056Z	info	ads	Full push, new service logging/elasticsearch-master.logging.svc.cluster.local
2023-02-10T01:13:46.008000Z	info	klog	successfully acquired lease istio-system/istio-namespace-controller-election
2023-02-10T01:13:46.008194Z	info	leader election lock obtained: istio-namespace-controller-election
2023-02-10T01:13:46.008325Z	info	kube	starting namespace controller for cluster Kubernetes
2023-02-10T01:13:46.010967Z	info	ads	Incremental push, service istiod.istio-system.svc.cluster.local at shard Kubernetes/Kubernetes has no endpoints
2023-02-10T01:13:46.038792Z	info	All controller caches have been synced up in 64.228312ms
2023-02-10T01:13:46.109502Z	info	controllers	starting	controller=namespace controller
2023-02-10T01:13:46.113739Z	info	ads	Push debounce stable[2] 324 for config ServiceEntry/kube-system/kube-dns.kube-system.svc.cluster.local and 74 more configs: 100.03394ms since last change, 110.202232ms since last push, full=true
2023-02-10T01:13:46.117086Z	info	ads	XDS: Pushing:2023-02-10T01:13:46Z/2 Services:75 ConnectedEndpoints:0 Version:2023-02-10T01:13:46Z/2
2023-02-10T01:13:46.167608Z	info	ads	All caches have been synced up in 1.29121259s, marking server ready
2023-02-10T01:13:46.167836Z	info	starting gRPC discovery service at [::]:15010
2023-02-10T01:13:46.167980Z	info	starting webhook service at [::]:15017
2023-02-10T01:13:46.168009Z	info	starting secure gRPC discovery service at [::]:15012
2023-02-10T01:13:46.630846Z	info	ads	Full push, new service istio-system/istiod.istio-system.svc.cluster.local
2023-02-10T01:13:46.731167Z	info	ads	Push debounce stable[3] 1 for config ServiceEntry/istio-system/istiod.istio-system.svc.cluster.local: 100.24904ms since last change, 100.248891ms since last push, full=true
2023-02-10T01:13:46.733471Z	info	ads	XDS: Pushing:2023-02-10T01:13:46Z/3 Services:75 ConnectedEndpoints:0 Version:2023-02-10T01:13:46Z/3
2023-02-10T01:13:46.758630Z	info	validationServer	configuration is invalid: gateway must have at least one server
2023-02-10T01:13:46.769248Z	info	validationController	Endpoint successfully rejected invalid config. Switching to fail-close.
2023-02-10T01:13:46.779365Z	info	validationController	Successfully updated validatingwebhookconfiguration istio-validator-istio-system (failurePolicy=Fail,resourceVersion=213808)
2023-02-10T01:13:46.789139Z	info	validationController	validatingwebhookconfiguration istio-validator-istio-system (failurePolicy=Fail, resourceVersion=213808) is up-to-date. No change required.
2023-02-10T01:13:46.965696Z	info	validationController	validatingwebhookconfiguration istio-validator-istio-system (failurePolicy=Fail, resourceVersion=213808) is up-to-date. No change required.
2023-02-10T01:13:49.805626Z	info	ads	Incremental push, service istio-ingressgateway.istio-system.svc.cluster.local at shard Kubernetes/Kubernetes has no endpoints
2023-02-10T01:13:49.908377Z	info	ads	Push debounce stable[4] 3 for config ServiceEntry/istio-system/istio-ingressgateway.istio-system.svc.cluster.local: 100.686419ms since last change, 139.246895ms since last push, full=true
2023-02-10T01:13:49.911693Z	info	ads	XDS: Pushing:2023-02-10T01:13:49Z/4 Services:76 ConnectedEndpoints:0 Version:2023-02-10T01:13:49Z/4
2023-02-10T01:13:52.295682Z	info	ads	ADS: new connection for node:istio-ingressgateway-7b4455498-l5bdz.istio-system-1
2023-02-10T01:13:52.302201Z	info	ads	CDS: PUSH request for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:171 size:171.2kB cached:0/170
2023-02-10T01:13:52.388120Z	info	ads	EDS: PUSH request for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:170 size:37.7kB empty:12 cached:0/170
2023-02-10T01:13:52.648547Z	info	ads	Incremental push, service istio-ingressgateway.istio-system.svc.cluster.local at shard Kubernetes/Kubernetes has no endpoints
2023-02-10T01:13:52.748903Z	info	ads	Push debounce stable[5] 1 for config ServiceEntry/istio-system/istio-ingressgateway.istio-system.svc.cluster.local: 100.240353ms since last change, 100.240152ms since last push, full=false
2023-02-10T01:13:52.748981Z	info	ads	XDS: Incremental Pushing:2023-02-10T01:13:49Z/4 ConnectedEndpoints:1 Version:2023-02-10T01:13:49Z/4
2023-02-10T01:13:52.963108Z	info	ads	LDS: PUSH request for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:0 size:0B
2023-02-10T01:13:53.328823Z	info	ads	CDS: PUSH for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:171 size:171.2kB cached:0/170
2023-02-10T01:13:53.333809Z	info	ads	EDS: PUSH for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:170 size:37.7kB empty:9 cached:3/170
2023-02-10T01:13:53.333887Z	info	ads	LDS: PUSH for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:0 size:0B
2023-02-10T01:13:53.355767Z	info	ads	Full push, new service istio-system/istio-ingressgateway.istio-system.svc.cluster.local
2023-02-10T01:13:53.455926Z	info	ads	Push debounce stable[6] 1 for config ServiceEntry/istio-system/istio-ingressgateway.istio-system.svc.cluster.local: 100.088047ms since last change, 100.087947ms since last push, full=true
2023-02-10T01:13:53.458608Z	info	ads	XDS: Pushing:2023-02-10T01:13:53Z/5 Services:76 ConnectedEndpoints:1 Version:2023-02-10T01:13:53Z/5
2023-02-10T01:13:53.460229Z	info	ads	CDS: PUSH for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:171 size:171.4kB cached:167/170
2023-02-10T01:13:53.460715Z	info	ads	EDS: PUSH INC for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:3 size:715B empty:0 cached:0/3
2023-02-10T01:13:53.460814Z	info	ads	LDS: PUSH for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:0 size:0B
2023-02-10T01:14:00.447283Z	info	validationController	Successfully updated validatingwebhookconfiguration istiod-default-validator (failurePolicy=Fail,resourceVersion=213994)
2023-02-10T01:14:00.452018Z	info	validationController	validatingwebhookconfiguration istiod-default-validator (failurePolicy=Fail, resourceVersion=213994) is up-to-date. No change required.
2023-02-10T01:14:25.374532Z	info	ads	Push debounce stable[7] 1 for config ServiceEntry/istio-system/istio-ingressgateway.istio-system.svc.cluster.local: 100.891166ms since last change, 100.890967ms since last push, full=true
2023-02-10T01:14:25.378776Z	info	ads	XDS: Pushing:2023-02-10T01:14:25Z/6 Services:76 ConnectedEndpoints:1 Version:2023-02-10T01:14:25Z/6
2023-02-10T01:14:25.380471Z	info	ads	CDS: PUSH for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:171 size:171.4kB cached:167/170
2023-02-10T01:14:25.381045Z	info	ads	EDS: PUSH INC for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:3 size:715B empty:0 cached:0/3
2023-02-10T01:14:25.381308Z	info	ads	LDS: PUSH for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:0 size:0B
2023-02-10T01:41:24.554432Z	info	ads	ADS: "10.111.197.21:59302" istio-ingressgateway-7b4455498-l5bdz.istio-system-1 terminated
2023-02-10T01:41:24.755187Z	info	ads	ADS: new connection for node:istio-ingressgateway-7b4455498-l5bdz.istio-system-2
2023-02-10T01:41:24.756329Z	info	ads	CDS: PUSH request for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:171 size:171.4kB cached:170/170
2023-02-10T01:41:24.757317Z	info	ads	EDS: PUSH request for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:170 size:38.2kB empty:0 cached:170/170
2023-02-10T01:42:26.068227Z	info	ads	CDS: PUSH for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:171 size:171.6kB cached:167/170
2023-02-10T01:42:26.068664Z	info	ads	EDS: PUSH INC for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:3 size:615B empty:0 cached:0/3
2023-02-10T01:42:26.068961Z	info	ads	LDS: PUSH for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:0 size:0B
2023-02-10T01:45:14.276121Z	info	rootcertrotator	Jitter complete, start rotator.
2023-02-10T02:10:38.227658Z	info	ads	ADS: "10.111.197.21:49192" istio-ingressgateway-7b4455498-l5bdz.istio-system-2 terminated
2023-02-10T02:10:38.300905Z	info	ads	ADS: new connection for node:istio-ingressgateway-7b4455498-l5bdz.istio-system-3
2023-02-10T02:10:38.302184Z	info	ads	CDS: PUSH request for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:171 size:171.6kB cached:167/170
2023-02-10T02:10:38.303204Z	info	ads	EDS: PUSH request for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:170 size:38.2kB empty:0 cached:170/170
2023-02-10T02:10:38.303449Z	info	ads	LDS: PUSH request for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:0 size:0B
2023-02-10T02:39:19.886389Z	info	ads	ADS: "10.111.197.21:59378" istio-ingressgateway-7b4455498-l5bdz.istio-system-3 terminated
2023-02-10T02:39:20.329826Z	info	ads	ADS: new connection for node:istio-ingressgateway-7b4455498-l5bdz.istio-system-4
2023-02-10T02:39:20.331526Z	info	ads	CDS: PUSH request for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:171 size:171.6kB cached:167/170
2023-02-10T02:39:20.333134Z	info	ads	EDS: PUSH request for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:170 size:38.2kB empty:0 cached:170/170
2023-02-10T02:39:20.333216Z	info	ads	LDS: PUSH request for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:0 size:0B
2023-02-10T02:45:14.276806Z	info	rootcertrotator	Check and rotate root cert.
2023-02-10T02:45:14.281387Z	info	rootcertrotator	Root cert is not about to expire, skipping root cert rotation.
2023-02-10T03:09:48.550873Z	info	ads	ADS: "10.111.197.21:33432" istio-ingressgateway-7b4455498-l5bdz.istio-system-4 terminated
2023-02-10T03:09:49.066375Z	info	ads	ADS: new connection for node:istio-ingressgateway-7b4455498-l5bdz.istio-system-5
2023-02-10T03:09:49.067993Z	info	ads	CDS: PUSH request for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:171 size:171.6kB cached:167/170
2023-02-10T03:09:49.069046Z	info	ads	EDS: PUSH request for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:170 size:38.2kB empty:0 cached:170/170
2023-02-10T03:09:49.069270Z	info	ads	LDS: PUSH request for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:0 size:0B
2023-02-10T03:41:02.222092Z	info	ads	ADS: "10.111.197.21:50592" istio-ingressgateway-7b4455498-l5bdz.istio-system-5 terminated
2023-02-10T03:41:02.730278Z	info	ads	ADS: new connection for node:istio-ingressgateway-7b4455498-l5bdz.istio-system-6
2023-02-10T03:41:02.731550Z	info	ads	CDS: PUSH request for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:171 size:171.6kB cached:167/170
2023-02-10T03:41:02.732596Z	info	ads	EDS: PUSH request for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:170 size:38.2kB empty:0 cached:170/170
2023-02-10T03:41:02.732671Z	info	ads	LDS: PUSH request for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:0 size:0B
2023-02-10T03:45:14.277281Z	info	rootcertrotator	Check and rotate root cert.
2023-02-10T03:45:14.283629Z	info	rootcertrotator	Root cert is not about to expire, skipping root cert rotation.
2023-02-10T04:08:28.029040Z	info	ads	ADS: "10.111.197.21:40774" istio-ingressgateway-7b4455498-l5bdz.istio-system-6 terminated
2023-02-10T04:08:28.368860Z	info	ads	ADS: new connection for node:istio-ingressgateway-7b4455498-l5bdz.istio-system-7
2023-02-10T04:08:28.370169Z	info	ads	CDS: PUSH request for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:171 size:171.6kB cached:167/170
2023-02-10T04:08:28.370997Z	info	ads	EDS: PUSH request for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:170 size:38.2kB empty:0 cached:170/170
2023-02-10T04:08:28.371070Z	info	ads	LDS: PUSH request for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:0 size:0B
2023-02-10T04:38:35.437338Z	info	ads	ADS: "10.111.197.21:38184" istio-ingressgateway-7b4455498-l5bdz.istio-system-7 terminated
2023-02-10T04:38:35.894124Z	info	ads	ADS: new connection for node:istio-ingressgateway-7b4455498-l5bdz.istio-system-8
2023-02-10T04:38:35.895318Z	info	ads	CDS: PUSH request for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:171 size:171.6kB cached:167/170
2023-02-10T04:38:35.896257Z	info	ads	EDS: PUSH request for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:170 size:38.2kB empty:0 cached:170/170
2023-02-10T04:38:35.896481Z	info	ads	LDS: PUSH request for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:0 size:0B
2023-02-10T04:45:14.276991Z	info	rootcertrotator	Check and rotate root cert.
2023-02-10T04:45:14.280989Z	info	rootcertrotator	Root cert is not about to expire, skipping root cert rotation.
2023-02-10T04:48:46.298274Z	info	ads	Push debounce stable[22] 1 for config Secret/druid/druid-postgresql: 100.276283ms since last change, 100.275936ms since last push, full=false
2023-02-10T04:48:46.298384Z	info	ads	XDS: Incremental Pushing:2023-02-10T01:42:26Z/7 ConnectedEndpoints:1 Version:2023-02-10T01:42:26Z/7
2023-02-10T04:48:46.767360Z	info	ads	Incremental push, service druid-coordinator.druid.svc.cluster.local at shard Kubernetes/Kubernetes has no endpoints
2023-02-10T04:48:46.776397Z	info	ads	Incremental push, service druid-router.druid.svc.cluster.local at shard Kubernetes/Kubernetes has no endpoints
2023-02-10T04:48:46.870015Z	info	ads	Incremental push, service druid-postgresql.druid.svc.cluster.local at shard Kubernetes/Kubernetes has no endpoints
2023-02-10T04:48:46.907016Z	info	ads	Incremental push, service druid-historical.druid.svc.cluster.local at shard Kubernetes/Kubernetes has no endpoints
2023-02-10T04:48:46.929987Z	info	ads	Incremental push, service druid-zookeeper-headless.druid.svc.cluster.local at shard Kubernetes/Kubernetes has no endpoints
2023-02-10T04:48:46.962188Z	info	ads	Incremental push, service druid-middle-manager.druid.svc.cluster.local at shard Kubernetes/Kubernetes has no endpoints
2023-02-10T04:48:47.021777Z	info	ads	Incremental push, service druid-broker.druid.svc.cluster.local at shard Kubernetes/Kubernetes has no endpoints
2023-02-10T04:48:47.038851Z	info	ads	Incremental push, service druid-postgresql-headless.druid.svc.cluster.local at shard Kubernetes/Kubernetes has no endpoints
2023-02-10T04:48:47.052132Z	info	ads	Incremental push, service druid-zookeeper.druid.svc.cluster.local at shard Kubernetes/Kubernetes has no endpoints
2023-02-10T04:48:47.153281Z	info	ads	Push debounce stable[23] 20 for config ServiceEntry/druid/druid-postgresql.druid.svc.cluster.local and 8 more configs: 101.08016ms since last change, 427.732182ms since last push, full=true
2023-02-10T04:48:47.155620Z	info	ads	XDS: Pushing:2023-02-10T04:48:47Z/8 Services:85 ConnectedEndpoints:1 Version:2023-02-10T04:48:47Z/8
2023-02-10T04:48:47.157270Z	info	ads	CDS: PUSH for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:182 size:180.3kB cached:167/181
2023-02-10T04:48:47.157912Z	info	ads	LDS: PUSH for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:0 size:0B
2023-02-10T04:48:47.184429Z	info	ads	EDS: PUSH request for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:11 size:656B empty:11 cached:0/11 filtered:170
2023-02-10T04:48:47.263930Z	info	Sidecar injection request for druid/druid-router-68cb4b7bcf-***** (actual name not yet known)
2023-02-10T04:48:47.272688Z	info	Sidecar injection request for druid/druid-coordinator-7f48fc79fc-***** (actual name not yet known)
2023-02-10T04:48:47.272894Z	info	Sidecar injection request for druid/druid-broker-7cfd556bf9-***** (actual name not yet known)
2023-02-10T04:48:47.887267Z	info	Sidecar injection request for druid/druid-zookeeper-0
2023-02-10T04:48:47.887514Z	info	Sidecar injection request for druid/druid-historical-0
2023-02-10T04:48:47.898480Z	info	Sidecar injection request for druid/druid-postgresql-0
2023-02-10T04:48:47.903748Z	info	Sidecar injection request for druid/druid-middle-manager-0
2023-02-10T04:48:49.701542Z	info	ads	Incremental push, service druid-broker.druid.svc.cluster.local at shard Kubernetes/Kubernetes has no endpoints
2023-02-10T04:48:49.801882Z	info	ads	Push debounce stable[24] 1 for config ServiceEntry/druid/druid-broker.druid.svc.cluster.local: 100.262706ms since last change, 100.262527ms since last push, full=false
2023-02-10T04:48:49.801990Z	info	ads	XDS: Incremental Pushing:2023-02-10T04:48:47Z/8 ConnectedEndpoints:1 Version:2023-02-10T04:48:47Z/8
2023-02-10T04:48:50.077436Z	info	ads	Incremental push, service druid-router.druid.svc.cluster.local at shard Kubernetes/Kubernetes has no endpoints
2023-02-10T04:48:50.178575Z	info	ads	Push debounce stable[25] 1 for config ServiceEntry/druid/druid-router.druid.svc.cluster.local: 101.075556ms since last change, 101.075342ms since last push, full=false
2023-02-10T04:48:50.178677Z	info	ads	XDS: Incremental Pushing:2023-02-10T04:48:47Z/8 ConnectedEndpoints:1 Version:2023-02-10T04:48:47Z/8
2023-02-10T04:48:50.529270Z	info	ads	Incremental push, service druid-coordinator.druid.svc.cluster.local at shard Kubernetes/Kubernetes has no endpoints
2023-02-10T04:48:50.629611Z	info	ads	Push debounce stable[26] 1 for config ServiceEntry/druid/druid-coordinator.druid.svc.cluster.local: 100.2544ms since last change, 100.254226ms since last push, full=false
2023-02-10T04:48:50.629702Z	info	ads	XDS: Incremental Pushing:2023-02-10T04:48:47Z/8 ConnectedEndpoints:1 Version:2023-02-10T04:48:47Z/8
2023-02-10T04:49:05.090401Z	info	ads	ADS: new connection for node:druid-broker-7cfd556bf9-nrjsw.druid-9
2023-02-10T04:49:05.099798Z	info	ads	CDS: PUSH request for node:druid-broker-7cfd556bf9-nrjsw.druid resources:185 size:192.4kB cached:0/181
2023-02-10T04:49:05.259024Z	info	ads	EDS: PUSH request for node:druid-broker-7cfd556bf9-nrjsw.druid resources:154 size:31.4kB empty:0 cached:154/154
2023-02-10T04:49:05.750859Z	info	ads	ADS: new connection for node:druid-router-68cb4b7bcf-7lnzf.druid-10
2023-02-10T04:49:05.759996Z	info	ads	CDS: PUSH request for node:druid-router-68cb4b7bcf-7lnzf.druid resources:185 size:192.4kB cached:0/181
2023-02-10T04:49:05.942405Z	info	ads	LDS: PUSH request for node:druid-broker-7cfd556bf9-nrjsw.druid resources:153 size:779.4kB
2023-02-10T04:49:06.016004Z	info	ads	EDS: PUSH request for node:druid-router-68cb4b7bcf-7lnzf.druid resources:154 size:31.4kB empty:0 cached:154/154
2023-02-10T04:49:06.293440Z	info	ads	LDS: PUSH request for node:druid-router-68cb4b7bcf-7lnzf.druid resources:153 size:779.4kB
2023-02-10T04:49:06.364605Z	info	ads	ADS: new connection for node:druid-coordinator-7f48fc79fc-dr9lg.druid-11
2023-02-10T04:49:06.378169Z	info	ads	CDS: PUSH request for node:druid-coordinator-7f48fc79fc-dr9lg.druid resources:185 size:192.4kB cached:0/181
2023-02-10T04:49:06.550876Z	info	ads	EDS: PUSH request for node:druid-coordinator-7f48fc79fc-dr9lg.druid resources:154 size:31.4kB empty:0 cached:154/154
2023-02-10T04:49:06.594489Z	info	ads	RDS: PUSH request for node:druid-broker-7cfd556bf9-nrjsw.druid resources:114 size:100.6kB cached:0/114
2023-02-10T04:49:06.890246Z	info	ads	RDS: PUSH request for node:druid-router-68cb4b7bcf-7lnzf.druid resources:114 size:100.6kB cached:0/114
2023-02-10T04:49:07.002901Z	info	ads	LDS: PUSH request for node:druid-coordinator-7f48fc79fc-dr9lg.druid resources:153 size:779.4kB
2023-02-10T04:49:07.646207Z	info	ads	RDS: PUSH request for node:druid-coordinator-7f48fc79fc-dr9lg.druid resources:114 size:100.6kB cached:0/114
2023-02-10T04:50:08.277523Z	info	ads	CDS: PUSH for node:druid-router-68cb4b7bcf-7lnzf.druid resources:185 size:192.4kB cached:0/181
2023-02-10T04:50:08.278504Z	info	ads	EDS: PUSH for node:druid-router-68cb4b7bcf-7lnzf.druid resources:154 size:31.4kB empty:0 cached:154/154
2023-02-10T04:50:08.297423Z	info	ads	Full push, new service druid/druid-router.druid.svc.cluster.local
2023-02-10T04:50:08.302304Z	info	ads	LDS: PUSH for node:druid-router-68cb4b7bcf-7lnzf.druid resources:153 size:779.4kB
2023-02-10T04:50:08.331829Z	info	ads	RDS: PUSH for node:druid-router-68cb4b7bcf-7lnzf.druid resources:114 size:100.6kB cached:0/114
2023-02-10T04:50:08.397674Z	info	ads	Push debounce stable[27] 1 for config ServiceEntry/druid/druid-router.druid.svc.cluster.local: 100.175876ms since last change, 100.175624ms since last push, full=true
2023-02-10T04:50:08.401528Z	info	ads	XDS: Pushing:2023-02-10T04:50:08Z/9 Services:85 ConnectedEndpoints:4 Version:2023-02-10T04:50:08Z/9
2023-02-10T04:50:08.407161Z	info	ads	CDS: PUSH for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:182 size:180.4kB cached:178/181
2023-02-10T04:50:08.408105Z	info	ads	EDS: PUSH INC for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:1 size:248B empty:0 cached:0/1
2023-02-10T04:50:08.408171Z	info	ads	LDS: PUSH for node:istio-ingressgateway-7b4455498-l5bdz.istio-system resources:0 size:0B
2023-02-10T04:50:08.409132Z	info	ads	CDS: PUSH for node:druid-coordinator-7f48fc79fc-dr9lg.druid resources:185 size:192.4kB cached:180/181
2023-02-10T04:50:08.409968Z	info	ads	EDS: PUSH INC for node:druid-coordinator-7f48fc79fc-dr9lg.druid resources:1 size:248B empty:0 cached:1/1
2023-02-10T04:50:08.410538Z	info	ads	CDS: PUSH for node:druid-router-68cb4b7bcf-7lnzf.druid resources:185 size:192.4kB cached:181/181
2023-02-10T04:50:08.411170Z	info	ads	EDS: PUSH INC for node:druid-router-68cb4b7bcf-7lnzf.druid resources:1 size:248B empty:0 cached:1/1
2023-02-10T04:50:08.413375Z	info	ads	CDS: PUSH for node:druid-broker-7cfd556bf9-nrjsw.druid resources:185 size:192.4kB cached:181/181
2023-02-10T04:50:08.414142Z	info	ads	EDS: PUSH INC for node:druid-broker-7cfd556bf9-nrjsw.druid resources:1 size:248B empty:0 cached:1/1
2023-02-10T04:50:08.450547Z	info	ads	LDS: PUSH for node:druid-router-68cb4b7bcf-7lnzf.druid resources:153 size:779.4kB
2023-02-10T04:50:08.464248Z	info	ads	LDS: PUSH for node:druid-coordinator-7f48fc79fc-dr9lg.druid resources:153 size:779.4kB
2023-02-10T04:50:08.464251Z	info	ads	RDS: PUSH for node:druid-router-68cb4b7bcf-7lnzf.druid resources:114 size:100.6kB cached:0/114
2023-02-10T04:50:08.470928Z	info	ads	LDS: PUSH for node:druid-broker-7cfd556bf9-nrjsw.druid resources:153 size:779.4kB
2023-02-10T04:50:08.475263Z	info	ads	RDS: PUSH for node:druid-broker-7cfd556bf9-nrjsw.druid resources:114 size:100.6kB cached:39/114
2023-02-10T04:50:08.476559Z	info	ads	RDS: PUSH for node:druid-coordinator-7f48fc79fc-dr9lg.druid resources:114 size:100.6kB cached:47/114

@howardjohn
Copy link
Member

Logs of the pod not starting istio-proxy container, not istiod

@RunlinXiao
Copy link
Author

Actually all pod started the istio-proxy container successfully, but it will show the readiness probe to the port 15021 failed.
kubectl describe pod/druid-broker-7cfd556bf9-zv67r.

Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  11m                  default-scheduler  Successfully assigned druid/druid-broker-7cfd556bf9-zv67r to gke-k8saas-k8s-bvzvb-default-pool--v0-e23daf9d-dx76
  Normal   Pulled     11m                  kubelet            Container image "docker.io/istio/proxyv2:1.16.1" already present on machine
  Normal   Created    11m                  kubelet            Created container istio-init
  Normal   Started    11m                  kubelet            Started container istio-init
  Normal   Started    11m                  kubelet            Started container istio-proxy
  Normal   Pulled     11m                  kubelet            Container image "docker.io/istio/proxyv2:1.16.1" already present on machine
  Normal   Created    11m                  kubelet            Created container istio-proxy
  Warning  Unhealthy  11m (x2 over 11m)    kubelet            Readiness probe failed: Get "http://10.111.197.100:15021/healthz/ready": dial tcp 10.111.197.100:15021: connect: connection refused
  Normal   Created    9m44s (x2 over 11m)  kubelet            Created container druid
  Normal   Started    9m44s (x2 over 11m)  kubelet            Started container druid
  Normal   Pulled     9m44s (x2 over 11m)  kubelet            Container image "apache/druid:24.0.0" already present on machine
  Normal   Killing    9m44s                kubelet            Container druid failed liveness probe, will be restarted
  Warning  Unhealthy  8m44s (x6 over 10m)  kubelet            Readiness probe failed: HTTP probe failed with statuscode: 500
  Warning  Unhealthy  6m4s (x10 over 10m)  kubelet            Liveness probe failed: HTTP probe failed with statuscode: 500
  Warning  BackOff    62s (x5 over 104s)   kubelet            Back-off restarting failed container

And if use kb logs -f pod/druid-broker-7cfd556bf9-zv67r, there is no message about istio-proxy:

Setting druid.host=10.111.197.100 in /tmp/conf/druid/cluster/query/broker/runtime.properties
Setting druid.indexer.logs.type=file in /tmp/conf/druid/cluster/query/broker/runtime.properties
Setting druid.extensions.loadList=["druid-histogram", "druid-datasketches", "druid-lookups-cached-global", "postgresql-metadata-storage"] in /tmp/conf/druid/cluster/query/broker/runtime.properties
Setting druid.processing.numMergeBuffers=2 in /tmp/conf/druid/cluster/query/broker/runtime.properties
Setting druid.emitter.http.recipientBaseUrl=http://druid_exporter_url:druid_exporter_port/druid in /tmp/conf/druid/cluster/query/broker/runtime.properties
Setting druid.metadata.storage.type=postgresql in /tmp/conf/druid/cluster/query/broker/runtime.properties
Setting druid.processing.numThreads=1 in /tmp/conf/druid/cluster/query/broker/runtime.properties
Setting druid.metadata.storage.connector.connectURI=jdbc:postgresql://druid-postgresql:5432/druid in /tmp/conf/druid/cluster/query/broker/runtime.properties
Setting druid.storage.type=local in /tmp/conf/druid/cluster/query/broker/runtime.properties
Setting druid.metadata.storage.connector.user=druid in /tmp/conf/druid/cluster/query/broker/runtime.properties
Setting druid.indexer.logs.directory=/opt/data/indexing-logs in /tmp/conf/druid/cluster/query/broker/runtime.properties
Setting druid.zk.service.host=druid-zookeeper-headless:2181 in /tmp/conf/druid/cluster/query/broker/runtime.properties
Setting druid.emitter=noop in /tmp/conf/druid/cluster/query/broker/runtime.properties
Setting druid.processing.buffer.sizeBytes=50000000 in /tmp/conf/druid/cluster/query/broker/runtime.properties
Setting druid.metadata.storage.connector.password=druid in /tmp/conf/druid/cluster/query/broker/runtime.properties
Setting druid.emitter.logging.logLevel=debug in /tmp/conf/druid/cluster/query/broker/runtime.properties

I'm using the same druid helm chart for deployment without istio+spire and it works well, but when added the istio+spire, it start to have problem...

@howardjohn
Copy link
Member

-c istio-proxy on the logs command

@RunlinXiao
Copy link
Author 

❯ kb logs -f pod/druid-broker-7cfd556bf9-zv67r -n druid -c istio-proxy
W0210 08:56:46.090417   66126 gcp.go:119] WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.26+; use gcloud instead.
To learn more, consult https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke
2023-02-10T15:36:25.019843Z	info	FLAG: --concurrency="2"
2023-02-10T15:36:25.019867Z	info	FLAG: --domain="druid.svc.cluster.local"
2023-02-10T15:36:25.019875Z	info	FLAG: --help="false"
2023-02-10T15:36:25.019881Z	info	FLAG: --log_as_json="false"
2023-02-10T15:36:25.019885Z	info	FLAG: --log_caller=""
2023-02-10T15:36:25.019890Z	info	FLAG: --log_output_level="default:info"
2023-02-10T15:36:25.019894Z	info	FLAG: --log_rotate=""
2023-02-10T15:36:25.019901Z	info	FLAG: --log_rotate_max_age="30"
2023-02-10T15:36:25.019906Z	info	FLAG: --log_rotate_max_backups="1000"
2023-02-10T15:36:25.019911Z	info	FLAG: --log_rotate_max_size="104857600"
2023-02-10T15:36:25.019916Z	info	FLAG: --log_stacktrace_level="default:none"
2023-02-10T15:36:25.019925Z	info	FLAG: --log_target="[stdout]"
2023-02-10T15:36:25.019931Z	info	FLAG: --meshConfig="./etc/istio/config/mesh"
2023-02-10T15:36:25.019935Z	info	FLAG: --outlierLogPath=""
2023-02-10T15:36:25.019940Z	info	FLAG: --proxyComponentLogLevel="misc:error"
2023-02-10T15:36:25.019946Z	info	FLAG: --proxyLogLevel="warning"
2023-02-10T15:36:25.019951Z	info	FLAG: --serviceCluster="istio-proxy"
2023-02-10T15:36:25.019956Z	info	FLAG: --stsPort="0"
2023-02-10T15:36:25.019961Z	info	FLAG: --templateFile=""
2023-02-10T15:36:25.019966Z	info	FLAG: --tokenManagerPlugin="GoogleTokenExchange"
2023-02-10T15:36:25.019972Z	info	FLAG: --vklog="0"
2023-02-10T15:36:25.019977Z	info	Version 1.16.1-f6d7bf648e571a6a523210d97bde8b489250354b-Clean
2023-02-10T15:36:25.024918Z	info	Maximum file descriptors (ulimit -n): 1048576
2023-02-10T15:36:25.025176Z	info	Proxy role	ips=[10.111.197.100] type=sidecar id=druid-broker-7cfd556bf9-zv67r.druid domain=druid.svc.cluster.local
2023-02-10T15:36:25.025268Z	info	Apply proxy config from env {}

2023-02-10T15:36:25.030624Z	info	Effective config: binaryPath: /usr/local/bin/envoy
concurrency: 2
configPath: ./etc/istio/proxy
controlPlaneAuthPolicy: MUTUAL_TLS
discoveryAddress: istiod.istio-system.svc:15012
drainDuration: 45s
parentShutdownDuration: 60s
proxyAdminPort: 15000
serviceCluster: istio-proxy
statNameLength: 189
statusPort: 15020
terminationDrainDuration: 5s
tracing:
  zipkin:
    address: zipkin.istio-system:9411

2023-02-10T15:36:25.030649Z	info	JWT policy is third-party-jwt
2023-02-10T15:36:25.030655Z	info	using credential fetcher of JWT type in example.org trust domain
2023-02-10T15:36:25.038401Z	info	platform detected is GCP
2023-02-10T15:36:25.040988Z	info	Workload SDS socket not found. Starting Istio SDS Server
2023-02-10T15:36:25.041020Z	info	CA Endpoint istiod.istio-system.svc:15012, provider Citadel
2023-02-10T15:36:25.041049Z	info	Using CA istiod.istio-system.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem
2023-02-10T15:36:25.041159Z	info	citadelclient	Citadel client using custom root cert: var/run/secrets/istio/root-cert.pem
2023-02-10T15:36:25.042183Z	info	Opening status port 15020
2023-02-10T15:36:25.112684Z	info	ads	All caches have been synced up in 96.484303ms, marking server ready
2023-02-10T15:36:25.113604Z	info	xdsproxy	Initializing with upstream address "istiod.istio-system.svc:15012" and cluster "Kubernetes"
2023-02-10T15:36:25.115146Z	info	starting Http service at 127.0.0.1:15004
2023-02-10T15:36:25.115320Z	info	sds	Starting SDS grpc server
2023-02-10T15:36:25.150371Z	warn	Error fetching GCP Metadata property gcp_gce_instance_created_by: metadata: GCE metadata "instance/attributes/created-by" not defined
2023-02-10T15:36:25.154144Z	warn	Error fetching GCP Metadata property gcp_gce_instance_template: metadata: GCE metadata "instance/attributes/instance-template" not defined
2023-02-10T15:36:25.160195Z	warn	Error fetching GCP Metadata property gcp_gce_instance: metadata: GCE metadata "instance/name" not defined
2023-02-10T15:36:25.183248Z	info	Pilot SAN: [istiod.istio-system.svc]
2023-02-10T15:36:25.205280Z	info	Starting proxy agent
2023-02-10T15:36:25.205371Z	info	starting
2023-02-10T15:36:25.205422Z	info	Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --parent-shutdown-time-s 60 --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --log-format %Y-%m-%dT%T.%fZ	%l	envoy %n	%v -l warning --component-log-level misc:error --concurrency 2]
2023-02-10T15:36:25.476541Z	info	xdsproxy	connected to upstream XDS server: istiod.istio-system.svc:15012
2023-02-10T15:36:25.890804Z	info	ads	ADS: new connection for node:druid-broker-7cfd556bf9-zv67r.druid-1
2023-02-10T15:36:25.891395Z	info	ads	ADS: new connection for node:druid-broker-7cfd556bf9-zv67r.druid-2
2023-02-10T15:36:25.909693Z	info	cache	generated new workload certificate	latency=795.168079ms ttl=23h59m59.090322714s
2023-02-10T15:36:25.909893Z	info	cache	Root cert has changed, start rotating root cert
2023-02-10T15:36:25.910091Z	info	ads	XDS: Incremental Pushing:0 ConnectedEndpoints:2 Version:
2023-02-10T15:36:25.910415Z	info	cache	returned workload trust anchor from cache	ttl=23h59m59.089593s
2023-02-10T15:36:25.910509Z	info	cache	returned workload certificate from cache	ttl=23h59m59.089497825s
2023-02-10T15:36:25.910873Z	info	cache	returned workload trust anchor from cache	ttl=23h59m59.08913607s
2023-02-10T15:36:25.911463Z	info	ads	SDS: PUSH request for node:druid-broker-7cfd556bf9-zv67r.druid resources:1 size:4.0kB resource:default
2023-02-10T15:36:25.912166Z	info	ads	SDS: PUSH request for node:druid-broker-7cfd556bf9-zv67r.druid resources:1 size:1.1kB resource:ROOTCA
2023-02-10T15:36:25.912526Z	info	cache	returned workload trust anchor from cache	ttl=23h59m59.087481434s
2023-02-10T15:36:27.205539Z	info	Readiness succeeded in 2.220675857s
2023-02-10T15:36:27.207266Z	info	Envoy proxy is ready
[2023-02-10T15:36:45.029Z] "- - -" 0 UF,URX - - "delayed_connect_error:_111" 0 0 44 - "-" "-" "-" "-" "10.111.197.103:2181" outbound|2181||druid-zookeeper-headless.druid.svc.cluster.local - 10.111.197.103:2181 10.111.197.100:57202 - -
[2023-02-10T15:36:46.921Z] "- - -" 0 UF,URX - - "delayed_connect_error:_111" 0 0 1 - "-" "-" "-" "-" "10.111.197.103:2181" outbound|2181||druid-zookeeper-headless.druid.svc.cluster.local - 10.111.197.103:2181 10.111.197.100:48152 - -
[2023-02-10T15:36:48.473Z] "- - -" 0 UF,URX - - "delayed_connect_error:_111" 0 0 0 - "-" "-" "-" "-" "10.111.197.103:2181" outbound|2181||druid-zookeeper-headless.druid.svc.cluster.local - 10.111.197.103:2181 10.111.197.100:48174 - -
[2023-02-10T15:36:50.128Z] "- - -" 0 UF,URX - - "delayed_connect_error:_111" 0 0 0 - "-" "-" "-" "-" "10.111.197.103:2181" outbound|2181||druid-zookeeper-headless.druid.svc.cluster.local - 10.111.197.103:2181 10.111.197.100:48184 - -
[2023-02-10T15:36:51.550Z] "- - -" 0 UF,URX - - "delayed_connect_error:_111" 0 0 3 - "-" "-" "-" "-" "10.111.197.103:2181" outbound|2181||druid-zookeeper-headless.druid.svc.cluster.local - 10.111.197.103:2181 10.111.197.100:48208 - -
[2023-02-10T15:36:53.517Z] "- - -" 0 UF,URX - - "delayed_connect_error:_111" 0 0 0 - "-" "-" "-" "-" "10.111.197.103:2181" outbound|2181||druid-zookeeper-headless.druid.svc.cluster.local - 10.111.197.103:2181 10.111.197.100:48216 - -
[2023-02-10T15:37:32.416Z] "- - -" 0 - - - "-" 49 0 3 - "-" "-" "-" "-" "10.111.197.103:2181" outbound|2181||druid-zookeeper-headless.druid.svc.cluster.local 10.111.197.100:39386 10.111.197.103:2181 10.111.197.100:39378 - -
2023-02-10T15:37:33.174477Z	error	Request to probe app failed: Get "http://10.111.197.100:8082/status/health": dial tcp 127.0.0.6:0->10.111.197.100:8082: connect: connection refused, original URL path = /app-health/druid/livez
app URL path = /status/health
2023-02-10T15:37:33.175463Z	error	Request to probe app failed: Get "http://10.111.197.100:8082/status/health": dial tcp 127.0.0.6:0->10.111.197.100:8082: connect: connection refused, original URL path = /app-health/druid/readyz
app URL path = /status/health
[2023-02-10T15:37:34.454Z] "- - -" 0 - - - "-" 49 0 3 - "-" "-" "-" "-" "10.111.197.103:2181" outbound|2181||druid-zookeeper-headless.druid.svc.cluster.local 10.111.197.100:39414 10.111.197.103:2181 10.111.197.100:39402 - -
[2023-02-10T15:37:36.390Z] "- - -" 0 - - - "-" 49 0 3 - "-" "-" "-" "-" "10.111.197.103:2181" outbound|2181||druid-zookeeper-headless.druid.svc.cluster.local 10.111.197.100:49590 10.111.197.103:2181 10.111.197.100:49588 - -
2023-02-10T15:37:37.668057Z	error	Request to probe app failed: Get "http://10.111.197.100:8082/status/health": dial tcp 127.0.0.6:0->10.111.197.100:8082: connect: connection refused, original URL path = /app-health/druid/readyz
app URL path = /status/health
[2023-02-10T15:37:38.301Z] "- - -" 0 - - - "-" 49 0 3 - "-" "-" "-" "-" "10.111.197.103:2181" outbound|2181||druid-zookeeper-headless.druid.svc.cluster.local 10.111.197.100:49610 10.111.197.103:2181 10.111.197.100:49600 - -
[2023-02-10T15:37:40.314Z] "- - -" 0 - - - "-" 49 0 3 - "-" "-" "-" "-" "10.111.197.103:2181" outbound|2181||druid-zookeeper-headless.druid.svc.cluster.local 10.111.197.100:49638 10.111.197.103:2181 10.111.197.100:49624 - -
[2023-02-10T15:37:42.010Z] "- - -" 0 - - - "-" 49 0 3 - "-" "-" "-" "-" "10.111.197.103:2181" outbound|2181||druid-zookeeper-headless.druid.svc.cluster.local 10.111.197.100:49646 10.111.197.103:2181 10.111.197.100:49642 - -
2023-02-10T15:37:43.174391Z	error	Request to probe app failed: Get "http://10.111.197.100:8082/status/health": dial tcp 127.0.0.6:0->10.111.197.100:8082: connect: connection refused, original URL path = /app-health/druid/livez
app URL path = /status/health
2023-02-10T15:37:43.174406Z	error	Request to probe app failed: Get "http://10.111.197.100:8082/status/health": dial tcp 127.0.0.6:0->10.111.197.100:8082: connect: connection refused, original URL path = /app-health/druid/readyz

@howardjohn
Copy link
Member

This means your app has a readiness probe defined on port 8082 and that is failing. It is NOT the istio proxy failing, but rather your apps own readiness probe.

@RunlinXiao
Copy link
Author

I see, thanks for the information John! But I have tested the app working well without istio + spire, any recommendations for me to find out the root cause of this failing?

@RunlinXiao
Copy link
Author

I think it might related to #25177

@jaellio
Copy link
Contributor

jaellio commented Feb 25, 2023

Did you encounter the same errors using Istio without SPIRE?

@istio-policy-bot istio-policy-bot added the lifecycle/stale Indicates a PR or issue hasn't been manipulated by an Istio team member for a while label May 26, 2023
@istio-policy-bot
Copy link

🚧 This issue or pull request has been closed due to not having had activity from an Istio team member since 2023-02-25. If you feel this issue or pull request deserves attention, please reopen the issue. Please see this wiki page for more information. Thank you for your contributions.

Created by the issue and PR lifecycle manager.

@istio-policy-bot istio-policy-bot added the lifecycle/automatically-closed Indicates a PR or issue that has been closed automatically. label Jun 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
lifecycle/automatically-closed Indicates a PR or issue that has been closed automatically. lifecycle/stale Indicates a PR or issue hasn't been manipulated by an Istio team member for a while
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@howardjohn @keithmattix @istio-policy-bot @jaellio @RunlinXiao