Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to start up istio pilot pod using tetrate fips image #102

Closed
zonml opened this issue Jun 1, 2022 · 4 comments
Closed

Unable to start up istio pilot pod using tetrate fips image #102

zonml opened this issue Jun 1, 2022 · 4 comments
Assignees
@psbrar99

Comments

@zonml
Copy link

zonml commented Jun 1, 2022

Hello,
I installed istiod:1.13.2 with opensource image(non-fips) successfully. But failed to start up pilot pod using tetrate fips image. Error logs as following. I couldn't find a clue to it. I'd appreciate it if you guys took a look.
.

Install istiod with helm chart

releases:

  • name: istiod
    condition: istiod.enabled
    namespace: istio-system
    forceNamespace: istio-system
    chart: istio/istiod
    version: "1.13.2"
    values:
    • ../values/91-istiod.yaml.gotmpl
      wait: true

Image was pulled from IronBank and pushed to our AWS ECR.

pilot:
image: xxxxxxx.dkr.ecr.us-east-1.amazonaws.com/k8s-component/tetrate/istio/pilot:1.13.2-tetratefips-v0

Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  5m48s                  default-scheduler  Successfully assigned istio-system/istiod-5475bd8f69-wfhvk to ip-10-190-1-37.ec2.internal
  Warning  Unhealthy  5m43s (x2 over 5m45s)  kubelet            Readiness probe failed: Get "http://10.190.1.204:8080/ready": dial tcp 10.190.1.204:8080: connect: connection refused
  Normal   Created    4m58s (x4 over 5m46s)  kubelet            Created container discovery
  Normal   Started    4m58s (x4 over 5m46s)  kubelet            Started container discovery
  Normal   Pulled     4m6s (x5 over 5m47s)   kubelet            Container image "xxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/k8s-component/tetrate/istio/pilot:1.13.2-tetratefips-v0@sha256:eaf8c4f4b9d200ef9a6bd7f95a750eef3c02173aa1b50cfb89dce5299d79ccf4" already present on machine
  Warning  BackOff    39s (x28 over 5m42s)   kubelet            Back-off restarting failed container

2022-06-01T08:20:55.318190Z     info    FLAG: --caCertFile=""
2022-06-01T08:20:55.318250Z     info    FLAG: --clusterAliases="[]"
2022-06-01T08:20:55.318261Z     info    FLAG: --clusterID="Kubernetes"
2022-06-01T08:20:55.318267Z     info    FLAG: --clusterRegistriesNamespace="istio-system"
2022-06-01T08:20:55.318274Z     info    FLAG: --configDir=""
2022-06-01T08:20:55.318280Z     info    FLAG: --ctrlz_address="localhost"
2022-06-01T08:20:55.318291Z     info    FLAG: --ctrlz_port="9876"
2022-06-01T08:20:55.318297Z     info    FLAG: --domain="cluster.local"
2022-06-01T08:20:55.318303Z     info    FLAG: --grpcAddr=":15010"
2022-06-01T08:20:55.318311Z     info    FLAG: --help="false"
2022-06-01T08:20:55.318317Z     info    FLAG: --httpAddr=":8080"
2022-06-01T08:20:55.318323Z     info    FLAG: --httpsAddr=":15017"
2022-06-01T08:20:55.318332Z     info    FLAG: --keepaliveInterval="30s"
2022-06-01T08:20:55.318339Z     info    FLAG: --keepaliveMaxServerConnectionAge="30m0s"
2022-06-01T08:20:55.318345Z     info    FLAG: --keepaliveTimeout="10s"
2022-06-01T08:20:55.318350Z     info    FLAG: --kubeconfig=""
2022-06-01T08:20:55.318358Z     info    FLAG: --kubernetesApiBurst="160"
2022-06-01T08:20:55.318366Z     info    FLAG: --kubernetesApiQPS="80"
2022-06-01T08:20:55.318372Z     info    FLAG: --log_as_json="false"
2022-06-01T08:20:55.318378Z     info    FLAG: --log_caller=""
2022-06-01T08:20:55.318384Z     info    FLAG: --log_output_level="default:info"
2022-06-01T08:20:55.318392Z     info    FLAG: --log_rotate=""
2022-06-01T08:20:55.318398Z     info    FLAG: --log_rotate_max_age="30"
2022-06-01T08:20:55.318404Z     info    FLAG: --log_rotate_max_backups="1000"
2022-06-01T08:20:55.318410Z     info    FLAG: --log_rotate_max_size="104857600"
2022-06-01T08:20:55.318416Z     info    FLAG: --log_stacktrace_level="default:none"
2022-06-01T08:20:55.318481Z     info    FLAG: --log_target="[stdout]"
2022-06-01T08:20:55.318512Z     info    FLAG: --meshConfig="./etc/istio/config/mesh"
2022-06-01T08:20:55.318520Z     info    FLAG: --monitoringAddr=":15014"
2022-06-01T08:20:55.318527Z     info    FLAG: --namespace="istio-system"
2022-06-01T08:20:55.318696Z     info    FLAG: --networksConfig="./etc/istio/config/meshNetworks"
2022-06-01T08:20:55.318715Z     info    FLAG: --plugins="[ext_authz,authn,authz]"
2022-06-01T08:20:55.318722Z     info    FLAG: --profile="true"
2022-06-01T08:20:55.318735Z     info    FLAG: --registries="[Kubernetes]"
2022-06-01T08:20:55.318741Z     info    FLAG: --resync="1m0s"
2022-06-01T08:20:55.318747Z     info    FLAG: --secureGRPCAddr=":15012"
2022-06-01T08:20:55.318753Z     info    FLAG: --shutdownDuration="10s"
2022-06-01T08:20:55.318759Z     info    FLAG: --tls-cipher-suites="[]"
2022-06-01T08:20:55.318765Z     info    FLAG: --tlsCertFile=""
2022-06-01T08:20:55.318770Z     info    FLAG: --tlsKeyFile=""
2022-06-01T08:20:55.318778Z     info    FLAG: --vklog="0"
2022-06-01T08:20:55.352095Z     info    klog    Config not found: /var/run/secrets/remote/config
2022-06-01T08:20:55.360935Z     info    initializing mesh configuration ./etc/istio/config/mesh
2022-06-01T08:20:55.462159Z     info    controllers     starting        controller=configmap istio
2022-06-01T08:20:55.462576Z     info    Loaded MeshNetworks config from Kubernetes API server.
2022-06-01T08:20:55.462622Z     info    mesh networks configuration updated to: {
    "networks": {
    }
}
2022-06-01T08:20:55.464968Z     info    Loaded MeshConfig config from Kubernetes API server.
2022-06-01T08:20:55.465801Z     info    mesh configuration updated to: {
    "proxyListenPort": 15001,
    "connectTimeout": "10s",
    "protocolDetectionTimeout": "0s",
    "ingressClass": "istio",
    "ingressService": "istio-ingressgateway",
    "ingressControllerMode": "STRICT",
    "enableTracing": true,
    "defaultConfig": {
        "configPath": "./etc/istio/proxy",
        "binaryPath": "/usr/local/bin/envoy",
        "serviceCluster": "istio-proxy",
        "drainDuration": "45s",
        "parentShutdownDuration": "60s",
        "discoveryAddress": "istiod.istio-system.svc:15012",
        "proxyAdminPort": 15000,
        "controlPlaneAuthPolicy": "MUTUAL_TLS",
        "statNameLength": 189,
        "concurrency": 2,
        "tracing": {
            "zipkin": {
                "address": "zipkin.istio-system:9411"
            }
        },
        "statusPort": 15020,
        "terminationDrainDuration": "5s"
    },
    "outboundTrafficPolicy": {
        "mode": "ALLOW_ANY"
    },
    "enableAutoMtls": true,
    "trustDomain": "cluster.local",
    "trustDomainAliases": [
    ],
    "defaultServiceExportTo": [
        "*"
    ],
    "defaultVirtualServiceExportTo": [
        "*"
    ],
    "defaultDestinationRuleExportTo": [
        "*"
    ],
    "rootNamespace": "istio-system",
    "localityLbSetting": {
        "enabled": true
    },
    "dnsRefreshRate": "5s",
    "certificates": [
    ],
    "thriftConfig": {

    },
    "serviceSettings": [
    ],
    "enablePrometheusMerge": true,
    "extensionProviders": [
        {
            "name": "prometheus",
            "prometheus": {

            }
        },
        {
            "name": "stackdriver",
            "stackdriver": {

            }
        },
        {
            "name": "envoy",
            "envoyFileAccessLog": {
                "path": "/dev/stdout"
            }
        }
    ],
    "defaultProviders": {

    }
}
2022-06-01T08:20:55.561936Z     info    initializing mesh networks from mesh config watcher
2022-06-01T08:20:55.562582Z     info    mesh configuration: {
    "proxyListenPort": 15001,
    "connectTimeout": "10s",
    "protocolDetectionTimeout": "0s",
    "ingressClass": "istio",
    "ingressService": "istio-ingressgateway",
    "ingressControllerMode": "STRICT",
    "enableTracing": true,
    "defaultConfig": {
        "configPath": "./etc/istio/proxy",
        "binaryPath": "/usr/local/bin/envoy",
        "serviceCluster": "istio-proxy",
        "drainDuration": "45s",
        "parentShutdownDuration": "60s",
        "discoveryAddress": "istiod.istio-system.svc:15012",
        "proxyAdminPort": 15000,
        "controlPlaneAuthPolicy": "MUTUAL_TLS",
        "statNameLength": 189,
        "concurrency": 2,
        "tracing": {
            "zipkin": {
                "address": "zipkin.istio-system:9411"
            }
        },
        "statusPort": 15020,
        "terminationDrainDuration": "5s"
    },
    "outboundTrafficPolicy": {
        "mode": "ALLOW_ANY"
    },
    "enableAutoMtls": true,
    "trustDomain": "cluster.local",
    "trustDomainAliases": [
    ],
    "defaultServiceExportTo": [
        "*"
    ],
    "defaultVirtualServiceExportTo": [
        "*"
    ],
    "defaultDestinationRuleExportTo": [
        "*"
    ],
    "rootNamespace": "istio-system",
    "localityLbSetting": {
        "enabled": true
    },
    "dnsRefreshRate": "5s",
    "certificates": [
    ],
    "thriftConfig": {

    },
    "serviceSettings": [
    ],
    "enablePrometheusMerge": true,
    "extensionProviders": [
        {
            "name": "prometheus",
            "prometheus": {

            }
        },
        {
            "name": "stackdriver",
            "stackdriver": {

            }
        },
        {
            "name": "envoy",
            "envoyFileAccessLog": {
                "path": "/dev/stdout"
            }
        }
    ],
    "defaultProviders": {

    }
}
2022-06-01T08:20:55.562614Z     info    version: 1.13.2-tetratefips-v0-af687222b70be38751d8d0238045bc606f54f8ff-Clean
2022-06-01T08:20:55.562992Z     info    flags: {
   "ServerOptions": {
      "HTTPAddr": ":8080",
      "HTTPSAddr": ":15017",
      "GRPCAddr": ":15010",
      "MonitoringAddr": ":15014",
      "EnableProfiling": true,
      "TLSOptions": {
         "CaCertFile": "",
         "CertFile": "",
         "KeyFile": "",
         "TLSCipherSuites": null,
         "CipherSuits": null
      },
      "SecureGRPCAddr": ":15012"
   },
   "InjectionOptions": {
      "InjectionDirectory": "./var/lib/istio/inject"
   },
   "PodName": "istiod-5475bd8f69-wfhvk",
   "Namespace": "istio-system",
   "Revision": "default",
   "MeshConfigFile": "./etc/istio/config/mesh",
   "NetworksConfigFile": "./etc/istio/config/meshNetworks",
   "RegistryOptions": {
      "FileDir": "",
      "Registries": [
         "Kubernetes"
      ],
      "KubeOptions": {
         "SystemNamespace": "",
         "MeshServiceController": null,
         "ResyncPeriod": 60000000000,
         "DomainSuffix": "cluster.local",
         "ClusterID": "Kubernetes",
         "ClusterAliases": {},
         "Metrics": null,
         "XDSUpdater": null,
         "NetworksWatcher": null,
         "MeshWatcher": null,
         "EndpointMode": 1,
         "KubernetesAPIQPS": 80,
         "KubernetesAPIBurst": 160,
         "SyncInterval": 0,
         "SyncTimeout": null,
         "DiscoveryNamespacesFilter": null
      },
      "ClusterRegistriesNamespace": "istio-system",
      "KubeConfig": "",
      "DistributionCacheRetention": 60000000000,
      "DistributionTrackingEnabled": true
   },
   "CtrlZOptions": {
      "Port": 9876,
      "Address": "localhost"
   },
   "Plugins": [
      "ext_authz",
      "authn",
      "authz"
   ],
   "KeepaliveOptions": {
      "Time": 30000000000,
      "Timeout": 10000000000,
      "MaxServerConnectionAge": 1800000000000,
      "MaxServerConnectionAgeGrace": 10000000000
   },
   "ShutdownDuration": 10000000000,
   "JwtRule": ""
}
2022-06-01T08:20:55.563009Z     info    initializing mesh handlers
2022-06-01T08:20:55.563200Z     info    model   reloading network gateways
2022-06-01T08:20:55.563218Z     info    creating CA and initializing public key
2022-06-01T08:20:55.563276Z     info    Use self-signed certificate as the CA certificate
2022-06-01T08:20:55.567335Z     info    pkica   Load signing key and cert from existing secret istio-system:istio-ca-secret
2022-06-01T08:20:55.568141Z     info    pkica   Using existing public key: -----BEGIN CERTIFICATE-----
xxxxxxxxxxxxxxxx......................
-----END CERTIFICATE-----

2022-06-01T08:20:55.568203Z     info    rootcertrotator Set up back off time 18m8s to start rotator.
2022-06-01T08:20:55.568227Z     info    initializing controllers
2022-06-01T08:20:55.568284Z     info    No certificates specified, skipping K8S DNS certificate controller
2022-06-01T08:20:55.568650Z     info    rootcertrotator Jitter is enabled, wait 18m8s before starting root cert rotator.
2022-06-01T08:20:55.759169Z     warn    kube    Skipping CRD gateway.networking.k8s.io/v1alpha2/GatewayClass as it is not present
2022-06-01T08:20:55.759242Z     warn    kube    Skipping CRD gateway.networking.k8s.io/v1alpha2/Gateway as it is not present
2022-06-01T08:20:55.759253Z     warn    kube    Skipping CRD gateway.networking.k8s.io/v1alpha2/HTTPRoute as it is not present
2022-06-01T08:20:55.759260Z     warn    kube    Skipping CRD gateway.networking.k8s.io/v1alpha2/ReferencePolicy as it is not present
2022-06-01T08:20:55.759268Z     warn    kube    Skipping CRD gateway.networking.k8s.io/v1alpha2/TCPRoute as it is not present
2022-06-01T08:20:55.759278Z     warn    kube    Skipping CRD gateway.networking.k8s.io/v1alpha2/TLSRoute as it is not present
2022-06-01T08:20:55.759677Z     info    Adding Kubernetes registry adapter
2022-06-01T08:20:55.759720Z     info    handling remote clusters in *controller.Multicluster
2022-06-01T08:20:55.759759Z     info    initializing Istiod DNS certificates host: istiod.istio-system.svc, custom host:
2022-06-01T08:20:56.081424Z     info    Generating istiod-signed cert for [istiod.istio-system.svc istiod-remote.istio-system.svc istio-pilot.istio-system.svc]:
 -----BEGIN CERTIFICATE-----
xxxxxx.........................
-----END CERTIFICATE-----

2022-06-01T08:20:56.081633Z     info    No plugged-in cert at etc/cacerts/ca-key.pem; self-signed cert is used
2022-06-01T08:20:56.081992Z     info    x509 cert - Issuer: "O=cluster.local", Subject: "", SN: d010855a107d57c9aa300d69fa811358, NotBefore: "2022-06-01T08:18:56Z", NotAfter: "2032-05-29T08:20:56Z"
2022-06-01T08:20:56.082004Z     info    Istiod certificates are reloaded
2022-06-01T08:20:56.082094Z     info    spiffe  Added 1 certs to trust domain cluster.local in peer cert verifier
2022-06-01T08:20:56.082104Z     info    initializing secure discovery service
2022-06-01T08:20:56.082150Z     info    initializing secure webhook server for istiod webhooks
2022-06-01T08:20:56.088222Z     info    initializing sidecar injector
2022-06-01T08:20:56.097142Z     info    initializing config validator
2022-06-01T08:20:56.097197Z     info    initializing Istiod admin server
2022-06-01T08:20:56.097378Z     info    initializing registry event handlers
2022-06-01T08:20:56.097470Z     info    starting discovery service
2022-06-01T08:20:56.097514Z     info    handling remote clusters in *kube.Multicluster
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x47 pc=0x7f963d0f4bd0]

runtime stack:
runtime.throw({0x3818d1a, 0x7f963e5a2880})
        runtime/panic.go:1198 +0x71
runtime.sigpanic()
        runtime/signal_unix.go:719 +0x396

goroutine 99 [syscall]:
runtime.cgocall(0x2bc6540, 0xc000093d90)
        runtime/cgocall.go:156 +0x5c fp=0xc000093d68 sp=0xc000093d30 pc=0x40653c
net._C2func_getaddrinfo(0xc001665430, 0x0, 0xc001695e00, 0xc0008cb8c8)
        _cgo_gotypes.go:91 +0x56 fp=0xc000093d90 sp=0xc000093d68 pc=0x55b0f6
net.cgoLookupIPCNAME.func1({0xc001665430, 0xc000093df8, 0xc000093f38}, 0xc0016652b0, 0x203000)
        net/cgo_unix.go:163 +0x9f fp=0xc000093de8 sp=0xc000093d90 pc=0x55ce3f
net.cgoLookupIPCNAME({0x379df63, 0x3}, {0xc0016652b0, 0x589daa})
        net/cgo_unix.go:163 +0x16d fp=0xc000093f38 sp=0xc000093de8 pc=0x55c68d
net.cgoIPLookup(0x5881e5, {0x379df63, 0xc000093fd0}, {0xc0016652b0, 0xc0005e1a80})
        net/cgo_unix.go:220 +0x3b fp=0xc000093fa8 sp=0xc000093f38 pc=0x55cefb
net.cgoLookupIP·dwrap·25()
        net/cgo_unix.go:230 +0x36 fp=0xc000093fe0 sp=0xc000093fa8 pc=0x55d376
runtime.goexit()
        runtime/asm_amd64.s:1581 +0x1 fp=0xc000093fe8 sp=0xc000093fe0 pc=0x46cc81
created by net.cgoLookupIP
        net/cgo_unix.go:230 +0x125

goroutine 1 [select]:
net.(*Resolver).lookupIPAddr(0x61af840, {0x3dc3a68, 0xc000078038}, {0x379df63, 0x20}, {0xc0016652b0, 0x9})
        net/lookup.go:302 +0x5c7
net.(*Resolver).internetAddrList(0x3dc3a68, {0x3dc3a68, 0xc000078038}, {0x379df63, 0x3}, {0xc0016652b0, 0xe})
        net/ipsock.go:288 +0x67a
net.(*Resolver).resolveAddrList(0x410065, {0x3dc3a68, 0xc000078038}, {0x37a26fc, 0x6}, {0x379df63, 0x7f963d312ad8}, {0xc0016652b0, 0xe}, {0x0, ...})
        net/dial.go:221 +0x41b
net.(*ListenConfig).Listen(0xc001541698, {0x3dc3a68, 0xc000078038}, {0x379df63, 0xc0015416a8}, {0xc0016652b0, 0xe})
        net/dial.go:626 +0x85
net.Listen({0x379df63, 0x5}, {0xc0016652b0, 0x2})
        net/dial.go:712 +0x4b
istio.io/pkg/ctrlz.Run(0xc000c70810, {0x0, 0x0, 0x17})
        istio.io/pkg@v0.0.0-20220304033656-f98ba9ebf791/ctrlz/ctrlz.go:168 +0x66a
istio.io/istio/pilot/pkg/bootstrap.NewServer(0xc000afef00, {0x0, 0x0, 0x0})
        istio.io/istio/pilot/pkg/bootstrap/server.go:350 +0x161f
istio.io/istio/pilot/cmd/pilot-discovery/app.newDiscoveryCommand.func2(0xc000afec80, {0xc00044ade0, 0x6, 0x6})
        istio.io/istio/pilot/cmd/pilot-discovery/app/cmd.go:92 +0x4e
github.com/spf13/cobra.(*Command).execute(0xc000afec80, {0xc00044ad80, 0x6, 0x6})
        github.com/spf13/cobra@v1.3.0/command.go:856 +0x60e
github.com/spf13/cobra.(*Command).ExecuteC(0xc000afea00)
        github.com/spf13/cobra@v1.3.0/command.go:974 +0x3bc
github.com/spf13/cobra.(*Command).Execute(...)
        github.com/spf13/cobra@v1.3.0/command.go:902
main.main()
        istio.io/istio/pilot/cmd/pilot-discovery/main.go:27 +0x25

goroutine 6 [chan receive]:
k8s.io/klog/v2.(*loggingT).flushDaemon(0x0)
        k8s.io/klog/v2@v2.40.1/klog.go:1283 +0x6a
created by k8s.io/klog/v2.init.0
        k8s.io/klog/v2@v2.40.1/klog.go:420 +0xfb

goroutine 7 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc000070500)
        go.opencensus.io@v0.23.0/stats/view/worker.go:276 +0xb9
created by go.opencensus.io/stats/view.init.0
        go.opencensus.io@v0.23.0/stats/view/worker.go:34 +0x92

goroutine 40 [select]:
k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc001173a40)
        k8s.io/client-go@v0.23.1/util/workqueue/delaying_queue.go:231 +0x34e
created by k8s.io/client-go/util/workqueue.newDelayingQueue
        k8s.io/client-go@v0.23.1/util/workqueue/delaying_queue.go:68 +0x247

goroutine 23 [select]:
istio.io/pkg/cache.(*ttlCache).evicter(0xc000139f00, 0xc00054daa0)
        istio.io/pkg@v0.0.0-20220304033656-f98ba9ebf791/cache/ttlCache.go:123 +0xb2
created by istio.io/pkg/cache.NewTTLWithCallback
        istio.io/pkg@v0.0.0-20220304033656-f98ba9ebf791/cache/ttlCache.go:102 +0x165

goroutine 41 [select]:
istio.io/istio/pkg/kube/controllers.Queue.Run({{0x3e2e458, 0xc00055c460}, 0xc000c38e00, {0xc000c386b0, 0xf}, 0x0, 0xc000976230, 0xc000b70360}, 0xc0000c63c0)
        istio.io/istio/pkg/kube/controllers/queue.go:107 +0x225
istio.io/istio/pkg/kube/configmapwatcher.(*Controller).Run(0xc0005e0200, 0xc0000c63c0)
        istio.io/istio/pkg/kube/configmapwatcher/configmapwatcher.go:81 +0x1eb
created by istio.io/istio/pkg/config/mesh/kubemesh.NewConfigMapWatcher
        istio.io/istio/pkg/config/mesh/kubemesh/watcher.go:59 +0x252

goroutine 24 [select]:
istio.io/istio/pilot/pkg/model.(*JwksResolver).refresher(0xc000b2c000)
        istio.io/istio/pilot/pkg/model/jwks_resolver.go:385 +0xb2
created by istio.io/istio/pilot/pkg/model.newJwksResolverWithCABundlePaths
        istio.io/istio/pilot/pkg/model/jwks_resolver.go:218 +0x313

goroutine 33 [select]:
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc0005e0280)
        k8s.io/client-go@v0.23.1/tools/cache/shared_informer.go:752 +0x156
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        k8s.io/apimachinery@v0.23.1/pkg/util/wait/wait.go:73 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        k8s.io/apimachinery@v0.23.1/pkg/util/wait/wait.go:71 +0x88

goroutine 66 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc000c6aac8, 0x0)
        runtime/sema.go:513 +0x13d
sync.(*Cond).Wait(0x379d901)
        sync/cond.go:56 +0x8c
golang.org/x/net/http2.(*pipe).Read(0xc000c6aab0, {0xc0005b4e00, 0x200, 0x200})
        golang.org/x/net@v0.0.0-20220114011407-0dd24b26b47d/http2/pipe.go:76 +0xeb
golang.org/x/net/http2.transportResponseBody.Read({0x100000000000000}, {0xc0005b4e00, 0x0, 0xc00087dcb0})
        golang.org/x/net@v0.0.0-20220114011407-0dd24b26b47d/http2/transport.go:2384 +0x85
encoding/json.(*Decoder).refill(0xc00099cb40)
        encoding/json/stream.go:165 +0x17f
encoding/json.(*Decoder).readValue(0xc00099cb40)
        encoding/json/stream.go:140 +0xbb
encoding/json.(*Decoder).Decode(0xc00099cb40, {0x31cd580, 0xc000965248})
        encoding/json/stream.go:63 +0x78
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc0005e4f00, {0xc0011db800, 0x400, 0x400})
        k8s.io/apimachinery@v0.23.1/pkg/util/framer/framer.go:152 +0x19c
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc00080e820, 0xc0000c73e0, {0x3d87520, 0xc00115b500})
        k8s.io/apimachinery@v0.23.1/pkg/runtime/serializer/streaming/streaming.go:77 +0xa7
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc0000396e0)
        k8s.io/client-go@v0.23.1/rest/watch/decoder.go:49 +0x4f
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc00115b4c0)
        k8s.io/apimachinery@v0.23.1/pkg/watch/streamwatcher.go:105 +0x11c
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
        k8s.io/apimachinery@v0.23.1/pkg/watch/streamwatcher.go:76 +0x135

goroutine 30 [IO wait]:
internal/poll.runtime_pollWait(0x7f963fdf9118, 0x72)
        runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc000b2cd00, 0xc001410000, 0x0)
        internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000b2cd00, {0xc001410000, 0x8d42, 0x8d42})
        internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc000b2cd00, {0xc001410000, 0xc001414718, 0x1a})
        net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc0008ca0e8, {0xc001410000, 0x6e8919, 0xc0011f57f0})
        net/net.go:183 +0x45
crypto/tls.(*atLeastReader).Read(0xc00139dea8, {0xc001410000, 0x0, 0x40cd6d})
        crypto/tls/conn.go:777 +0x3d
bytes.(*Buffer).ReadFrom(0xc000b52278, {0x3d2a860, 0xc00139dea8})
        bytes/buffer.go:204 +0x98
crypto/tls.(*Conn).readFromUntil(0xc000b52000, {0x3d43520, 0xc0008ca0e8}, 0x462f)
        crypto/tls/conn.go:799 +0xe5
crypto/tls.(*Conn).readRecordOrCCS(0xc000b52000, 0x0)
        crypto/tls/conn.go:606 +0x112
crypto/tls.(*Conn).readRecord(...)
        crypto/tls/conn.go:574
crypto/tls.(*Conn).Read(0xc000b52000, {0xc0011f9000, 0x1000, 0x919e60})
        crypto/tls/conn.go:1277 +0x16f
bufio.(*Reader).Read(0xc0009c1260, {0xc000148740, 0x9, 0x934e22})
        bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x3d2a5c0, 0xc0009c1260}, {0xc000148740, 0x9, 0x9}, 0x9)
        io/io.go:328 +0x9a
io.ReadFull(...)
        io/io.go:347
golang.org/x/net/http2.readFrameHeader({0xc000148740, 0x9, 0xc0013fccf0}, {0x3d2a5c0, 0xc0009c1260})
        golang.org/x/net@v0.0.0-20220114011407-0dd24b26b47d/http2/frame.go:237 +0x6e
golang.org/x/net/http2.(*Framer).ReadFrame(0xc000148700)
        golang.org/x/net@v0.0.0-20220114011407-0dd24b26b47d/http2/frame.go:498 +0x95
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc0011f5f98)
        golang.org/x/net@v0.0.0-20220114011407-0dd24b26b47d/http2/transport.go:2101 +0x130
golang.org/x/net/http2.(*ClientConn).readLoop(0xc000b3f980)
        golang.org/x/net@v0.0.0-20220114011407-0dd24b26b47d/http2/transport.go:1997 +0x6f
created by golang.org/x/net/http2.(*Transport).newClientConn
        golang.org/x/net@v0.0.0-20220114011407-0dd24b26b47d/http2/transport.go:725 +0xac5

goroutine 55 [select]:
k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc001172660)
        k8s.io/client-go@v0.23.1/util/workqueue/delaying_queue.go:231 +0x34e
created by k8s.io/client-go/util/workqueue.newDelayingQueue
        k8s.io/client-go@v0.23.1/util/workqueue/delaying_queue.go:68 +0x247

goroutine 56 [select]:
k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc0011727e0)
        k8s.io/client-go@v0.23.1/util/workqueue/delaying_queue.go:231 +0x34e
created by k8s.io/client-go/util/workqueue.newDelayingQueue
        k8s.io/client-go@v0.23.1/util/workqueue/delaying_queue.go:68 +0x247

goroutine 44 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc0000e6a28, 0x1)
        runtime/sema.go:513 +0x13d
sync.(*Cond).Wait(0xc00055cbc0)
        sync/cond.go:56 +0x8c
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc0000e6a00, 0xc000976470)
        k8s.io/client-go@v0.23.1/tools/cache/delta_fifo.go:527 +0x233
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc000b703f0)
        k8s.io/client-go@v0.23.1/tools/cache/controller.go:183 +0x36
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7f963dbcd548)
        k8s.io/apimachinery@v0.23.1/pkg/util/wait/wait.go:155 +0x67
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x13d1828, {0x3d427a0, 0xc00092e930}, 0x1, 0xc0000c63c0)
        k8s.io/apimachinery@v0.23.1/pkg/util/wait/wait.go:156 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000b70458, 0x3b9aca00, 0x0, 0x0, 0x7f963dba3840)
        k8s.io/apimachinery@v0.23.1/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
        k8s.io/apimachinery@v0.23.1/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*controller).Run(0xc000b703f0, 0xc0000c63c0)
        k8s.io/client-go@v0.23.1/tools/cache/controller.go:154 +0x2fb
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc0000e6500, 0xc001173a40)
        k8s.io/client-go@v0.23.1/tools/cache/shared_informer.go:414 +0x498
created by istio.io/istio/pkg/kube/configmapwatcher.(*Controller).Run
        istio.io/istio/pkg/kube/configmapwatcher/configmapwatcher.go:76 +0xc5

goroutine 72 [select]:
k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc0009c0cc0)
        k8s.io/client-go@v0.23.1/util/workqueue/delaying_queue.go:231 +0x34e
created by k8s.io/client-go/util/workqueue.newDelayingQueue
        k8s.io/client-go@v0.23.1/util/workqueue/delaying_queue.go:68 +0x247

goroutine 32 [chan receive]:
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
        k8s.io/client-go@v0.23.1/tools/cache/shared_informer.go:782 +0x49
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7f963dbcc4d0)
        k8s.io/apimachinery@v0.23.1/pkg/util/wait/wait.go:155 +0x67
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00087df38, {0x3d427a0, 0xc0005e4cf0}, 0x1, 0xc0000c73e0)
        k8s.io/apimachinery@v0.23.1/pkg/util/wait/wait.go:156 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0, 0x3b9aca00, 0x0, 0x0, 0xc00087df88)
        k8s.io/apimachinery@v0.23.1/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
        k8s.io/apimachinery@v0.23.1/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*processorListener).run(0xc0005e0280)
        k8s.io/client-go@v0.23.1/tools/cache/shared_informer.go:781 +0x6b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        k8s.io/apimachinery@v0.23.1/pkg/util/wait/wait.go:73 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        k8s.io/apimachinery@v0.23.1/pkg/util/wait/wait.go:71 +0x88

goroutine 48 [chan receive]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc000bc61c0, 0x0)
        k8s.io/client-go@v0.23.1/tools/cache/shared_informer.go:638 +0x45
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
        k8s.io/apimachinery@v0.23.1/pkg/util/wait/wait.go:56 +0x22
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        k8s.io/apimachinery@v0.23.1/pkg/util/wait/wait.go:73 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        k8s.io/apimachinery@v0.23.1/pkg/util/wait/wait.go:71 +0x88

goroutine 49 [chan receive]:
k8s.io/client-go/tools/cache.(*controller).Run.func1()
        k8s.io/client-go@v0.23.1/tools/cache/controller.go:130 +0x28
created by k8s.io/client-go/tools/cache.(*controller).Run
        k8s.io/client-go@v0.23.1/tools/cache/controller.go:129 +0x105

goroutine 50 [select]:
k8s.io/client-go/tools/cache.(*Reflector).watchHandler(0xc001181500, {0x0, 0x0, 0x61b26c0}, {0x3d877c8, 0xc00115b4c0}, 0xc0011f1d18, 0xc0002cb1a0, 0xc0000c63c0)
        k8s.io/client-go@v0.23.1/tools/cache/reflector.go:469 +0x1b6
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc001181500, 0xc0000c63c0)
        k8s.io/client-go@v0.23.1/tools/cache/reflector.go:429 +0x696
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
        k8s.io/client-go@v0.23.1/tools/cache/reflector.go:221 +0x26
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7f963dbcd548)
        k8s.io/apimachinery@v0.23.1/pkg/util/wait/wait.go:155 +0x67
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc001148ac0, {0x3d42780, 0xc0011a4ff0}, 0x1, 0xc0000c63c0)
        k8s.io/apimachinery@v0.23.1/pkg/util/wait/wait.go:156 +0xb6
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc001181500, 0xc0000c63c0)
        k8s.io/client-go@v0.23.1/tools/cache/reflector.go:220 +0x1f8
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
        k8s.io/apimachinery@v0.23.1/pkg/util/wait/wait.go:56 +0x22
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        k8s.io/apimachinery@v0.23.1/pkg/util/wait/wait.go:73 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        k8s.io/apimachinery@v0.23.1/pkg/util/wait/wait.go:71 +0x88

goroutine 53 [select]:
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func2()
        k8s.io/client-go@v0.23.1/tools/cache/reflector.go:374 +0x12d
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
        k8s.io/client-go@v0.23.1/tools/cache/reflector.go:368 +0x378

goroutine 54 [select]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc000c6aa80, 0xc00053f300)
        golang.org/x/net@v0.0.0-20220114011407-0dd24b26b47d/http2/transport.go:1323 +0xaa8
golang.org/x/net/http2.(*clientStream).doRequest(0x0, 0x0)
        golang.org/x/net@v0.0.0-20220114011407-0dd24b26b47d/http2/transport.go:1185 +0x1e
created by golang.org/x/net/http2.(*ClientConn).RoundTrip
        golang.org/x/net@v0.0.0-20220114011407-0dd24b26b47d/http2/transport.go:1114 +0x30f

goroutine 67 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc00095bed0, 0x0)
        runtime/sema.go:513 +0x13d
sync.(*Cond).Wait(0x341c940)
        sync/cond.go:56 +0x8c
k8s.io/client-go/util/workqueue.(*Type).Get(0xc001173920)
        k8s.io/client-go@v0.23.1/util/workqueue/queue.go:157 +0x9e
istio.io/istio/pkg/kube/controllers.Queue.processNextItem({{0x3e2e458, 0xc00055c460}, 0xc000c38e00, {0xc000c386b0, 0xf}, 0x0, 0xc000976230, 0xc000b70360})
        istio.io/istio/pkg/kube/controllers/queue.go:131 +0x95
istio.io/istio/pkg/kube/controllers.Queue.Run.func1()
        istio.io/istio/pkg/kube/controllers/queue.go:103 +0x4e
created by istio.io/istio/pkg/kube/controllers.Queue.Run
        istio.io/istio/pkg/kube/controllers/queue.go:101 +0x1d3

goroutine 69 [select]:
istio.io/istio/security/pkg/pki/ca.(*SelfSignedCARootCertRotator).Run(0xc0000aee20, 0xc0000c63c0)
        istio.io/istio/security/pkg/pki/ca/selfsignedcarootcertrotator.go:84 +0x119
created by istio.io/istio/security/pkg/pki/ca.(*IstioCA).Run
        istio.io/istio/security/pkg/pki/ca/ca.go:304 +0x88

goroutine 70 [select]:
k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc0007ec900)
        k8s.io/client-go@v0.23.1/util/workqueue/delaying_queue.go:231 +0x34e
created by k8s.io/client-go/util/workqueue.newDelayingQueue
        k8s.io/client-go@v0.23.1/util/workqueue/delaying_queue.go:68 +0x247

goroutine 98 [select]:
net.cgoLookupIP({0x3dc3a30, 0xc001698440}, {0x379df63, 0x9}, {0xc0016652b0, 0x0})
        net/cgo_unix.go:231 +0x1b7
net.(*Resolver).lookupIP(0x61af840, {0x3dc3a30, 0xc001698440}, {0x379df63, 0x3}, {0xc0016652b0, 0x9})
        net/lookup_unix.go:97 +0x128
net.glob..func1({0x3dc3a30, 0xc001698440}, 0x3, {0x379df63, 0x0}, {0xc0016652b0, 0xc001672598})
        net/hook.go:23 +0x3d
net.(*Resolver).lookupIPAddr.func1()
        net/lookup.go:296 +0x9f
internal/singleflight.(*Group).doCall(0x61af850, 0xc000801db0, {0xc0016652c0, 0xd}, 0xc001682880)
        internal/singleflight/singleflight.go:95 +0x3b
created by internal/singleflight.(*Group).DoChan
        internal/singleflight/singleflight.go:88 +0x2f1
@zinuga
Copy link

zinuga commented Jun 7, 2022

@pmerrison

@psbrar99 psbrar99 self-assigned this Jun 7, 2022
@zonml
Copy link
Author

zonml commented Jun 9, 2022

It works well if I use below tetrate images. But the pilot pod failed to start up when I used IronBank images. I have to use the IronBank(FedRamp) images for fips istio installation.

containers.istio.tetratelabs.com/pilot:1.13.2-tetratefips-v0 and containers.istio.tetratelabs.com/proxyv2:1.13.2-tetratefips-v0

@zonml
Copy link
Author

zonml commented Jun 24, 2022

Our EKS node image is centos7. Not sure if it's related to this. After replacing the base image of IronBank pilot Dockerfile, it works now.

@psbrar99
Copy link
Contributor

@zonml : you are right, the issue was with base image, this is fixed in 1.13.5, please use that one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@psbrar99 psbrar99

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@zinuga @zonml @psbrar99