Kiali Interface Jaeger GRPC client error

[dragosrosculete]

Hello,

In Kiali interface I am getting this error when trying to access traces

Could not fetch traces.
GetAppTraces, Jaeger GRPC client error: rpc error: code = Unavailable desc = connection closed

I have installed Istio, Jager and Kiali like this:
I am using Istio 1.9.4, installed with Istioctl with the following command and profile

istioctl install --set profile=empty -y -f manifests/profiles/custom_profile.yaml

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
spec:
  hub: docker.io/istio
  tag: 1.9.4
  # You may override parts of meshconfig by uncommenting the following lines.
  meshConfig:
    enableTracing: true
    defaultConfig:
      tracing:
        zipkin:
          address: jaeger-collector.observability:9411
      proxyMetadata: {}
    enablePrometheusMerge: true
  # Traffic management feature
  components:
    base:
      enabled: true
    pilot:
      enabled: true
    # Istio Gateway feature
    ingressGateways:
    - name: istio-ingressgateway-internal
      enabled: true
      label:
          istio: istio-ingressgateway-internal
      k8s:
        serviceAnnotations:
          alb.ingress.kubernetes.io/healthcheck-path: /healthz/ready
          alb.ingress.kubernetes.io/healthcheck-port: "15021"
        service:
          type: NodePort
          ports:
            - name: web
              protocol: TCP
              port: 80
              targetPort: 8080
              nodePort: 30080
            - name: websecure
              protocol: TCP
              port: 443
              targetPort: 8443
              nodePort: 30443
            - name: status
              protocol: TCP
              port: 15021
              targetPort: 15021
              nodePort: 30021
        overlays:
        - apiVersion: apps/v1alpha1
          kind: Deployment
          name: istio-ingressgateway-internal
          patches:
          - path: spec.template.spec.containers[name:istio-proxy].lifecycle
            value:
               preStop:
                 exec:
                   command: ["sh", "-c", "sleep 10"]

    - name: istio-ingressgateway-external
      enabled: true
      label:
          istio: istio-ingressgateway-external
      k8s:
        serviceAnnotations:
          alb.ingress.kubernetes.io/healthcheck-path: /healthz/ready
          alb.ingress.kubernetes.io/healthcheck-port: "15021"
        service:
          type: NodePort
          ports:
            - name: web
              protocol: TCP
              port: 80
              targetPort: 8080
              nodePort: 31080
            - name: websecure
              protocol: TCP
              port: 443
              targetPort: 8443
              nodePort: 31443
            - name: status
              protocol: TCP
              port: 15021
              targetPort: 15021
              nodePort: 31021
        overlays:
        - apiVersion: apps/v1alpha1
          kind: Deployment
          name: istio-ingressgateway-external
          patches:
          - path: spec.template.spec.containers[name:istio-proxy].lifecycle
            value:
               preStop:
                 exec:
                   command: ["sh", "-c", "sleep 10"]

    egressGateways:
    - name: istio-egressgateway
      enabled: false

• Installed Jaeger Operator 1.22 using the official doc https://www.jaegertracing.io/docs/1.22/operator/.
  The only thing modified is the following value in order to scan all namespaces.

        - name: WATCH_NAMESPACE
          value:

kubectl create namespace observability
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/crds/jaegertracing.io_jaegers_crd.yaml
kubectl create -n observability -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/service_account.yaml
kubectl create -n observability -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role.yaml
kubectl create -n observability -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role_binding.yaml

kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/cluster_role.yaml
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/cluster_role_binding.yaml

kubectl create -n observability -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/operator.yaml

• Installed the Jaeger agent as a daemonset with ElasticSearch as backed
  #install daemonset#

cat <<EOF | kubectl apply -f -
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: jaeger
  namespace: observability
spec:
  agent:
    strategy: DaemonSet
    options:
      log-level: debug
  storage:
    type: elasticsearch
    options:
      es:
        server-urls: http://elk.devops.blablabla.com
        index-prefix: jaeger
EOF

After applying I am seeing this:
kubectl get pods -n observability
NAME READY STATUS RESTARTS AGE
jaeger-74fb94db6b-dtmdt 1/1 Running 0 2m10s
jaeger-agent-daemonset-55nkf 1/1 Running 0 2m
jaeger-agent-daemonset-fpnl5 1/1 Running 0 2m
jaeger-agent-daemonset-jqpp6 1/1 Running 0 119s
jaeger-operator-6954c97677-xwlvg 1/1 Running 0 2m51s

kubectl get svc -n observability
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
jaeger-agent ClusterIP None 5775/UDP,5778/TCP,6831/UDP,6832/UDP 2m23s
jaeger-collector ClusterIP 172.20.6.94 9411/TCP,14250/TCP,14267/TCP,14268/TCP 2m23s
jaeger-collector-headless ClusterIP None 9411/TCP,14250/TCP,14267/TCP,14268/TCP 2m23s
jaeger-operator-metrics ClusterIP 172.20.124.79 8383/TCP,8686/TCP 2m45s
jaeger-query ClusterIP 172.20.143.204 16686/TCP 2m23s

• Kiali installation is done with helm
  #Kiali installation

helm install --set cr.create=true --set cr.namespace=istio-system --namespace kiali-operator --repo https://kiali.org/helm-charts --version 1.35 kiali-operator kiali-operator

And the CR.yaml

cat <<EOF | kubectl apply -f -
apiVersion: kiali.io/v1alpha1
kind: Kiali
metadata:
  namespace: istio-system
  name: kiali
spec:
  istio_namespace: "istio-system"
  istio_component_namespaces:
    prometheus: monitoring
  auth:
    strategy: token
  deployment:
    accessible_namespaces:
    - '**'
  external_services:
      prometheus:
        cache_duration: 10
        cache_enabled: true
        cache_expiration: 300
        url: "http://prometheus-k8s.monitoring:9090"
      tracing:
        enable: true
        in_cluster_url: 'http://jaeger-query.observability:16686'
EOF

#Execute a curl from a kiali pod
kubectl exec kiali-86c8874c6f-btpqk -n istio-system -- curl http://jaeger-query.observability:16686

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3252    0  3252    0     0   453k      0 --:--:-- --:--:-- --:--:--  453k
<!doctype html><html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><meta http-equiv="cache-control" content="max-age=0"/><meta http-equiv="cache-control" content="no-cache"/><meta http-equiv="expires" content="0"/><meta http-equiv="expires" content="Tue, 01 Jan 1980 1:00:00 GMT"/><meta http-equiv="pragma" content="no-cache"/><base href="/" data-inject-target="BASE_URL"/><link rel="shortcut icon" href="./favicon.ico"><title>Jaeger UI</title><script>// Jaeger UI config data is embedded by the query-service via search-replace.
      // This is later merged with defaults into the redux `state.config` via
      // src/utils/config/get-config.js.
      // JAEGER_CONFIG_JS
      // the line above may be replaced by user-provided JS file that should define a UIConfig function.
      function getJaegerUiConfig() {
        if(typeof window.UIConfig === 'function') {
          return UIConfig();
        }
        const DEFAULT_CONFIG = null;
        const JAEGER_CONFIG = DEFAULT_CONFIG;
        return JAEGER_CONFIG;
      }
      // Jaeger version data is embedded by the query-service via search/replace.
      function getJaegerVersion() {
        const DEFAULT_VERSION = {"gitCommit":"", "gitVersion":"", "buildDate":""};
        const JAEGER_VERSION = {"gitCommit":"c991a00d5a1a34a5ba92e1d5af9d438a43f6e70c","gitVersion":"v1.22.0","buildDate":"2021-02-23T22:57:31Z"};
        return JAEGER_VERSION;
      }</script><link href="./static/css/1.dd6aa385.chunk.css" rel="stylesheet"><link href="./static/css/main.edb4cd5e.chunk.css" rel="stylesheet"></head><body><div id="jaeger-ui-root"></div><script>!function(l){function e(e){for(var r,t,n=e[0],o=e[1],u=e[2],f=0,i=[];f<n.length;f++)t=n[f],p[t]&&i.push(p[t][0]),p[t]=0;for(r in o)Object.prototype.hasOwnProperty.call(o,r)&&(l[r]=o[r]);for(s&&s(e);i.length;)i.shift()();return c.push.apply(c,u||[]),a()}function a(){for(var e,r=0;r<c.length;r++){for(var t=c[r],n=!0,o=1;o<t.length;o++){var u=t[o];0!==p[u]&&(n=!1)}n&&(c.splice(r--,1),e=f(f.s=t[0]))}return e}var t={},p={2:0},c=[];function f(e){if(t[e])return t[e].exports;var r=t[e]={i:e,l:!1,exports:{}};return l[e].call(r.exports,r,r.exports,f),r.l=!0,r.exports}f.m=l,f.c=t,f.d=function(e,r,t){f.o(e,r)||Object.defineProperty(e,r,{enumerable:!0,get:t})},f.r=function(e){"undefined"!=typeof Symbol&&Symbol.toStringTag&&Object.defineProperty(e,Symbol.toStringTag,{value:"Module"}),Object.defineProperty(e,"__esModule",{value:!0})},f.t=function(r,e){if(1&e&&(r=f(r)),8&e)return r;if(4&e&&"object"==typeof r&&r&&r.__esModule)return r;var t=Object.create(null);if(f.r(t),Object.defineProperty(t,"default",{enumerable:!0,value:r}),2&e&&"string"!=typeof r)for(var n in r)f.d(t,n,function(e){return r[e]}.bind(null,n));return t},f.n=function(e){var r=e&&e.__esModule?function(){return e.default}:function(){return e};return f.d(r,"a",r),r},f.o=function(e,r){return Object.prototype.hasOwnProperty.call(e,r)},f.p="./";var r=window.webpackJsonp=window.webpackJsonp||[],n=r.push.bind(r);r.push=e,r=r.slice();for(var o=0;o<r.length;o++)e(r[o]);var s=n;a()}([])</script><script src="./static/js/1.fe17bc3d.chunk.js"></script><script src="./static/js/main.ba26371f.chunk.js"></script></body></html>

[dragosrosculete]

Have also tried with a simple all in one Jaeger and I am still getting that error.
Launching Jaeger in istio-system gives same error

[dragosrosculete]

Found the problem. It seems it has to do with Jaeger Operator version
Doesn't work with jaeger-operator:1.22.1
Works with jaeger-operator:1.20

Thank you @lucasponce for helping with this issue .

[lucasponce]

Thanks @dragosrosculete for taking the time to report this issue.