integration with Prometheus/Grafana/Jaeger

[martonsereg]

the Helm chart can install all of these, but it's probably not the scope of the operator to manage these components, but we should figure out how to integrate them.

[chriscowley]

not one of the devs, but I believe for Jaeger you simply need to control the defaultConfig.zipkinAddress in the istio configMap.

Is this more of a documentation issue than an enhancement?

[waynz0r]

@chriscowley for using an existing Jaeger that would be probably enough, although we want to make it explicitly configurable in the custom resource describing Istio config.

[chriscowley]

@waynz0r certainly that would be a good MVP for tracing support.

For what its worth I just tried adding a service that matched the default value for defaultConfig.zipkinAddress in the operator that pointed to a Jaeger pod. Nothing :( no traces at all

Is there anyone at Banzai who has POCed something similar?

[waynz0r]

@chriscowley it should work, we are looking into it!

[waynz0r]

@chriscowley by our tests it generally works, but the default sampling rate is 1 out of every 100 requests. I created an issue to support changing this value (see #75).

Right now you can change it by editing the pilot deployment discovery container's PILOT_TRACE_SAMPLING env variable. Try change it to 100 for testing, you should see spans popping up in Jaeger.

[waynz0r]

@chriscowley we have just released v0.0.7 with the feature to allow sampling rate configuration in the CR.

Please let us know if you still have tracing problem!

[chriscowley]

@waynz0r 1 out of 100 seems a little light - certainly explains why I was seeing nothing from ~20 page refreshes of the bookinfo app.

I'll test it during the week.

[chriscowley]

Got bored and tested it anyway. The variable you added works perfectly, but there is still one piece of "glue" I needed to add.

The Istio seems to be hardwired to send the tracing sample to zipkin:16686. The Jaeger operator creates a service called <jaeger-name>-query which includes port 16686. I had to add a service that replicated that service called zipkin (my jaeger instance is called "tracing":

apiVersion: v1
kind: Service
metadata:
  labels:
    app: jaeger
    app.kubernetes.io/component: all-in-one
    app.kubernetes.io/instance: tracing
    app.kubernetes.io/managed-by: jaeger-operator
    app.kubernetes.io/name: tracing
    app.kubernetes.io/part-of: jaeger
  name: zipkin
  namespace: istio-system
spec:
  clusterIP: None
  ports:
  - name: zipkin
    port: 9411
    protocol: TCP
    targetPort: 9411
  - name: c-tchan-trft
    port: 14267
    protocol: TCP
    targetPort: 14267
  - name: c-binary-trft
    port: 14268
    protocol: TCP
    targetPort: 14268
  selector:
    app: jaeger
    app.kubernetes.io/component: all-in-one
    app.kubernetes.io/instance: tracing
    app.kubernetes.io/managed-by: jaeger-operator
    app.kubernetes.io/name: tracing
    app.kubernetes.io/part-of: jaeger
  sessionAffinity: None
  type: ClusterIP

In my Istio config I have got:

spec:
  ...
  enabledServices:
  - name: "zipkin"
    labelSelector: "app=jaeger"
  ...

which from some logic I cannot remember from Friday appeared reasonable to me (is that supposed to create a zipkin service as part of Istio that attaches to any pods with the label "app=jaeger"?) as I cannot remember where I found it.

[waynz0r]

@chriscowley thanks for the feedback!

The operator managed Istio components send tracing to zipkin.istio-system:9411, but creating that service is outside of the operator's scope right know.

The enabledServices variable is in the RemoteIstio spec and based on that the operator will create selectorless services at the remote side and keeps the endpoint IP of those services in sync with the IP of the actual running pods at the local side.

If you would be interested we can discuss this in more detail on our community slack (https://slack.banzaicloud.io #istio-operator)

[vhosakot]

I'm not able to install istio's prometheus and grafana. Is this issue going to be fixed anytime soon? Can we install istio's prometheus and grafana using banzaicloud/istio-operator?

I added the following section in the istio CR yaml https://github.com/banzaicloud/istio-operator/blob/release-1.1/config/samples/istio_v1beta1_istio.yaml under the spec: section and it did not work and istio's prometheus and grafana were not deployed after I installed the istio CR:

  prometheus:
    enabled: true
    hub: docker.io/prom
    tag: v2.7.1
  grafana:
    enabled: true
    image:
      repository: grafana/grafana
      tag: 6.0.0

[Laci21]

Hi @vhosakot, you are correct, right now the Istio operator cannot install neither Prometheus nor Grafana automatically for you (of course you can use both of them with the Istio operator if you install them for yourself, we use them together with the Istio operator in our Pipeline platform and they work just fine).

Our current view is that it probably should not be the responsibility of the Istio operator to install (and manage) these components. We would rather think of a solution where the operator could seamlessly integrate with them e.g. with the Prometheus Operator, so that the lifecycle of Prometheus can be managed by its own operator and responsibilities can be separated nicely instead of trying to do too much with the Istio operator itself. Now we consider going in this direction, but any opinions are very welcome.

[matyix]

@vhosakot we believe in the separation of concerns principle and the Istio operator is solely responsible for Istio. However, as @Laci21 mentioned, the Pipeline platform glues all these things together - by installing and managing the lifecycle of these components, whether they run on-premise, cloud or hybrid environment.

For a single click service mesh (Istio + lots of other things) experience I suggest to give it a try here: https://beta.banzaicloud.io/ui/

[Laci21]

As mentioned above, the operator's mere concern is to install (and upgrade) Istio, and nothing else. This is inline with the operator (Kubernetes controller) pattern and follows the separation of concerns paradigm.

You can always install Prometheus/Grafana/Jaeger afterwards - or if you are looking for a one shot Istio management tool built on top of this operator (including a UI, CLI, GraphQL API, with Jaeger, Prometheus, Grafana, and lots more) you might want to try our service mesh product, Backyards.

You can start with the Backyard CLI or read the release announcement.

If you are in quick-start mode and want to have a production Istio environment on a Kubernetes cluster with all the above in minutes just hit: curl https://getbackyards.sh | sh && backyards install -a --run-demo