Have Observability Stack apps deployed and running internally

[cokiengchiara]

Outcome: Deploy and run Observability Stack apps internally. Goal is to get experience and gain knowledge doing so, to set the ground for future customer use.

The apps in the Observability Stack are:

• Prometheus Operator for metrics
• Loki for logs
• Linkerd for service mesh
• Grafana for data visualization
• FluxCD for gitops config management
• Jaeger for tracing

Reference:

• How the Cloud-Native Stack Helps Writing Minimal Microservices

Assumptions

From @piontec in internal issue https://github.com/giantswarm/giantswarm/issues/14018:

• Decide where to put it: front-end cluster or testing installation (slightly towards testing installation)

• Frontend cluster
• Operations cluster

• Make sure sig-ux and sig-infra know about and are fine with this.

We need a CP where we put our tools (fluxcd) - gorilla seems to be the best choice
We deploy Loki to a separate new observability cluster we create on CP chosen

Apps

(from the most important):

Note: prometheus-operator should be already running for the control plane and/or frontend clusters. We just need to sync with biscuit to check if we need to improve anything.

• FluxCD - for gitops config management FluxCD app: Deploy and run for gitops config management (internal) #239
• Loki - for logs Make Loki chart deploy a usable High Availability (HA) setup #202
• Linkerd - service mesh Linkerd app is running in Operations and Frontend clusters #240
• Grafana - Grafana app: Add, deploy, and run in Playground Catalog #241

Bonus objectives #242

• Do PoC allowing us to stream logs from NGINX IC in real customers' TCs to our Loki
• Jaeger - deploy and collect traces.
  Optional, because we don't have explicit tracing instrumentation
• Move apps we're happy with after the cycle from Playground Catalog to Giant Swarm Catalog

[cokiengchiara]

@giantswarm/team-halo

Reminder from @calvix for sig-infra: As long as you set proper requests and limits we should be fine, so please focus on that.

[cokiengchiara]

Mon, 30. November 2020 update. Grafana is now available as an app in Giant Swarm Catalog.

[cokiengchiara]

• @ubergesundheit will install flux on gollum. Prepare repo. Remember to announce in sig-infra and sig-front-end when we touch front end clusters.
• @glitchcrab installs Linkerd w flux
• Loki -- @glitchcrab can help. @ced0ps would like to help w Loki.

[ubergesundheit]

Notified sig-ux in sig-ux sync today about the plan to install flux, linkerd and loki on gollum & the frontend cluster (c68pn)

[cokiengchiara]

• Write internal announcement about what we set out to accomplish in the crazy cycle, what we did, and what's next

[cokiengchiara]

Closing this as it's generally done. Follow up work in Loki cycle.