Support managing loki

[ibotty]

It would be great to allow grafana-operator to also manage the various components of loki.

[pb82]

I've had a quick look and I think Loki and Promtail could run as sidecar containers in the Grafana Pod. So this should already be possible by using the containers field of the Grafana CR. If you need to mount extra configuration you could store it in a secret and mount it to a volume using the secrets field.

We use this mechanism to add an Oauth proxy in front of Grafana, see our example.

If you'd like to try out adding Loki in this way i'd be very interested in the result.

Edit: the secrets and containers fields are documented here.

[ibotty]

I don't think, that's going to be enough. To ingest logs from the whole cluster (or at least parts of the container) one needs a sidecar in any container or preferably running promtail once on each node, reading docker or runc logs directly. I also think the operator should manage each Loki part separately to allow to scale it separately. I tend to think that it makes sense to integrate loki in grafana operator, because it will benefit from integration of Loki in Grafana, but maybe it's not in scope. What do you think? I might be able to contribute.

[pb82]

I see, so ideally Loki would run in its own Pod to scale it independently of Grafana? That makes sense. Its not currently on our Roadmap but if you want to contribute you're very welcome. If there is an extra Pod independent of the Grafana Pod then i think we also need a Loki CR similar to the Grafana CR.

The Prometheus Operator can be taken as an example: it not only manages Prometheus instances but also Alertmanager which runs in a separate Pod.

[NissesSenap]

We are currently discussing how to solve this issue. Our current plan points towards having a separate operator loki and grafana. You can find more information here: #497

A RedHat team have already started working on a loki operator, https://github.com/ViaQ/loki-operator please have a look and try it out.
Them and us would love feedback.
ping @ibotty

[ibotty]

Oh, interesting. I did not see the other operator. I am fine either way. I thought it made sense to bundle them together because it's nice to have the same label config from the same source.

[NissesSenap]

It's rather new so it's nothing strange with that.
Fully understandable but we think it will be easier to maintain them as separate operators.
We are just in the starting pits on how to manage this in the future, but we will defiantly take a look at the loki operator API and try to get it as similar as possible to the grafana-operator. This might of course include changes to the grafana-operator as well, but the more feedback we can get the better.

[HVBE]

@ibotty We've decided to close this issue off, as we'll be moving to a more plug-n-play approach with the Loki and Promtail operators, please feel free to leave your feedback here based on your evaluation of the viaq Loki operator, we'd love to hear your thoughts on this as we'll be evaluating this soon ourselves as well. Thanks!