Logging Resource Design

[fhennig]

This is part of this epic: #202

This ticket is about deciding how existing resources (DruidCluster, NifiCluster etc) should be modified and if we want additional CRDs that possibly should be referenced, like we do with S3Buckets.

Things that the ADR should specify:

• logging architecture and aggregation architecture (Decide about log agent architecture #259)
  • vector sidecar
  • vector aggregator - interchangable by the customer
  • opensearch dashboards - interchangable by the customer
• common "stackable log format"
  • timestamp, log level, message
  • what else? additional information added by vector?
    • sidecar is probably part of what we offer, can't be modified
  • this is our default, but the customer can supply their own log4j file if they want to
• common CRD fragment
• A log level should be configurable in the product already. I.e. don't log at DEBUG by default and filter later; that's too costly
• Log rotation should be used, logs can be discarded after vector has read them.
• stackable/logs/* is an emptyDir so it can be shared between containers (product and agent)

Open questions:

• Which things should be configurable and at which granularity?
• Log line metadata: everything is exported, and the user can't change this. (which pod, which role, timestamp, log message)?

Extra things to consider

• If the opensearch sink is unavailable, logs might be lost eventually
• It would be great but not strictly mandatory if we could support the hyperscaler log collectors
  • e.g. Google has "Cloud Logging" which uses fluentbit
  • do not spend more than half a day researching this please

[lfrancke]

Is this the ticket which defines the CRD layout for logging stuff?

[fhennig]

Yes, I've updated the description to clarify this

[lfrancke]

The result of this should be an ADR but it should not overlap with the ADR from #259

[lfrancke]

I've added one more point at the end, otherwise looks good, thanks!