[fluentd] Add separate aggregator chart

[stevehipwell]

The two use cases for Fluentd are very different in implementation and as such rather than trying to shoehorn both into a single chart it would make sense to have a separate fluentd-aggregator chart. I already have a fluentd-aggregator chart in my repo and would be happy to create a PR to add it to this repo and deprecate mine.

It would also be useful to have an aggregator Docker image to handle the common aggregation endpoints; an aggregator isn't as concerned with size and if the plugins are layered on top of the base image it would keep the layers easily portable. I've created my own image for my Helm chart, although the chart will work fine with the base Fluentd image (by default it outputs to null and has stdout debugging available), but an official version would be better.

I'd like to call out @mrd2 for his work on PR #55 for the Fluentd collector chart, this would be intended to augment this not replace it.

[naseemkullah]

Thanks @stevehipwell , would it be totally unfeasible to shoehorn it into the one @mrd2 worked on with a switch like mode: aengt|aggregator ? I understand a whole lot of logic would need to be added but is it still possible?

[stevehipwell]

@naseemkullah I'd personally avoid that approach (one of the reasons why I wrote a chart rather than using an existing well known Fluentd chart) due to additional complexity and cognitive load required. I prefer the two chart approach for something like this.

[naseemkullah]

I understand, but N.B. this has been done successfully for fluent bit :

helm-charts/charts/fluent-bit/values.yaml

Lines 3 to 4 in 3f01709

	# kind -- DaemonSet or Deployment
	kind: DaemonSet

[stevehipwell]

I know it's an option but does anyone use it and is it a "good" approach? If you want an example of what it could look like for Fluentd take a look at the bitnami chart.

[stevehipwell]

@mrd2 I'm offering to submit a PR for this.

[stevehipwell]

I'm still keen to explore this, specifically a separate Fluentd aggregator chart and image optimised for aggregating logs from Fluent Bit and distributing them.

[naseemkullah]

cc @edsiper

[dioguerra]

@stevehipwell are you still working on this?

[stevehipwell]

@dioguerra yes, I have a chart and corresponding Docker image that are currently used on a number of production clusters. I'd like to hand both over if at all possible.

[dioguerra]

Do you think we can squeeze this into fluentd?

It would be nice to make a Aggregator/Standalone switch. Does it make sense? How different is the aggregating functionality from edditing the respective fields?
I mean, at this point i'm asking, what can we add that it is not available in the bitnami offered fluentd...
https://github.com/bitnami/charts/tree/master/bitnami/fluentd

I will have a look when i get time. If this works as i am expecting i want to make a push and add it to magnum, but all in it's due time :)

[stevehipwell]

@dioguerra I guess the first question is if there is the appetite to create and maintain a Fluentd aggregator image in the Fluent group (unlike with Fluent Bit this is required)? If there is then it shouldn't be too hard to combine the charts if that's what's wanted. I'd also suggest aligning the fluent-bit & fluentd charts so they follow a common pattern; so probably a breaking change in both cases.