docs/logging-forwarder-aggregator/

[giscus[bot]]

docs/logging-forwarder-aggregator/

How to deploy logging collection, aggregation and distribution in our Raspberry Pi Kuberentes cluster. Deploy a forwarder/aggregator architecture using Fluentbit and Fluentd. Logs are routed to Elasticsearch and Loki, so log analysis can be done using Kibana and Grafana.

https://picluster.ricsanfre.com/docs/logging-forwarder-aggregator/

[i5Js]

Great tutorial.

Quick question, I don't want to get this accesible from WAN. Should I need to create the certificate?

Thanks.

[ricsanfre]

Thanks @i5Js,

It is up to you. You can decide not to secure communications between fluentbit collectors and fluentd aggregator using TLS or even not to use authentication (shared-key). I decided to do so I can have external fluent-bit collectors, running on external hosts (in my case 'gateway' and 'pimaster' nodes), pushing logs into fluentd running within the cluster.

If you do not want to use secure connections, you only need to change a couple of lines in both fluentbit and fluentd configuration.

Regards
Ricardo

[i5Js]

You have a typo when you create the secret:

metadata: but you have metadata

[i5Js]

Another typo:

volumes:

• name: fluentd-tls
  secret:
  secretName: fluentd-tls
  ---> - name: etcfluentd-template <---- That is duplicated, not needed
• name: etcfluentd-template

[ricsanfre]

Thanks Javi,

Typos fixed

[i5Js]

Probably I will have another typo, but I'm getting this error in the fluentd
2024-03-28 14:08:19 +0000 [warn]: #0 [out_es] Could not communicate to Elasticsearch, resetting connection and trying again. [400] {"error":{"root_cause":[{"type":"x_content_parse_exception","reason":"[1:12] [policy] unknown field [phases]"}],"type":"x_content_parse_exception","reason":"[1:12] [policy] unknown field [phases]"},"status":400}

[i5Js]

Running out of ideas... Google doesn't help.

[ricsanfre]

Which ES version are you using?

[i5Js]

Same than your blog.