Add fluentd configuration

[mofirouz]

We deploy CockroachDB on Google Cloud Platform and have a custom fluentd configuration setup so logs have the correct severity level, timestamp and more in Stackdriver Logging.

This PR contains the relevant parts of the fluentd configuration that is responsible for parsing the CockroachDB log output. This configmap does not depend on any cloud provider.

If you'd like, I can provide a GCP-specific configmap + daemonset to run fluentd.

[cockroach-teamcity]

This change is 

[cockroach-teamcity]

This change is 

[a-robinson]

Sorry for the delay on this, @mofirouz.

[a-robinson]

Would you mind adding some comments to the top of the file explaining the purpose of this file and instructions for how to use it?

In particular, it seems worth calling out which parts are the main contributions of this file and referencing the source of the rest of it in case this falls out of date over time.

[mofirouz]

Sure - Most of it is actually written from scratch since the examples online are very specific and not general purpose :)

[a-robinson]

Can you explain this filter stanza for me? If it drops all logs that don't match the provided format, then what's the point of the <match reform.**> stanza below? And why does this stanza match on just reform.** while the next one matches more specifically on reform.var.log.containers.cockroachdb**?

More generally, a comment on each stanza explaining its purpose would be great for future maintainability.

[mofirouz]

The filters are not exclusively ran:

• so the first <filter reform.**> is the default case matching on all log records. This is needed for all other containers running that are not CockroachDB.
• the second <filter reform.**> runs on all log lines that match the regex inside of it. One can add extra patterns (say for custom applications there).

Each filter will republish the log record into fluentd queue so it can be picked up by other filters, hence why we change the tag name to parsed after we are done with it in the second filter.

Lastly, we make sure to only Match once on log records from the CockroachDB container so that we update the severity text from E to error etc (normalising the severity text format).

[mofirouz]

Hi @a-robinson - apologies for the delay - this is still on my radar and will update the PR soon.

[mofirouz]

Hi @a-robinson - apologies for the delay - this is still on my radar and will update the PR soon.

[a-robinson]

Hi @mofirouz, any update on this?

[a-robinson]

Hi @mofirouz, any update on this?

[mofirouz]

@a-robinson I've added some general comments to this PR - let me know if you'd like more info.

P.s. (sorry closed by mistake)

[mofirouz]

@a-robinson I've added some general comments to this PR - let me know if you'd like more info.

P.s. (sorry closed by mistake)

[a-robinson]

Thanks for the comments! This looks good other than the one question about fractional seconds.