Observability audit

[binarylogic]

We've come across a few issues where logging was not present and failed to help the user understand what Vector was doing:

1. Expose unavailable Kafka source broker #1752
2. feat(sinks): Initial logdna sink implementation #1668 (comment)
3. kafka sink is silently failing #1842

I'd like to audit our components to ensure that observability as a whole is adequate. Trying to brute force this in a short amount of time isn't a good idea, so I'd prefer to do one component at a time spread across our team.

Tasks

• Install this GH action and add observability / logging as a checklist item.

Global

• Rate limiting logs (Automatically adjust backoff values RFC #1858)

Sources

• docker source (pending)
• file source (pending)
• journald source (pending)
• kafka source (pending)
• logplex source (pending)
• prometheus source (pending)
• socket source (pending)
• splunk_hec source (pending)
• statsd source (pending)
• stdin source (pending)
• syslog source (pending)
• vector source

Transforms

• add_fields transform (pending)
• add_tags transform (pending)
• ansi_stripper transform (pending)
• aws_ec2_metadata transform (pending)
• coercer transform (pending)
• concat transform (pending)
• field_filter transform (pending)
• geoip transform (pending)
• grok_parser transform (pending)
• json_parser transform (pending)
• log_to_metric transform (pending)
• logfmt_parser transform (pending)
• lua transform (pending)
• merge transform (pending)
• regex_parser transform (pending)
• remove_fields transform (pending)
• remove_tags transform (pending)
• sampler transform (pending)
• split transform (pending)
• tokenizer transform

Sinks

• aws_cloudwatch_logs sink (pending)
• aws_cloudwatch_metrics sink (pending)
• aws_kinesis_firehose sink (pending)
• aws_kinesis_streams sink (pending)
• aws_s3 sink (pending)
• blackhole sink (pending)
• clickhouse sink (pending)
• console sink (pending)
• datadog_metrics sink (pending)
• elasticsearch sink (pending)
• file sink (pending)
• gcp_pubsub sink (pending)
• gcp_stackdriver_logging sink (pending)
• http sink (pending)
• humio_logs sink (pending)
• kafka sink (pending)
• logdna sink (pending)
• loki sink (pending)
• new_relic_logs sink (pending)
• prometheus sink (pending)
• sematext sink (pending)
• socket sink (pending)
• splunk_hec sink (pending)
• statsd sink (pending)
• vector sink (pending)

[binarylogic]

This is blocked by #1761 since it probably makes more sense to perform this after that is complete. It's likely we'll want to change how we're observing Vector in general and incorporate metrics into this process.

[LucioFranco]

I'll add another one I found today, we should ensure that healthchecks get a span that associates it with the sink that its health checking.

Feb 18 15:55:19.677 ERROR vector::topology::builder: Healthcheck: Failed Reason: unknown API key - check your credentials

This above doesn't tell me which sink actually failed and may be confusing.

[binarylogic]

Closing in favor of #3192.