enhancement(observability): Add events_in_total & events_out_total metrics

[ktff]

Ref. #6367

---

Source events_in_total (component)

• Emitted for each event that came from outside the source.
• May contain labels marking the origin like: file path, IP, etc.
  • If it's emitted with the label, then it shouldn't be emitted without it.
• This is basically equivalent to events_processed_total although there are some differences, notably generator source won't emit it.
• Topology does emit this once for all sources without incrementing the counter so to cover the cases like generator source.

• Add to sources in a follow up PR. enhancement(observability): Add events_in_total to sources #6758

Source events_out_total (topology)

• Emitted for each event coming out of the source.

---

Transform events_in_total (topology)

• Emitted for each event passed to the transform.

Transform events_out_total (topology)

• Emitted for each event coming out of the transform.

---

Sink events_in_total (topology)

• Emitted for each event passed to the sink.
• This differs from the original idea of emitting it on entry to buffer, which would be really useful but
  • requires switching spans which is costly to call for each event
  • there are edge cases when reusing buffers on reload whose fixes would also have performance cost
  • In theory this all is avoidable/doable but I think it deserve it's own issue since it could end up quite complex and/or not worthy of doing, plus it can easily build on top of this PR. (EDIT: Let's roll with current implementation until we decide to instrument the buffers, then we can decide to either change this metric, or add a new gauge metric)

Sink events_out_total (topology)

• Emitted for each event acked by the sink.

[ktff]

cc. @jszwedko @bruceg @FungusHumungus

[lucperkins]

I feel like these should be events_* for the sake of consistency

[StephenWakely]

I'm not much of an expert in this area. When I run Vector is there a place I can look to see these events being emitted? They don't seem to show up in vector top.

[bruceg]

LGTM as long as the metrics actually shows up (ref #6433 (comment))

[jszwedko]

@FungusHumungus the usual way I look at these is piping an internal_metrics source to a console sink. Something like:

[sources.metrics]
type = "internal_metrics"

[sinks.console]
type = "console"
inputs = ["metrics"]
encoding.code = "json"

That'll emit the metrics to stdout on an interval.

[jszwedko]

It would be good to update vector top to use these instead of the current events_processed_total metric, but I think that can be done as a later step.

[StephenWakely]

Yeah this works nicely.

[ktff]

It would be good to update vector top to use these instead of the current events_processed_total metric, but I think that can be done as a later step.

Yep, there are some more preparations that need to be done before that step.

I feel like these should be events_* for the sake of consistency

@lucperkins they are in the code, it's my comments here that are out of snyc, I'll update them.