Add export --internal to access metrics

[jachris]

This PR makes it so that we store pipeline metrics in the database. They are marked as "internal", which means that they are not exported by default. Instead, we can use export --internal to do so. The implementation uses a #internal meta extractor as a marker, but this extractor can only be accessed from code and not user-written pipelines.

It also fixes a bug where export --live | where <filter> discarded the filter.

Example

export --internal |
where source == true |
summarize bytes=sum(output.approx_bytes) by timestamp resolution 1s |
sort timestamp desc

{
  "timestamp": "2023-11-10T11:15:54.000000",
  "bytes": 79927223
}
{
  "timestamp": "2023-11-10T11:15:53.000000",
  "bytes": 81788928
}
{
  "timestamp": "2023-11-10T11:15:52.000000",
  "bytes": 80740352
}
{
  "timestamp": "2023-11-10T11:15:51.000000",
  "bytes": 75497472
}
{
  "timestamp": "2023-11-10T11:15:50.000000",
  "bytes": 75497472
}
{
  "timestamp": "2023-11-10T11:15:49.000000",
  "bytes": 76546048
}
{
  "timestamp": "2023-11-10T11:15:48.000000",
  "bytes": 78643200
}

Tasklist

• Where to tick import source in pipeline manager?
• Write docs for metrics format.
• Add internal flag to event data
• export --live --internal does not work yet (to be done later)
• Create issue for follow-up PR

[dominiklohmann]

👀

[jachris]

I still have to write the docs for the actual metrics events, but besides that, this should be ready for review.

[dominiklohmann]

Two initial comments on the docs to start a discussion; still reviewing the code changes.