Summarize operator with pluggable aggregation functions

[dominiklohmann]

This rewrites the summarize pipeline operator to have pluggable aggregation functions through the new aggregation_function_plugin plugin type, which each implement a simple virtual aggregation_function interface for incremental aggregation.

In addition, this converts both the summarize plugin and the existing aggregation functions to native plugins, making them always available when using VAST.

The configuration for the aggregation also changed, allowing users to specify output field names explicitly and aggregating multiple columns into one. This is the new configuration that I primarily used for testing with suricata.flow events:

summarize:
  group-by:
    - timestamp
    - proto
    - event_type
  time-resolution: 1 hour
  aggregate:
    timestamp_min:
      min: timestamp
    timestamp_max:
      max: timestamp
    pkts_toserver: sum
    pkts_toclient: sum
    bytes_toserver: sum
    bytes_toclient: sum
    start: min
    end: max
    alerted: any
    ips:
      distinct:
        - src_ip
        - dest_ip

📝 Checklist

• All user-facing changes have changelog entries.
• The changes are reflected on vast.io, if necessary.
  • Document summarize changes.
  • Document aggregation_function_plugin plugin type.
• The PR description contains instructions for the reviewer, if necessary.

🎯 Review Instructions

I recommend going file-by-file, and testing locally with compaction. The following order is ideal:

1. Review the API of aggregation_function_plugin.
2. Review the API of aggregation_function
3. Review the summarize pipeline operator implementation.
4. Review the summarize configuration, and how it's bound to a schema to form an aggregation.
5. Review the individual aggregation function plugins.

This PR also includes the work from #2391, which I've closed accordingly.

[mavam]

Just wondering: should sample take an argument n that specifies the sample size? Right now, we have n = 1 hard-coded. To me, a sample function suggests that it's configurable in its desired output length.

[dominiklohmann]

Just wondering: should sample take an argument n that specifies the sample size? Right now, we have n = 1 hard-coded. To me, a sample function suggests that it's configurable in its desired output length.

I don't think so to be honest, at least not for the foreseeable future while we still use the YAML based configuration. It's hard to fit additional parametrization for the aggregations functions in it, and I'd want to wait with doing so until we have an actual use case or request for it.

[dispanser]

Looked over this together, minor comments inline