Keep only non-matched series in streaming aggregation

[Amper]

Is your feature request related to a problem? Please describe

At the moment regardless of matching in config streaming aggregation:

• keeps all input series altogether if keepInput=true
• drops all input series altogether if keepInput=false

But in typical case user need to keep non-matched input series and drop all matched input series.

Now this effect can only be achieved with two coordinated remoteWrite (see additional information of this issue).

Describe the solution you'd like

There are two options for implementation (without backward compatibility break):

• new flag streamAggr.dropMatched, that drops all matched series if flag streamAggr.keepInput is set
• new flag streamAggr.keepMismatched, that keep all matched series if flag streamAggr.keepInput isn't set

Update:
As described in this comment: based on user feedback it makes more sense to implement it with backward incompatible change and implement the following flow:

• if series did not match any rules - pass it to remote write
• if series did match some rule - pass aggregated results
  Setting keepInput=true in this case allows keeping full source data (both aggregated and original).

Describe alternatives you've considered

No response

Additional information

Discussion: https://victoriametrics.slack.com/archives/CGZF1H6L9/p1682204682016349

[oleg-tolmashov]

Also if using stream aggregation with some match filter without keepInput=true, it drops internal vmagent vm_ metrics, which is not a desired behaviour I suppose.

@Amper are there plans when it's going to be implemented?

[Amper]

Hi @oleg-tolmashov, i hope to do it in the next quarter. For now, you can use workaround with separate remoteWrite and dropping metrics with relabeling.

[zekker6]

Hi @oleg-tolmashov @Amper
We've discussed this issue last week and decided that it will make more sense to change existing behavior.
Based on user feedback, it makes more sense to keep metrics which did not match any streamAggr rules by default.

So the idea is to change default behavior to:

• if series did not match any rules - pass it to remote write
• if series did match some rule - pass aggregated results
  Setting keepInput=true in this case allows keeping full source data (both aggregated and original).

Also, I was going to implement this one next.

[Amper]

@zekker6 I got it, great!

[iamhritikkumar]

@zekker6 Do you have any updates on when it will be available in vmagent?

[valyala]

FYI, the commit 6265157 changes the default behavior for stream aggregation, so now non-matched input series are preserved in the output. The old behavior can be restored by passing -remoteWrite.streamAggr.dropInput command-line option to vmagent or by passing -streamAggr.dropInput command-line option to single-node VictoriaMetrics. In this case all the input series are dropped when stream aggregation is enabled.

This commit will be included in the next release. In the mean time it is possible to test the new stream aggregation behavior by building vmagent from this commit according to these docs and verifying that vmagent keeps input samples, which do not match any match filter in stream aggregation configs.

[hagen1778]

vmagent changes the default behavior for stream aggregation for matched series starting from v1.92.0. Closing the feature request as done.