Sampling connection-oriented weird-types that happen only once per connection

[ckreibich]

Sampling of connection-oriented weird-types that happen only once per connection (such as data_before_established) doesn't work well right now: you either get each of those weirds, or none of them. The logic to tweak here is in Reporter::PermitFlowWeird().

I believe that's a known problem, but figured I'd create a ticket in case it's not tracked. I also don't know whether 3.0 has changed anything in this regard, but if so, that'd be great to know.

[jsiwek]

The sampling state for connection-oriented weirds is stored per-connection, but what you expect/want would be for sampling state to be tracked globally per-weird-name?

If so, think it's not hard to add that as an option, but the current default behavior was mostly about being conservative: "people are less likely to miss redundant data we automatically throw out (or offer "compressed" stats for), and more likely to expect that we report all novel information by default".

[ckreibich]

Exactly — some level of global control would be ideal. I believe I'm missing some history here from the updates to earlier versions of the sampling code (2.5, iirc), so others may have additional context.

The problem with the current approach is that in environments where data_before_established comes up a lot it effectively bypasses sampling and blows up the weird-log.

[rsmmr]

I'm planning to add an additional config table that puts weirds on a list of samplings that are to be done globally, in contrast to their standard granularity (conn/flow).

[rsmmr]

@ckreibich how does #1154 look?

[ckreibich]

@rsmmr nice, yeah. Thanks for putting this together. I was wondering whether it would make sense to put together "candidate weirds" for use in the global list, but it seems that's really up to the user's sampling needs. So I think we're all set.