Add tunnel fields to Suricata shaper #2370
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
#2369 describes a problem where missing NDJSON typing config for rarely-seen Suricata fields generated a user-facing warning. This PR adds the additional config to the Suricata shaper to cover these fields.
Our switchover from legacy
types.json
to the new shaping configs in the time since the user reported the issue actually changes the symptom, so before I show the fix having its impact, I'll take a step back and show how the symptom was looking as of Brim commit brimdata/zui@8569148 which pointed atzqd
commit 8ab4607. In that case, because of the change in #2309, the user no longer was shown a warning, but the additional fields were silently imported with inferred types. So for instance, notice how thetunnel
fields are shown in black font that's used for strings, rather than the blue that's used for theip
type fields.More importantly, if the user doesn't notice this subtle difference and tries to treat them like the IP addresses they otherwise appear to be, they get unexpected behavior, such as this failing CIDR match.
Of course, if they catch the problem and do the cast at query time, they can fix it. But this is what we want the shaper to do, hence the changes in this PR.
If I start from the same Brim commit brimdata/zui@8569148 and advance my
zq
pointer to 6613aba to use the shaper in this branch, then re-import, now we get the correct typing out of the gate.A couple additional details:
tunnel.src_port
andtunnel.dest_port
fields in the Suricata EVE JSON. However, since the original user's issue showed these being generated, I've included them here in the shaping config.alert
events, the fact the user was not being informed thesetunnel
fields were being imported with inferred types seems hazardous. I've opened separate issue brimsec/brim#1519 regarding this, as it seems like something we'll need to address in the wider context of shaper UX.Fixes #2369