Stream rules using inverted 'field presense' do not work

[urban-moniker]

Expected Behavior

A stream rule that contains an inverted 'field presence' to prevent messages with fields present in the message should prevent a message with that field matching the stream.

Current Behavior

Messages incorrectly match the stream. If you test against one of these messages it reports that it would not match, but it does (as we are testing from a message that has matched the stream).

Steps to Reproduce (for bugs)

1. Create a stream rule that matches if a field is not present.
2. Messages with that field present still match the stream

Context

Trying to prevent certain messages matching a stream,

We were able to sidestep the issue by changing the rule to 'field must not contain: true' as it is a boolean value and should never be false.

Your Environment

• Graylog Version: 3.0
• Operating System: Ubuntu 18.04

[dennisoelkers]

I just reproduced this issue and the inverted field presence rule works as expected. Are you doing anything with pipeline rules that could modify the messages?

[urban-moniker]

Hi,

The field is created by a pipeline rule. We are actually conditionally creating 3 new fields using a pipeline, if any of these fields are present then the message should not match the stream (for your context we are inspecting an IP address field and creating tags that denote whether it is an RFC address, a loop back address or some other relevant sub-net etc).

I did some testing this morning and have realized that our workaround is not apparently working either, I have messages that match all the criteria for the rule (5 in total) and are not being included.

I will do some additional testing to see if it is related to the number of rules\combination.

Thanks

[urban-moniker]

Ahh, thinking about what you said, is the issue that the message has already matched the stream before it has been manipulated by the pipeline? We have attached the pipeline that generates the new fields to a stream that matches all messages from the input that receives them, but the stream we are talking about is a different one.

In this scenario do I actually need to attach a pipeline directly to my 'filtered' stream that creates and removes messages directly? I assume this is the case so will close the error.