data loss when pipeline rule throws an exception

[mbunkus]

This is the more general issue raised in #5405.

Expected behavior

Any message arriving in Graylog that is not explicitly discarded by the admin should end up in the database, no matter what.

Current behavior

If a Java exception is thrown during the execution of a rule in a pipeline, processing of that message cedes immediately and the message is lost. It is not stored in the database.

Possible Solution

I would argue that data must never be lost. Instead the current pipeline should be aborted, but message processing should continue somehow, at least end up in the database and the All messages stream.

Additionally the admin should be notified, both by email (there's a root_email attribute in the server.yml config file, after all) and by popup error messages (similar to e.g. when an input fails to start). At the moment the exception is reduced to a warning in the log file, and the fact that data loss occurred is not mentioned anywhere.

Steps to Reproduce (for bugs)

Please see #5405 for detailed instructions how to reproduce the issue. Instead of using the simulator, simply create the pipeline & rule as mentioned there, connect it to the All messages stream and create a message containing solely an IPv6 address, e.g. via logger 2001:db8::1

Your Environment

• Graylog Version: 2.4.6-1 (Ubuntu package from Graylog's repo)
• Elasticsearch Version: 5.6.13 (Ubuntu package from Elastico's repo)
• MongoDB Version: 4.0.4 (Ubuntu package from MongoDB.org's repo)
• Operating System: Ubuntu 18.04