extractors/pipeline make regex-replace capable to insert new-line/carriage-return

[jalogisch]

Expected Behavior

When using regex-replace you should be able to create a multiline message by replacing characters with a new line.

Look into this example:

Inserted Content of one field:

INFO_1___INFO_2___INFO_3___INFO_4_

Should become the following after replacement of __ to \n or similar.

INFO_1_
INFO_2_
INFO_3_
INFO_4_

The current regex-replace only allow something like, that is not the intention of this replacement.

INFO_1_\nINFO_2_\nINFO_3_\nINFO_4_

Context

When receiving data from sources where formatting is limited but the information should be more readable for humans formatting options should work in a very basic way. Creating single fields would only make sense when sub-fields are possible.
This is simple to make messages more readable to humans.

Your Environment

• Graylog Version: 2.4.6

[elcamlost]

I put my two cents at the issue.

I'm using graylog to feed it with multiline messages from php-fpm slow logs. And it's really hard to read messages like these

[0x00007fd2b481bb20] __ConvertProperties() /var/www/example.com/www/bitrix/modules/catalog/general/discount.php:3642\n[0x00007fd2b481b9c0] SetDiscountProductCache() /var/www/example.com/www/bitrix/modules/iblock/lib/component/base.php:3412\n[0x00007fd2b481b7a0] getIblockOffers() /var/www/example.com/www/bitrix/modules/iblock/lib/component/base.php:3141\n[0x00007fd2b481b6b0] processOffers() /var/www/example.com/www/bitrix/modules/iblock/lib/component/base.php:3589\n[0x00007fd2b481b640] prepareData() /var/www/example.com/www/bitrix/modules/iblock/lib/component/base.php:3541\n[0x00007fd2b481b5c0] processResultData() /var/www/example.com/www/bitrix/components/bitrix/catalog.section/class.php:86\n[0x00007fd2b481b550] processResultData() /var/www/example.com/www/bitrix/modules/iblock/lib/component/base.php:3946\n[0x00007fd2b481b4e0] loadData() /var/www/example.com/www/bitrix/modules/iblock/lib/component/elementlist.php:1058\n[0x00007fd2b481b470] loadData() /var/www/example.com/www/bitrix/modules/iblock/lib/component/base.php:3935\n[0x00007fd2b481b3f0] initialLoadAction() /var/www/example.com/www/bitrix/components/bitrix/catalog.section/class.php:323\n[0x00007fd2b481b370] initialLoadAction() /var/www/example.com/www/bitrix/modules/iblock/lib/component/base.php:4120\n[0x00007fd2b481b300] call_user_func() /var/www/example.com/www/bitrix/modules/iblock/lib/component/base.php:4120\n[0x00007fd2b481b260] doAction() /var/www/example.com/www/bitrix/modules/iblock/lib/component/base.php:4138\n[0x00007fd2b481b1d0] executeComponent() /var/www/example.com/www/bitrix/modules/main/classes/general/component.php:585\n[0x00007fd2b481b0d0] includeComponent() /var/www/example.com/www/bitrix/modules/main/classes/general/main.php:1035\n[0x00007fd2b481aef0] IncludeComponent() /var/www/example.com/www/local/templates/.default/components/bitrix/catalog/.default/element.php:304\n[0x00007fd2b481ad70] [INCLUDE_OR_EVAL]() /var/www/example.com/www/bitrix/modules/main/classes/general/component_template.php:714\n[0x00007fd2b481ac00] __IncludePHPTemplate() /var/www/example.com/www/bitrix/modules/main/classes/general/component_template.php:806\n[0x00007fd2b481aad0] IncludeTemplate() /var/www/example.com/www/bitrix/modules/main/classes/general/component.php:681\n[0x00007fd2b481aa10] showComponentTemplate() /var/www/example.com/www/bitrix/modules/main/classes/general/component.php:629

And there is no way to configure php-fpm produce logs in json or any other format, so extracting this ugly message into normal stacktrace like this

[0x00007fd2b481bb20] __ConvertProperties() /var/www/example.com/www/bitrix/modules/catalog/general/discount.php:3642
[0x00007fd2b481b9c0] SetDiscountProductCache() /var/www/example.com/www/bitrix/modules/iblock/lib/component/base.php:3412
[0x00007fd2b481b7a0] getIblockOffers() /var/www/example.com/www/bitrix/modules/iblock/lib/component/base.php:3141
[0x00007fd2b481b6b0] processOffers() /var/www/example.com/www/bitrix/modules/iblock/lib/component/base.php:3589
[0x00007fd2b481b640] prepareData() /var/www/example.com/www/bitrix/modules/iblock/lib/component/base.php:3541
[0x00007fd2b481b5c0] processResultData() /var/www/example.com/www/bitrix/components/bitrix/catalog.section/class.php:86
[0x00007fd2b481b550] processResultData() /var/www/example.com/www/bitrix/modules/iblock/lib/component/base.php:3946
[0x00007fd2b481b4e0] loadData() /var/www/example.com/www/bitrix/modules/iblock/lib/component/elementlist.php:1058
[0x00007fd2b481b470] loadData() /var/www/example.com/www/bitrix/modules/iblock/lib/component/base.php:3935
[0x00007fd2b481b3f0] initialLoadAction() /var/www/example.com/www/bitrix/components/bitrix/catalog.section/class.php:323
[0x00007fd2b481b370] initialLoadAction() /var/www/example.com/www/bitrix/modules/iblock/lib/component/base.php:4120
[0x00007fd2b481b300] call_user_func() /var/www/example.com/www/bitrix/modules/iblock/lib/component/base.php:4120
[0x00007fd2b481b260] doAction() /var/www/example.com/www/bitrix/modules/iblock/lib/component/base.php:4138
[0x00007fd2b481b1d0] executeComponent() /var/www/example.com/www/bitrix/modules/main/classes/general/component.php:585
[0x00007fd2b481b0d0] includeComponent() /var/www/example.com/www/bitrix/modules/main/classes/general/main.php:1035
[0x00007fd2b481aef0] IncludeComponent() /var/www/example.com/www/local/templates/.default/components/bitrix/catalog/.default/element.php:304
[0x00007fd2b481ad70] [INCLUDE_OR_EVAL]() /var/www/example.com/www/bitrix/modules/main/classes/general/component_template.php:714
[0x00007fd2b481ac00] __IncludePHPTemplate() /var/www/example.com/www/bitrix/modules/main/classes/general/component_template.php:806
[0x00007fd2b481aad0] IncludeTemplate() /var/www/example.com/www/bitrix/modules/main/classes/general/component.php:681
[0x00007fd2b481aa10] showComponentTemplate() /var/www/example.com/www/bitrix/modules/main/classes/general/component.php:629

will be really usefull.

[kikoreis]

I was pointed here when posting at https://community.graylog.org/t/using-extractor-to-create-a-field-that-gets-displayed-with-newlines/7325/4 -- I posted a bookmarklet there to handle this client-side.

Would anyone be able to point me at the code where this could be handled so I can try and understand where the problem arises from?

[kikoreis]

Note that this was once possible, at least as per: #1956

[netoht]

It maybe may works for you: #5971 (comment)