Regexp for 'error' returning false positives

[Boruch-Baum]

In debian, for kernel.log, using lnav version 0.8.5-2, lnav is reporting the following lines as errors, when scrolling using e/E, and when viewing the histogram report i:

1] A power-saving feature for laptops is to spin down mechanical disk drives, so when they need to be accessed, they are spun up again and the options passed to the "mount" command are logged:

EXT4-fs (sda3): re-mounted. Opts: errors=remount-ro,commit=600

2] When apparmor is being run in 'audit' mode, its log message includes the word "error":

audit: type=1400 audit(1591232817.717:23): apparmor="DENIED" operation="change_onexec" info="label not found" error=-2 profile="unconfined" 

[nodiscc]

The following message in /var/log/syslog also returns a false positive at the warning level:

Jul 16 21:22:18 dev1 ansible-command: Invoked with removes=/etc/apache2/mods-enabled/cgi.load _raw_params=a2dismod cgi warn=True _uses_shell=False stdin_add_newline=True strip_empty_ends=True argv=None chdir=None executable=None

I have many other such false positives at the error or warning level, for various services (all my logs are aggregated in /var/log/syslog).

Is there a way to configure lnav to not interpret these terms as expressions as warning/error?

[nodiscc]

@tstack do you have any advice about this? I think I need to exclude lines matching .*ansible.*warn=True from matxhing the warning filter in syslog_log format definition... This could become expensive if every line has to be matched against two conditions.

Do I have to edit this https://github.com/tstack/lnav/blob/master/src/formats/syslog_log.json#L18 ? Or is there another way?

[Boruch-Baum]

@tstack: You've been giving us the cold shoulder silent treatment on this for over four months now. Can we have just at least an acknowldgement? One possible way to address this issue in a user-extensible way would be to make the regexp's discoverable / modifiable as a run-time configuration option, either via a config file or environmental variable.

…

On 2020-09-26 11:49, nodiscc wrote: ***@***.*** do you have any advice about this? I think I need to exclude lines matching .*ansible.*warn=True from matxhing the warning filter in syslog_log format definition... This could become expensive if every line has to be matched against two conditions. Do I have to edit this [2]https://github.com/tstack/lnav/blob/master/src/formats/syslog_log.js on#L18 ? Or is there another way? — You are receiving this because you authored the thread. Reply to this email directly, [3]view it on GitHub, or [4]unsubscribe. References 1. https://github.com/tstack 2. https://github.com/tstack/lnav/blob/master/src/formats/syslog_log.json#L18 3. #733 (comment) 4. https://github.com/notifications/unsubscribe-auth/AAOE3KFTBTPOKUFO76T4HGTSHYZT3ANCNFSM4NSFO7IA

-- hkp://keys.gnupg.net CA45 09B5 5351 7C11 A9D1 7286 0036 9E45 1595 8BC0

[tstack]

One possible way to address this issue in a user-extensible way would be to make the regexp's discoverable / modifiable as a run-time configuration option, either via a config file or environmental variable.

You can override log format configuration properties by adding a JSON file into a subdirectory of the formats directory (like ~/.lnav/formats/installed/). For example, to not highlight any errors in syslog files, you could put the following JSON in ~/.lnav/formats/installed/syslog-override.json:

{
    "syslog_log": {
	"level": {
	    "error": ""
	}
    }
}

This works because of the way the configuration is loaded. lnav will process the built-in formats first, followed by files in the formats directories. As the files are read in, the internal state is updated. So, the contents of one file can override another.

You've been giving us the cold shoulder silent treatment on this for over four months now.

Sorry, but this is a case of "patches/pull-requests are welcome". It's not really a priority for me and with the 2020 hellscape being what it is, I've only got so much time to give to lnav.

[Boruch-Baum]

On 2020-10-12 12:01, Tim Stack wrote: You can override log format configuration properties by adding a JSON ...

Thanks. That right away becomes very useful.

Sorry, but this is a case of "patches/pull-requests are welcome".

If I come up with an improved regexp, I'll make that submission.

…

-- hkp://keys.gnupg.net CA45 09B5 5351 7C11 A9D1 7286 0036 9E45 1595 8BC0

[nodiscc]

I will look into this soon, and submit a pull request to provide an example in the documentation.
Edit: the regex to be improved is at https://github.com/tstack/lnav/blob/master/src/formats/syslog_log.json#L17

[nodiscc]

The log format has changed since my initial report so I cannot reproduce my exact issue (false positive on warn=True), but I have experimented with messages provided by @Boruch-Baum :

$ echo 'EXT4-fs (sda3): re-mounted. Opts: errors=remount-ro,commit=600' | logger -t testlnav

The message is detected as an error by lnav in the default configuration

After copying the default regex for syslog errors from https://github.com/tstack/lnav/blob/master/src/formats/syslog_log.json#L17 to /root/.lnav/formats/installed/syslog-override.json, and editing it to not match errors=

{
    "syslog_log": {
        "level": {
            "error": "(?:(?:(?<![a-zA-Z]))(?:(?i)error(?:s)?(?!=))(?:(?![a-zA-Z]))|failed|failure)"
        }
    }
}

The message is still detected as an error. Running sudo lnav -d debug.log shows that the syslog-override.json format file is correctly loaded:

2023-01-23T14:36:40.457 I log_format_loader.cc:819 loading formats from file: /root/.lnav/formats/installed/syslog-override.json
2023-01-23T14:36:40.457 I log_format_loader.cc:901   found format: syslog_log

So either my regexp is not correct (I just added he negative lookahead (?!=) after error(?:s)? and https://regexr.com/ shows that it is a correct way to exclude errors= from matches - you must set it to PRCE mode in the top right corner), or something else is wrong with @tstack's explanation (lnav will process the built-in formats first, followed by files in the formats directories.)

I will keep looking but any help is appreciated.

[nodiscc]

Update: the updated format/override file actually works correctly. My issue came from the fact that I was not consistently using sudo in my tests, so /root/.lnav/formats/installed/syslog-override.json was not always being loaded.

Is there a directory where I could place the override file, that would make it load the format for all users? lnav is installed from the debian package.

Edit: found it

2023-01-23T14:55:58.954 I log_format_loader.cc:881 loading formats from path: /etc/lnav/formats/*/*.json
2023-01-23T14:55:58.954 I log_format_loader.cc:881 loading formats from path: /etc/lnav/formats/*/*.json

deploy@demo1:~$ sudo mkdir -p /etc/lnav/formats/installed/
deploy@demo1:~$ sudo mv /root/.lnav/formats/installed/syslog-override.json /etc/lnav/formats/installed/
deploy@demo1:~$ lnav /var/log/syslog
# no false positive anymore
deploy@demo1:~$ sudo lnav /var/log/syslog
# no false positive anymore

I will provide a PR to the docs.

[nodiscc]

PR #1113

[Boruch-Baum]

Thanks for working on this!

…

On 2023-01-23 05:54, nodiscc wrote: Update: the updated format/override file actually works correctly. My issue came from the fact that I was not consistently using sudo in my tests, so /root/.lnav/formats/installed/syslog-override.json was not always being loaded. Is there a directory where I could place the override file, that would make it load the format for all users? — Reply to this email directly, [1]view it on GitHub, or [2]unsubscribe. You are receiving this because you were mentioned. Message ID: ***@***.***> References 1. #733 (comment) 2. https://github.com/notifications/unsubscribe-auth/AAOE3KHLAPQRWROHNKW4BGTWT2ERNANCNFSM4NSFO7IA

-- hkp://keys.gnupg.net CA45 09B5 5351 7C11 A9D1 7286 0036 9E45 1595 8BC0