Add a pattern exclusion option to Logcollector

[vikman90]

Logcollector allows using wildcarded patterns to set up file monitoring. These patterns are based on globbing pathnames (glob(7)). However, defining patterns to exclude files is not easy.

Add an exclusion option to the localfile stanza, like this:

<localfile>
  <log_format>syslog</log_format>
  <location>/var/log/*.log</location>
  <exclude>/var/log/*.log.*</exclude>
</localfile>

Actually, Logcollector will monitor the files defined by <location> - <exclude>.

[TJOSERAFAEL]

Hi @vikman90,

The implementation is completed in this branch https://github.com/wazuh/wazuh/tree/3.9-logcollector-exclude. For now this will work only for globs patterns, not single files or an array of single files because it its not specified to do so yet, we need to clarify it.

Take this configuration for example:

 <localfile>
    <log_format>syslog</log_format>
    <location>/home/rafa/Desktop/test-log/*.txt</location>
    <exclude>/home/rafa/Desktop/test-log/a*</exclude>
  </localfile>

It will exclude all files beginning with an a. See the following log:

2019/03/14 17:28:48 ossec-logcollector[23687] main.c:179 at main(): DEBUG: Starting ...
2019/03/14 17:28:48 ossec-logcollector[23687] mq_op.c:47 at StartMQ(): DEBUG: (unix_domain) Maximum send buffer set to: '212992'.
2019/03/14 17:28:48 ossec-logcollector[23687] logcollector.c:984 at check_pattern_expand(): INFO: (1957): New file that matches the '/home/rafa/Desktop/test-log/*.txt' pattern: '/home/rafa/Desktop/test-log/a1.txt'.
2019/03/14 17:28:48 ossec-logcollector[23687] logcollector.c:996 at check_pattern_expand(): DEBUG: (1961): Files being monitored: 1/1000.
2019/03/14 17:28:48 ossec-logcollector[23687] logcollector.c:881 at set_read(): INFO: (1950): Analyzing file: '/home/rafa/Desktop/test-log/a1.txt'.
2019/03/14 17:28:48 ossec-logcollector[23687] logcollector.c:900 at set_read(): DEBUG: Socket target for '/home/rafa/Desktop/test-log/a1.txt' -> agent
2019/03/14 17:28:48 ossec-logcollector[23687] logcollector.c:984 at check_pattern_expand(): INFO: (1957): New file that matches the '/home/rafa/Desktop/test-log/*.txt' pattern: '/home/rafa/Desktop/test-log/a2.txt'.
2019/03/14 17:28:48 ossec-logcollector[23687] logcollector.c:996 at check_pattern_expand(): DEBUG: (1961): Files being monitored: 2/1000.
2019/03/14 17:28:48 ossec-logcollector[23687] logcollector.c:984 at check_pattern_expand(): INFO: (1957): New file that matches the '/home/rafa/Desktop/test-log/*.txt' pattern: '/home/rafa/Desktop/test-log/a3.txt'.
2019/03/14 17:28:48 ossec-logcollector[23687] logcollector.c:996 at check_pattern_expand(): DEBUG: (1961): Files being monitored: 3/1000.
2019/03/14 17:28:48 ossec-logcollector[23687] logcollector.c:984 at check_pattern_expand(): INFO: (1957): New file that matches the '/home/rafa/Desktop/test-log/*.txt' pattern: '/home/rafa/Desktop/test-log/n2.txt'.
2019/03/14 17:28:48 ossec-logcollector[23687] logcollector.c:996 at check_pattern_expand(): DEBUG: (1961): Files being monitored: 4/1000.
2019/03/14 17:28:48 ossec-logcollector[23687] logcollector.c:984 at check_pattern_expand(): INFO: (1957): New file that matches the '/home/rafa/Desktop/test-log/*.txt' pattern: '/home/rafa/Desktop/test-log/test1.txt'.
2019/03/14 17:28:48 ossec-logcollector[23687] logcollector.c:996 at check_pattern_expand(): DEBUG: (1961): Files being monitored: 5/1000.
2019/03/14 17:28:48 ossec-logcollector[23687] logcollector.c:984 at check_pattern_expand(): INFO: (1957): New file that matches the '/home/rafa/Desktop/test-log/*.txt' pattern: '/home/rafa/Desktop/test-log/test2.txt'.
2019/03/14 17:28:48 ossec-logcollector[23687] logcollector.c:996 at check_pattern_expand(): DEBUG: (1961): Files being monitored: 6/1000.
2019/03/14 17:28:48 ossec-logcollector[23687] logcollector.c:1068 at check_pattern_expand_excluded(): DEBUG: (1964): File excluded: '/home/rafa/Desktop/test-log/a1.txt'.
2019/03/14 17:28:48 ossec-logcollector[23687] logcollector.c:1069 at check_pattern_expand_excluded(): DEBUG: (1961): Files being monitored: 5/1000.
2019/03/14 17:28:48 ossec-logcollector[23687] logcollector.c:1068 at check_pattern_expand_excluded(): DEBUG: (1964): File excluded: '/home/rafa/Desktop/test-log/a2.txt'.
2019/03/14 17:28:48 ossec-logcollector[23687] logcollector.c:1069 at check_pattern_expand_excluded(): DEBUG: (1961): Files being monitored: 4/1000.
2019/03/14 17:28:48 ossec-logcollector[23687] logcollector.c:1068 at check_pattern_expand_excluded(): DEBUG: (1964): File excluded: '/home/rafa/Desktop/test-log/a3.txt'.
2019/03/14 17:28:48 ossec-logcollector[23687] logcollector.c:1069 at check_pattern_expand_excluded(): DEBUG: (1961): Files being monitored: 3/1000.
2019/03/14 17:28:48 ossec-logcollector[23687] logcollector.c:125 at LogCollectorStart(): DEBUG: Entering LogCollectorStart().

[TJOSERAFAEL]

Hi @vikman90,

there was an issue with the implementation, every time the main thread reloaded the wildcards, the excluded files appeared in the info message as new files.

With the following configuration:

  <localfile>
    <log_format>syslog</log_format>
    <location>/home/rafa/Desktop/test-log/*</location>
    <exclude>/home/rafa/Desktop/test-log/e*</exclude>
  </localfile>

See the following log bellow:

2019/03/22 16:09:19 ossec-logcollector: INFO: (1957): New file that matches the '/home/rafa/Desktop/test-log/*' pattern: '/home/rafa/Desktop/test-log/a2.txt'.
2019/03/22 16:09:19 ossec-logcollector: INFO: (1957): New file that matches the '/home/rafa/Desktop/test-log/*' pattern: '/home/rafa/Desktop/test-log/a3.txt'.
2019/03/22 16:09:19 ossec-logcollector: INFO: (1957): New file that matches the '/home/rafa/Desktop/test-log/*' pattern: '/home/rafa/Desktop/test-log/a4.txt'.
2019/03/22 16:09:19 ossec-logcollector: INFO: (1957): New file that matches the '/home/rafa/Desktop/test-log/*' pattern: '/home/rafa/Desktop/test-log/e1.txt'.
2019/03/22 16:09:19 ossec-logcollector: INFO: (1957): New file that matches the '/home/rafa/Desktop/test-log/*' pattern: '/home/rafa/Desktop/test-log/e2.txt'.
2019/03/22 16:09:19 ossec-logcollector: INFO: (1957): New file that matches the '/home/rafa/Desktop/test-log/*' pattern: '/home/rafa/Desktop/test-log/e3.txt'.
2019/03/22 16:09:19 ossec-logcollector: INFO: (1957): New file that matches the '/home/rafa/Desktop/test-log/*' pattern: '/home/rafa/Desktop/test-log/test_file'.
2019/03/22 16:09:19 ossec-logcollector: INFO: (1957): New file that matches the '/home/rafa/Desktop/test-log/*' pattern: '/home/rafa/Desktop/test-log/test_file1'.
2019/03/22 16:09:19 ossec-logcollector: INFO: (1957): New file that matches the '/home/rafa/Desktop/test-log/*' pattern: '/home/rafa/Desktop/test-log/test_file2'.
2019/03/22 16:09:19 ossec-logcollector: INFO: (1957): New file that matches the '/home/rafa/Desktop/test-log/*' pattern: '/home/rafa/Desktop/test-log/test_file3'.
2019/03/22 16:09:19 ossec-logcollector: INFO: (1950): Analyzing file: '/home/rafa/Desktop/test2.txt'.
2019/03/22 16:09:19 ossec-logcollector: INFO: Started (pid: 10551).
2019/03/22 16:10:23 ossec-logcollector: INFO: (1957): New file that matches the '/home/rafa/Desktop/test-log/*' pattern: '/home/rafa/Desktop/test-log/e1.txt'.
2019/03/22 16:10:23 ossec-logcollector: INFO: (1957): New file that matches the '/home/rafa/Desktop/test-log/*' pattern: '/home/rafa/Desktop/test-log/e2.txt'.
2019/03/22 16:10:23 ossec-logcollector: INFO: (1957): New file that matches the '/home/rafa/Desktop/test-log/*' pattern: '/home/rafa/Desktop/test-log/e3.txt'.

This is now fixed in this commit 4b0a62e

Regards.

[TJOSERAFAEL]

Hi @vikman90 ,

the check_pattern_expand_exclude function has now been ported for Windows operating systems. Commit: 9637420

Regards.

[TJOSERAFAEL]

Merged at #2929