add filecheck module

[ilyam8]

Fixes: netdata/netdata#9794

Filecheck collects files and directories statistics.

Files statistics:

• existence
• time since the last modification
• size

Directories statistics:

• existence
• time since the last modification
• number of files

---

It produces the following file check charts (chart dimension - filepath):

• File Existence in boolean
• File Time Since the Last Modification in seconds
• File Size in bytes

Dir check charts (chart dimension - dirpath):

• Dir Existence in boolean
• Dir Time Since the Last Modification in seconds
• Dir Number of Files in files

---

Example configuration:

jobs:
 - name: files_dirs_example
   files:
     include:
       - '/path/to/file1'
       - '/path/to/file2'
   dirs:
     include:
       - '/path/to/dir1'
       - '/path/to/dir2'

---

Limitations of the current version:

• file/dir path doesn't support any wildcards, should be an absolute path to a file/dir.
• stat call is used to gather metrics on every iteration for every file/dir.

We will improve this module later, the goal is to provide the functionality.

[codecov-commenter]

Codecov Report

Merging #445 into master will decrease coverage by 0.14%.
The diff coverage is 68.18%.

[Ferroin]

Two things come to mind that could probably be easily improved here:

• We should document whether the existence check verifies the inode type. IOW, if we are checking for the existence of a directory, do we actually ensure it's a directory before proceeding to the other checks? If so, it should be documented. If not, then we should almost certainly make it behave this way and then document that it behaves this way.
• It would be nice to be able to skip certain checks on a given job. Obviously this won't eliminate the need to call stat() in most cases, but it will help avoid people having extra charts they don't actually need.

[ilyam8]

@Ferroin thx for suggestions!

do we actually ensure it's a directory before proceeding to the other checks

yes, same goes for a file

It would be nice to be able to skip certain checks on a given job.

It is one stat call per item (file, dir).

but it will help avoid people having extra charts they don't actually need.

There are 3 charts for file checks and 3 charts for dir checks (files/dirs - dimensions). Doesnt look like a problem.

In general i would like to have some global filtering by netdata daemon instead of making everything optional in every collector.

[ilyam8]

but it will help avoid people having extra charts they don't actually need.

If there are any requests we will add this, np. But not in this PR.

[thiagoftsm]

I created the following structure to test:

bash-5.0$ ls -lh 
total 4.0K
-rw-r--r-- 3 thiago thiago    0 Sep 22 18:51 hard_link
drwxr-xr-x 2 thiago thiago 4.0K Sep 22 18:51 mydir
-rw-r--r-- 3 thiago thiago    0 Sep 22 18:51 myfile
lrwxrwxrwx 1 thiago thiago    5 Sep 22 18:51 soft_link -> mydir
-rw-r--r-- 3 thiago thiago    0 Sep 22 18:51 soft_link_file

I observed that when I changed myfile inserting content on it, the module made exactly what I was expecting:

BEGIN 'filecheck_files_dirs_example.file_existence' 9999721
SET 'file_/tmp/go_d_tests/myfile_exists' = 1
SET 'file_/tmp/go_d_tests/soft_link_file_exists' = 1
SET 'file_/tmp/go_d_tests/hard_link_exists' = 1
END

BEGIN 'filecheck_files_dirs_example.file_mtime_ago' 9999721
SET 'file_/tmp/go_d_tests/myfile_mtime_ago' = 8
SET 'file_/tmp/go_d_tests/soft_link_file_mtime_ago' = 8
SET 'file_/tmp/go_d_tests/hard_link_mtime_ago' = 8
END

BEGIN 'filecheck_files_dirs_example.file_size' 9999721
SET 'file_/tmp/go_d_tests/myfile_size_bytes' = 459202
SET 'file_/tmp/go_d_tests/soft_link_file_size_bytes' = 459202
SET 'file_/tmp/go_d_tests/hard_link_size_bytes' = 459202
END

BEGIN 'filecheck_files_dirs_example.dir_existence' 9999721
SET 'dir_/tmp/go_d_tests/mydir_exists' = 1
SET 'dir_/tmp/go_d_tests/soft_link_exists' = 1
END

BEGIN 'filecheck_files_dirs_example.dir_mtime_ago' 9999721
SET 'dir_/tmp/go_d_tests/mydir_mtime_ago' = 690
SET 'dir_/tmp/go_d_tests/soft_link_mtime_ago' = 690
END

BEGIN 'filecheck_files_dirs_example.dir_num_of_files' 9999721
SET 'dir_/tmp/go_d_tests/mydir_num_of_files' = 0
SET 'dir_/tmp/go_d_tests/soft_link_num_of_files' = 0
END

BEGIN 'netdata.execution_time_of_filecheck_files_dirs_example' 9999721
SET 'time' = 0
END

But, when I removed the file my_file I saw this content:

BEGIN 'filecheck_files_dirs_example.file_size' 9999990
SET 'file_/tmp/go_d_tests/myfile_size_bytes' = 
SET 'file_/tmp/go_d_tests/soft_link_file_size_bytes' = 1
SET 'file_/tmp/go_d_tests/hard_link_size_bytes' = 1
END

I think I am missing something, but I was expecting the dimension would receive mx[fileDimID(filepath, "exists")] = 0 that would avoid it to be printed, or we would send an additional information for Netdata saying that the dimension is not more present.

[ilyam8]

What is wrong here? No file - no size.

BEGIN 'filecheck_files_dirs_example.file_size' 9999990
SET 'file_/tmp/go_d_tests/myfile_size_bytes' = 
SET 'file_/tmp/go_d_tests/soft_link_file_size_bytes' = 1
SET 'file_/tmp/go_d_tests/hard_link_size_bytes' = 1
END

---

that would avoid it to be printed
or we would send an additional information for Netdata saying that the dimension is not more present.

I dont think so, it is ok to have - in the charts - it gives you information.

[thiagoftsm]

The PR is working as expected. I tested it using the reported structure and doing the following tests:

• First test: I set directories, files and links, and the configuration file had the correct specification for each one of these files.
• Second test: I changed the configuration file moving the files to directory section and vice versa. The collector identified the "mistake" and it did not move forward.

LGTM.

[ilyam8]

the configuration file moving the files to directory section and vice versa

All such scenarios (add/remove files/dirs during runtime) should be in tests, testing manually is not good. We will add them later

[thiagoftsm]

the configuration file moving the files to directory section and vice versa

All such scenarios (add/remove files/dirs during runtime) should be in tests, testing manually is not good. We will add them later

I agree, but I have to understand the whole collector, so the tests were necessary! 😄

[FMC17]

Hello, I'm trying use this plugin, but I think it isn't working properly.

My config, using:

cd /etc/netdata # Replace this path with your Netdata config directory
sudo ./edit-config go.d/filecheck.conf

Is:

# ------MODULE-CONFIGURATION--------
#  GLOBAL 
 update_every: 10
 autodetection_retry: 0
 priority: 70000
#
#
#  JOBS 
jobs:
 - name: fileMonitor
   files:
     include:
       - '/root/trees/soccer.pkl'
       - '/root/trees/basket.pkl'

Testing it with this code:

cd /usr/libexec/netdata/plugins.d/
sudo -u netdata -s

./go.d.plugin -d -m filecheck

The output is:

[ INFO  ] main[main] agent.go:106 instance is started
[ INFO  ] main[main] setup.go:39 loading config file
[ INFO  ] main[main] setup.go:47 looking for 'go.d.conf' in [/etc/netdata /usr/lib/netdata/conf.d]
[ INFO  ] main[main] setup.go:54 found '/etc/netdata/go.d.conf
[ INFO  ] main[main] setup.go:61 config successfully loaded
[ INFO  ] main[main] agent.go:110 using config: enabled 'true', default_run 'true', max_procs '0'
[ INFO  ] main[main] setup.go:66 loading modules
[ INFO  ] main[main] setup.go:85 enabled/registered modules: 1/49
[ INFO  ] main[main] setup.go:90 building discovery config
[ INFO  ] main[main] setup.go:116 looking for 'filecheck.conf' in [/etc/netdata/go.d /usr/lib/netdata/conf.d/go.d]
[ INFO  ] main[main] setup.go:123 found '/etc/netdata/go.d/filecheck.conf
[ INFO  ] main[main] setup.go:128 dummy/read/watch paths: 0/1/0
[ INFO  ] discovery[manager] manager.go:90 registered discoverers: [file discovery: [file reader]]
[ INFO  ] discovery[manager] manager.go:95 instance is started
[ INFO  ] run[manager] run.go:30 instance is started
[ INFO  ] build[manager] build.go:106 instance is started
[ INFO  ] discovery[file manager] discovery.go:71 instance is started
[ INFO  ] discovery[file reader] read.go:39 instance is started
[ INFO  ] discovery[file reader] read.go:40 instance is stopped
[ DEBUG ] run[manager] run.go:41 tick 0
[ DEBUG ] run[manager] run.go:41 tick 1
[ DEBUG ] run[manager] run.go:41 tick 2
[ DEBUG ] run[manager] run.go:41 tick 3
[ DEBUG ] run[manager] run.go:41 tick 4
[ DEBUG ] run[manager] run.go:41 tick 5
[ DEBUG ] run[manager] run.go:41 tick 6
[ DEBUG ] run[manager] run.go:41 tick 7
....
^C[ INFO  ] main[main] agent.go:191 received interrupt signal (2). Terminating...

It continue without any more information, and I can't show anything in my cloud dashboard.

What could be the problem?

Thanks!

[odyslam]

Hey @Fran17mc

Welcome to our little community :) One of the Agent engineers will come on this shortly. The interruption at the end is due to your action, right? It appears to be working normally.

[ilyam8]

~ solved in https://github.com/netdata/go.d.plugin/issues/473