fscrawler not indexing old files

[cadm-frank]

Hello,
we noticed that fscrawler does not index files with old timestamps, once fscrawler has finished the first run.
This happens e.g. when files are moved around by users on a Samba share.
Touching those files is not an option, as it would modify users' files.
Reindexing everything every couple of minutes is out as well, due to the amount of time required for a full index, which in our case is multiple days.

I think this is basically #136 and #419, but the problem is that the timestamps on a file or directory cannot give you the full information needed if that file needs to be indexed or not.
Here's an example:

[dfrank@nixos:/tmp/tmp.P09rB42uzI]$ mkdir test
[dfrank@nixos:/tmp/tmp.P09rB42uzI]$ touch test/mynewfile
[dfrank@nixos:/tmp/tmp.P09rB42uzI]$ LANG=C stat test
  File: test
  Size: 60              Blocks: 0          IO Block: 4096   directory
Device: 22h/34d Inode: 62          Links: 2
Access: (0755/drwxr-xr-x)  Uid: ( 1000/  dfrank)   Gid: (  100/   users)
Access: 2021-09-13 14:22:47.987975017 +0200
Modify: 2021-09-13 14:22:36.399085365 +0200
Change: 2021-09-13 14:22:36.399085365 +0200
 Birth: -
[dfrank@nixos:/tmp/tmp.P09rB42uzI]$ LANG=C stat test/mynewfile
  File: test/mynewfile
  Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: 22h/34d Inode: 63          Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/  dfrank)   Gid: (  100/   users)
Access: 2021-09-13 14:22:36.399085365 +0200
Modify: 2021-09-13 14:22:36.399085365 +0200
Change: 2021-09-13 14:22:36.399085365 +0200
 Birth: -
[dfrank@nixos:/tmp/tmp.P09rB42uzI]$ mv test test_has_been_moved
[dfrank@nixos:/tmp/tmp.P09rB42uzI]$ LANG=C stat test_has_been_moved
  File: test_has_been_moved
  Size: 60              Blocks: 0          IO Block: 4096   directory
Device: 22h/34d Inode: 62          Links: 2
Access: (0755/drwxr-xr-x)  Uid: ( 1000/  dfrank)   Gid: (  100/   users)
Access: 2021-09-13 14:23:11.534750814 +0200
Modify: 2021-09-13 14:22:36.399085365 +0200
Change: 2021-09-13 14:23:08.399780667 +0200
 Birth: -
[dfrank@nixos:/tmp/tmp.P09rB42uzI]$ LANG=C stat test_has_been_moved/mynewfile
  File: test_has_been_moved/mynewfile
  Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: 22h/34d Inode: 63          Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/  dfrank)   Gid: (  100/   users)
Access: 2021-09-13 14:22:36.399085365 +0200
Modify: 2021-09-13 14:22:36.399085365 +0200
Change: 2021-09-13 14:22:36.399085365 +0200
 Birth: -

Note that test_has_been_moved/mynewfile has the exact same timestamps as test/mynewfile, but still needs to be reindexed, even if it appears old.
So when the ctime of a directory changs, you'd have to force a full reindex of that directory and all of its subdirectories if you don't have a list of known files to compare to.
But forcing a reindex of a full directory on a ctime change is an issue, because ctime is also updated when a new file is created in the directory:

[dfrank@nixos:/tmp/tmp.P09rB42uzI]$ touch test_has_been_moved/test2
[dfrank@nixos:/tmp/tmp.P09rB42uzI]$ LANG=C stat test_has_been_moved
  File: test_has_been_moved
  Size: 80              Blocks: 0          IO Block: 4096   directory
Device: 22h/34d Inode: 62          Links: 2
Access: (0755/drwxr-xr-x)  Uid: ( 1000/  dfrank)   Gid: (  100/   users)
Access: 2021-09-13 14:23:11.534750814 +0200
Modify: 2021-09-13 14:34:13.952444045 +0200
Change: 2021-09-13 14:34:13.952444045 +0200
 Birth: -

So in essence, forcing a reindex for a directory and all its subdirectories on a ctime change may cause some unnecessary indexing depending on circumstances, but it should catch every file move.

[helsonxiao]

We're facing the same problem. Here's our solution.

#1230

[dadoonet]

@helsonxiao That's interesting. I was kind of afraid to overload Elasticsearch with such queries running for every single folder. Note that there is a hardcoded limit in the code as it can not retrieve more than 10000 files from a single dir which is a limit.

I think we could try to implement something similar to your PR but with the following changes:

• Implement this behavior with a feature flag (on by default) so we can always have the previous behavior if needed
• change the hardcoded limit to a parameter with the default value of 10000 (it can go up to 100000)
• get back from the response the information that with have more than the fixed limit and log warn that information by tracing the folder name and the number of documents we found

WDYT? Would you like to work on such a change?

[helsonxiao]

It looks good! I've seen some issues having the same problem. But I am afraid I don't have time recently. Anyone could help on it will be very appreciated!

[cadm-frank]

@helsonxiao That patch seems to work fine on a quick test run I just made. I'll have to test a bit more, but it looks promising.

[cadm-frank]

We're currently rolling this patch out and it does not seem to have a relevant performance impact.
If I understand the patch correctly, it's basically just a "have I indexed this filepath before?" type of check, meaning that it still could cause file changes to go unnoticed by fscrawler as it ignores differences in file's mtimes.
Example:

• User breaks a file by making unintended changes
• fscrawler indexes that broken file
• file is restored from backup to a point without the changes introduced by the user and gets the old mtime
  At this point fscrawler will not reindex the file as it's known and does not have a new mtime, despite having indexed a version that does not exist in the filesystem anymore.

I'm not too worried about that scenario as it's probably rare, but it's good to know the limits :)