Running "occ files:scan" ignores request to scan (and clear) a missing file

[eharris]

Steps to reproduce

1. Upload a test file to the Nextcloud instance
2. Delete the file from the underlying file store (NOT via Nextcloud)
3. Run "occ files:scan --path=[path to now missing file] to make it realize the file is gone
4. Observe that occ errors out with "Path not found: [path to missing file]" error rather than completing the requested action and updating/removing the missing file.

Expected behaviour

Nextcloud should recognize that while the file it thinks is there is gone from the underlying datastore, it still needs to correct its records and remove it from the filesystem view it presents to users/apps.

Actual behaviour

The occ command errors out with "Path not found: " error, yet continues to show that the file exists through webdav, which is an incorrect and unwanted inconsistency.
Attempts to access the file that appears to be there but isn't will continue to produce errors.

Additional Notes

This worked as expected on Nextcloud 9.0.58, which is what I was running before updating to Nextcloud 15 and then 16. Both Nextcloud 15 and 16 have this problem. I haven't tested later versions, but I would guess it might still be there as well.

There is an obvious workaround, which is to scan the whole directory that contains the missing file, rather than to specify that occ should scan just the file that is trying to be fixed/cleared. That is not a reasonable workaround though, because the scan takes a prohibitively long time and severely impacts performance of the system for that time if the containing directory has many other subfolders and hundreds of thousands of files.

Server configuration

Operating system: Debian Buster 10 (stock)

Web server: N/A

Database: mariadb-server-10.3 1:10.3.22-0+deb10u1 (stock Debian)

PHP version: php7.3 7.3.11-1~deb10u1

Nextcloud version: Nextcloud 16.0.8
(I verified this issue also occurred on 15.0.14)

Updated from an older Nextcloud/ownCloud or fresh install: Updated originally from Nextcloud 9.0.58

Where did you install Nextcloud from: https://download.nextcloud.com/server/releases/nextcloud-16.0.8.tar.bz2

Signing status:

Signing status

No errors have been found.

Are you using external storage, if yes which one: no

Are you using encryption: no

Are you using an external user-backend, if yes which one: no

[szaimen]

Hi there, the path is relative to nextclouds datadir and not the path in the system.
Since this is is a support request, I would like to ask you to get help on https://help.nextcloud.com
Thanks!

[eharris]

This is not a support request, it is a bug. It is also not a pathing issue. The steps to reproduce are clearly laid out.

[szaimen]

So what was the complete command that you entered? (I am especially interested in path to missing file )

[eharris]

Any file that doesn't exist. Drop a file somewhere. Run scan on that specific filename. See it succeed. Delete the file from the filesystem but NOT through nextcloud. Run the same scan on the same filename again (intending to make Nextcloud realize the file is gone) and see that it doesn't work.

[szaimen]

And what would have been the path to the missing file in your example?

[szaimen]

Ah I see what the problem here is: the path that you specify here needs to be an in Nextclouds datadir existing folder or for a user existing virtual folder. It isn't allowed to be a file.

[eharris]

It seems to work as a file if the file exists. Just not if it doesn't. I don't want to rescan a whole directory tree that has tons of subdirectories when all I need to update is one file.

[szaimen]

But you could limit the scanner path to the folder where the file was in? Or do you regularly need to do that?
Why not scheduling files:scan on a cronjob basis if the files are regularly changed/deleted externally?

[szaimen]

Here is a good comment how you can do this with a script https://help.nextcloud.com/t/automate-occ-filescan/35282/2

[eharris]

I wasn't aware there was a way to make it scan a single level subdirectory without descending children of that subdirectory. How is that done?

As to why not automate scanning, because millions of files on many remote mounted filesystems generates massive amounts of network traffic and takes an exceedingly long time (both of which are individually cost-prohibitive), when it should be trivial and excessively fast since I know exactly what file needs to be updated. And a periodic rescan will only catch changes after some period of time, and is not a good solution when I already know (and immediately) what file changed.

[szaimen]

Whats about this config.php setting: would that work?

[eharris]

The filesystem_check_changes option might work for a workaround, but I'd have to test it. But that still doesn't fix the originally reported problem of scan working on an individual file if it exists, but not if it doesn't.

[eharris]

I think this should still be considered a bug. Being able to add but not remove specific files is an asymmetry that needs to be fixed.

[ruedigerkupper]

I am not sure if I correctly understand what you did: Did you manually (not through NC) delete a file below NC's data directory? If so, this is the cause of the problem: Manually manipulating NC's data dir will always confuse NC and cause any kind of strange behavior. This is different in locations included through NC's "external storage" feature: This will correctly handle files that have been deleted through external processes.
So, in short, if you felt that you should delete a file below NC's data dir, this indicates you should better be using "external storage".

[Jip-Hop]

Even when using external storage it could be beneficial to trigger occ files:scan --path= when changes are detected. For example with syncthing-nextcloud I'm listening for events from syncthing. I know exactly when a file is created/modified/deleted and can therefore call occ files:scan --path= for the exact path that needs re-indexing. But calling occ files:scan for a file which has been deleted doesn't work and it's not efficient to rescan the parent folder for each delete event.