Network timeout for federated shares causes repeated partial download of large files

[Derkades]

Describe the bug
Nextcloud seems to have a 30 second timeout for downloading files from federated shares. The Nextcloud developers don't consider this a bug, saying it is required for responsiveness.

This is also an issue during Memories indexing. It fails to index the file with the error message: "Failed to index file ...: Failed to get local file: cURL error 28: Operation timed out after 30000 milliseconds with ... out of ... bytes received". This error is visible when doing a manual index using occ, but the issue still occurs with only the Nextcloud default cron job.

Due to the error, it appears that Memories does not consider the file as indexed and will happily try again. This results in large network traffic load every 15 minutes as it attempts, and fails, to download the file.

While ideally I would like Memories to download the file and generate a thumbnail for it, that is probably not going to happen. So instead, it would be nice if Memories marked this file as indexed without a thumbnail.

Screenshots

Platform:

• Memories Version: 6.0.1
• Nextcloud Version: 27.1.3
• PHP Version: 8.1

[pulsejet]

This is potentially a general problem with external storage. Memories needs the entire file to be available locally for indexing, which doesn't play nice with this.

So instead, it would be nice if Memories marked this file as indexed without a thumbnail.

We can't really mark it as "indexed" because no information can be inferred without the file itself. So essentially this will cause a bunch of empty thumbs at the top of the timeline (cause we don't know the date either). I think the best solution here is to have some configuration to limit the size of file to index from external storage.

[pulsejet]

The Nextcloud developers don't consider this a bug,

BTW, I wouldn't consider this a bug either. When we reach the 30s mark, it's hard to say if we're actually receiving the file we want or something terrible is happening. E.g. if downloads were allowed beyond 30s, it's possible that all the bandwidth would be clogged up forever because some large files are being downloaded (forever).

[pulsejet]

Related #933. We probably need another table here to track files that failed to be indexed, then have a flag to retry these in the index command.

[pulsejet]

We can't really mark it as "indexed" because no information can be inferred without the file itself.

I just realised this isn't true. The preview generator might have the preview for whatever reason (maybe they are / could be shared in federations?), and it's always possible to fall back to modification time for the date/time. This isn't great by any means but seems like a reasonable compromise. We still do need a flag on the index entry to allow "retrying".

[simonspa]

As far as I can see they indeed did not see it as a bug, but added a config parameter anyway. Does setting davstorage.request_timeout to something larger solve your issue?

[Derkades]

Thanks, that indeed solves the issue

[pulsejet]

Fixed on master