Need ctime of files after uploads

[nickvergessen]

Ref nextcloud/files_retention#23

Steps

1. Set up retention to delete files after 14 days.
2. Sync or upload a file from last year
3. Run retention cron job

Expected

The file remains on the instance for 14 days

Actually

The file is immediately deleted, because the mtime of the file is 2016.

Problem

The problem is that since some time we use the original mtime of the file when writing the entry into the database instead of "current time". I don't know when that changed, but it's a bit annoying. Same happens with the "recent files" filter. It will not list such files, although you just uploaded them.

So we need some kind of timestamp when the file was first seen/discovered by nextcloud.

@icewind1991 you have any idea why and when this was changed?
Also cc @rullzer for the client, @MorrisJobke for the customer related issue and @schiessle as per yesterday.

[KB7777]

Wow, huge problem.

[icewind1991]

Maybe storage_mtime

[js94x]

Unfortunately, no. The column storage_mtime contains the same value as mtime.

After an upload over the web gui the original (old) mtime is set too.

OS: Windows 10
Browser: Firefox 51.0.1 (32-Bit)
Nextcloud: 11.0.1

[rullzer]

The client does this by design. Because if you edit something while you have no connection you don't want to see a changed time on the web wich differes (also you will most likely redownload the file then).

From my POV keeping the time of the file is correct. We should just also keep track of the uploaded time.

[icewind1991]

#3822 mostly fixes this imo

[nickvergessen]

Well it doesn't really help with the retention app, the behaviour is still "undesired/unintentional".

[icewind1991]

I would argue that with retention you do want the original mtime from the sync client.

If retention is set to 14 days, I expect it to trigger 14 days after I create the file, regardless of the fact that the client was offline for a week.

[nickvergessen]

Well but when you created the file four weeks ago and just put it into the sync folder just now. You don't want it to be deleted in the night when retention triggers, but 2 weeks after upload?

[rapITler]

Hi,

we facing the same issue. The uploads should get automatically deleted after 14 days after upload.
But some of our users reported that their files were dropped much earlier than expected.
After some tests and researches we found that nextcloud triggers on the mtime instead of the upload time.

Hopefully it will be fixed soon because this is a showstopper for our planned rollout.

[nickvergessen]

We really need to look into this.

[js94x]

I´ve seen that the milestone was modified from 12 to 13 a month ago. Is there already a commit which changes the behavior with ctime?

[rapITler]

Damn really bad news for us!
We postponed the rollout in our company to wait on NC12 only for that fix :(.

[maxmeyer]

Any progress on that one?

[marinofaggiana]

Any progress on that one?

[jospoortvliet]

@rapITler and others: if this is a issue for an enterprise PoC/deployment, contact our sales team, we can help either find an alternative solution or speed up fixing this.

[MorrisJobke]

@icewind1991 Mind to have a look into collecting the cdate as well?

[MorrisJobke]

ref icewind1991/SearchDAV#3

[despens]

I'd like to reference #15192 here, "Consistent handling of last modified dates when copying/moving files" ... it seems like different nextcloud components have different ideas about what to do with the dates.

[amarttinez]

Also what is the use case for the upload time?

Retention as per initial report comment. We need to delete files X days after they were uploaded.

Hi @nickvergessen @rullzer, for the moment, as a "workaround" what I'm doing is creating a 1day file-retention, manually executing cron.php and deleting the previously created 1day file-retention. Of course, it's not a good solution because not all files will be deleted (but almost all of them) and also because you need to do this manually.

Until a solution is found, that could nearly solve my problem. Regarding that, is it possible to execute a script to create a 1day file-retention, executing cron.php and deleting 1day file-retention? I could'nt find a working way to create or delete file-retentions. That would be great.

[marinofaggiana]

Possible that this is open from Feb. 2017 without a serious milestone ?

[jakubeliasz]

Could someone please explain why the behaviour differs depending on file size? It seems that files which are 10 MiB and less get their timestamp rewritten to upload time, while bigger files keep the original modify date.

I created 3 files with dd locally:

$ ls -al *10
-rw-rw-r--. 1 jakub jakub 10354688 Nov 13 00:36 almost10
-rw-rw-r--. 1 jakub jakub 10485760 Nov 13 00:36 exactly10
-rw-rw-r--. 1 jakub jakub 10616832 Nov 13 00:36 over10
$ stat *10 | egrep "File|Size|Modify"
  File: almost10
  Size: 10354688  	Blocks: 20224      IO Block: 4096   regular file
Modify: 2019-11-13 00:36:08.025100807 +0100
  File: exactly10
  Size: 10485760  	Blocks: 20480      IO Block: 4096   regular file
Modify: 2019-11-13 00:36:18.356070527 +0100
  File: over10
  Size: 10616832  	Blocks: 20736      IO Block: 4096   regular file
Modify: 2019-11-13 00:36:28.463040904 +0100

and uploaded separately to my home folder with "+" > "Upload file".
Largest file shows the original timestamp in UI and filesystem:

# ls -al *10
-rw-r--r-- 1 www-data www-data 10354688 Nov 13 00:45 almost10
-rw-r--r-- 1 www-data www-data 10485760 Nov 13 00:45 exactly10
-rw-r--r-- 1 www-data www-data 10616832 Nov 13 00:36 over10
# stat *10 | egrep "File|Size|Modify"
  File: almost10
  Size: 10354688  	Blocks: 20224      IO Block: 4096   regular file
Modify: 2019-11-13 00:45:11.850246461 +0100
  File: exactly10
  Size: 10485760  	Blocks: 20480      IO Block: 4096   regular file
Modify: 2019-11-13 00:45:02.934138994 +0100
  File: over10
  Size: 10616832  	Blocks: 20736      IO Block: 4096   regular file
Modify: 2019-11-13 00:36:28.000000000 +0100

PS. As a workaround to have retention working as expected I added flowupload app and asked users to use that instead, since this always seem to update the modify time to upload time.

Do you expect to fix this before the end of this year with 17.0.2 release?

[tobiasKaminsky]

@juliushaertl I think you mentioned a fix for this?

[juliushaertl]

Yes, for the file upload of smaller 10MB files: #17847

[jakubeliasz]

Thank you for quick feedback! This makes it more clear why this happens.
Still the question remains, since all files will keep their original modify timestamp, what about retention? See nextcloud/files_retention#70

[juliushaertl]

I guess that can be addressed once we have a dedicated column to store the upload time #17765

[jakubeliasz]

While waiting for a fix (NC18?) I found the cleanest workaround to have retention working properly for all files would be to remove X-OC-Mtime header via haproxy/nginx at the backend. Thank you!

[oparoz]

#17765 was merged.

[lefty556]

remove X-OC-Mtime

@jakubeliasz Did you have an example nginx config to do that?

[zahradehghanpur]

remove X-OC-Mtime header worded for me .
In Below there is my config on httpd server on centos:

<IfModule headers_module> RequestHeader unset "X-OC-Mtime" </IfModule>