Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make sure timestamps are valid #2253

Merged
merged 1 commit into from Apr 25, 2019
Merged

Make sure timestamps are valid #2253

merged 1 commit into from Apr 25, 2019

Conversation

fd0
Copy link
Member

@fd0 fd0 commented Apr 24, 2019

What is the purpose of this change? What does it change?

Sometimes restic gets bogus timestamps which cannot be converted to JSON, because the stdlib JSON encoder returns an error if the year is not within [0, 9999]. We now make sure that we at least record some timestamp and cap the year either to 0000 or 9999. Before, restic would refuse to save the file at all, so this improves the status quo.

Was the change discussed in an issue or in the forum before?

closes #2174, #1173

Checklist

  • I have read the Contribution Guidelines
  • I have added tests for all changes in this PR
  • There's a new file in changelog/unreleased/ that describes the changes for our users (template here)
  • I have run gofmt on the code in all commits
  • All commit messages are formatted in the same style as the other commits in the repo
  • I'm done, this Pull Request is ready for review

Sometimes restic gets bogus timestamps which cannot be converted to
JSON, because the stdlib JSON encoder returns an error if the year is
not within [0, 9999]. We now make sure that we at least record _some_
timestamp and cap the year either to 0000 or 9999. Before, restic would
refuse to save the file at all, so this improves the status quo.

This fixes #2174 and #1173
@codecov-io
Copy link

codecov-io commented Apr 24, 2019

Codecov Report

Merging #2253 into master will increase coverage by 0.14%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #2253      +/-   ##
==========================================
+ Coverage    50.8%   50.94%   +0.14%     
==========================================
  Files         178      178              
  Lines       14530    14529       -1     
==========================================
+ Hits         7382     7402      +20     
+ Misses       6080     6063      -17     
+ Partials     1068     1064       -4
Impacted Files Coverage Δ
internal/restic/node.go 43.83% <100%> (+2.99%) ⬆️
internal/backend/test/tests.go 60.61% <0%> (+0.65%) ⬆️
internal/backend/s3/s3.go 62.4% <0%> (+2.25%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c776245...939f3e9. Read the comment docs.

@robvalca
Copy link

Thanks a lot Alex.
I've quick-tested this branch and seems to do the job. Tried backup/restore. The only thing is that the affected file is always backed-up in the incrementals but this is expected as it differs from the file in the repo. But I think that this is better than not backing up anything : )

@fd0
Copy link
Member Author

fd0 commented Apr 24, 2019

Thanks for the feedback! Yeah, that's the effect of the mtime being bogus. Can you maybe post a stat of one of the affected files? If/when we switch to using ctime for modification detection (see #2212), that will probably go away as the ctime is set by the OS.

@robvalca
Copy link

Hi Alex,
In our case the original ctime is already some kind weird but yes, seems that this would not happen if the ctime was checked instead of the mtime.

This is the original file:

File: ‘.swanproject’
Size: 0         	Blocks: 0          IO Block: 4096   regular empty file
Device: 2ch/44d	Inode: 5           Links: 1
Access: (0644/-rw-r--r--)  Uid: (11111/aaaaa)   Gid: ( 1111/      aaaaa)
Context: system_u:object_r:fusefs_t:s0
Access: 4428626-11-10 23:45:52.007621616 +0100
Modify: 4428626-11-10 23:45:52.007621616 +0100
Change: 1970-03-30 06:29:12.-2013272080  +0100
 Birth: -

This is the restored file:

File: ‘.swanproject’
Size: 0         	Blocks: 0          IO Block: 4096   regular empty file
Device: fd01h/64769d	Inode: 14680157    Links: 1
Access: (0644/-rw-r--r--)  Uid: (11111/aaaaa)   Gid: ( 1111/      aaaa)
Context: unconfined_u:object_r:user_tmp_t:s0
Access: 1816-02-08 05:16:08.073898992 +0034
Modify: 1816-02-08 05:16:08.073898992 +0034
Change: 2019-04-24 17:15:15.365003503 +0200
 Birth: -

I'll would be happy to help if you need something else.
Cheers,

@fd0
Copy link
Member Author

fd0 commented Apr 25, 2019

Thanks for the stats, it looks like all timestamps are bogus. Huh. So I don't particularly care about that and will merge this PR now ;)

@fd0 fd0 merged commit 939f3e9 into master Apr 25, 2019
fd0 added a commit that referenced this pull request Apr 25, 2019
Make sure timestamps are valid
@fd0 fd0 deleted the fix-2174 branch April 25, 2019 07:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Invalid mtimes (year > 9999) fails snapshot
3 participants