Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.Sign up
Ability to dump folders to tar via stdout #2124
What is the purpose of this change? What does it change?
This enhances the dump command so that it will recognise if it was called on a file or a folder.
This implementation is a POC to test the viability and the speed of this solution, so it's not finished in terms of what it restores:
I tested this with a repository consisting of about 500 files and about 1.3Gb data. The speed compared to a normal restore is practically the same (even a small bit faster). And I'd like this pull request to serve as a discussion ground to finish this cleanly.
Was the change discussed in an issue or in the forum before?
@@ Coverage Diff @@ ## master #2124 +/- ## ========================================= - Coverage 51.05% 46.7% -4.35% ========================================= Files 176 173 -3 Lines 14348 14442 +94 ========================================= - Hits 7325 6745 -580 - Misses 5951 6674 +723 + Partials 1072 1023 -49
The build is currently failing due to the method I use to write the xattrs to the tar file. There are two ways to handle the xattrs: https://golang.org/pkg/archive/tar/#Header
As the Xattrs map is deprecated I used PAXRecords which is only available in go 1.10+. So the build for go 1.9 fails.
Any suggestions how I should handle this? One possibility would be to use build tags and implement both versions (not a fan of this but if there's no other way). Also just for curiosity: why is there still a build that uses go 1.9? Is this for some backward compatibility?
fd0 left a comment
Hey, thank you very much for your contribution, this function is great! Please accept my apologies for not reviewing this earlier, it's a rough time right now for me (at least concerning "spare time for restic in the evenings") ;)
As for your question: we try to support older versions of Go mainly so that people on Debian stable can build it. At the moment, the latest version for it available via the
A few issues I noticed while reviewing it:
Save this in a repository, then run
It looks like there's no information about the ACL in it... Can you have a look? :)
Thank you very much for the feedback. Keep hanging in there, you get through that rough time :)
I thought I fixed that one already... Maybe I forgot to push the commit. I'll have a look again.
You mean like mtime,ctime etc? To be honest I didn't really think about adding them. Good point, thx.
I'll have a look at that. I developed and tested the feature under OSX where the whole tar/xattr story is somewhat weird and broken. I'll try to replicate your test and fix this under Linux.
I found the issue with the ACLs and tar. ACLs have to be encoded completely different for a tar, even though they are technically saved as an xattr. The ACLs are saved in some binary format in the xattrs.
While researching about this encoding I found this implementation in go: https://github.com/maxymania/go-system/blob/master/posix_acl/posix_acl.go. Unfortunately this is actually the only information I was able to find how that encoding actually works. It might be burried somewhere deep inside some POSIX specification.
I added a modified version of that code to Restic which does actually write the correct ACL headers:
The only difference here is that the user gets saved with its ID instead of the actual name provided by the system.
fd0 left a comment
I had a look, it looks great! I have two remarks:
The rest looks great, I'll merge this after we agree on the two points above. Thank you very much for your work!
Thanks for the feedback!
I rebased everything into one clean commit.