Rclone backend #1657
Rclone backend #1657
Conversation
internal/backend/rest/rest.go
Outdated
| url *url.URL | ||
| sem *backend.Semaphore | ||
| client *http.Client | ||
| backend.Layout | ||
| } | ||
|
|
||
| const ( | ||
| contentTypeV1 = "application/vnd.x.restic.rest.v1" | ||
| contentTypeV2 = "application/vnd.x.restic.rest.v2" | ||
| ContentTypeV1 = "application/vnd.x.restic.rest.v1" |
There was a problem hiding this comment.
exported const ContentTypeV1 should have comment (or a comment on this block) or be unexported
fd0
force-pushed
the
rclone-backend
branch
3 times, most recently
from
0f864bf to
ff20223
Compare
| rtest "github.com/restic/restic/internal/test" | ||
| ) | ||
|
|
||
| const rcloneConfig = ` |
There was a problem hiding this comment.
You could do away with this config if you want - see below
| } | ||
|
|
||
| cfg := rclone.NewConfig() | ||
| cfg.Program = fmt.Sprintf("rclone --config %q", cfgfile) |
There was a problem hiding this comment.
You can remove --config %q and the saving of the file above and just use
cfg.Remote = repodir
rclone remotes are either remote:remote/path or /local/path. (There is a little bit of ambiguity there I'm sure you'll notice which you can workaround with ./localdirwithcolon:/path)
There was a problem hiding this comment.
Ah, nice, I didn't know that.
internal/backend/rclone/config.go
Outdated
| // Config contains all configuration necessary to start rclone. | ||
| type Config struct { | ||
| Program string `option:"program" help:"path to rclone (default: rclone)"` | ||
| Args string `option:"args" help:"arguments for running rclone (default: restic serve --stdio)"` |
There was a problem hiding this comment.
That should probably be (default: serve restic --stdio)
internal/backend/rclone/backend.go
Outdated
|
|
||
| // Close terminates the backend. | ||
| func (be *Backend) Close() error { | ||
| debug.Log("exting rclone") |
| @@ -0,0 +1,72 @@ | |||
| package rclone | |||
There was a problem hiding this comment.
It ocurred to me that this is the kind of useful little object that might go in its own repo. Rclone could then import it too!
There was a problem hiding this comment.
That also occurred to me, but I prefer to not couple the source code too much for now. We can always move it out later (ideally, when vgo is merged)
Codecov Report
@@ Coverage Diff @@
## master #1657 +/- ##
=========================================
- Coverage 52.43% 52.2% -0.23%
=========================================
Files 143 148 +5
Lines 11415 11639 +224
=========================================
+ Hits 5985 6076 +91
- Misses 4507 4627 +120
- Partials 923 936 +13
Continue to review full report at Codecov.
|
|
So, the code is basically done, it just needs some docs. I could use some help testing the backend :) |
What sort of tests? I could run the backend unit tests (using rclone's unit test) against all the test accounts for all the different providers quite easily? Would that help? |
That'd be interesting! It may transfer a non-trivial amount of data though... What I meant was more like manual tests, accessing repos stored in some service and created directly with restic indirectly via rclone and so on. Report back on how it feels :) |
|
Docs are done, please let me know if there's anything odd! |
|
I commented on one small thing in 2beaad1. |
|
In addition to the comment I mentioned above, I suggest you also add rclone to the list of backends in the README.rst file :) |
|
Cool, thanks! |
|
I ran the integration tests against all the remotes last night... The results look good though not perfect. Straight passes for
Fails of some kind for
I'll dig into these. Here are the logs in case you are interested: integration-tests.zip B2There were no errors in the rclone log, but this test failed I think that is worth investigating further as it seems consistent HubicHubic failed with service problems or eventual consistency problems. rclone would deal with this sort of thing with lots of retries... OnedriveFailed because of a recent commit to rclone! |
|
I had a look through the docs - they look great :-) You might want to add something like this, which restic users might find easier since the restic configuration is often done with environment variables. .... Rclone can be configured with environment variables also, so for instance if you wanted to set a bandwidth limit for rclone you could |
|
Thanks for the comments, I've added the suggested text :) |
|
@ncw hm, is it possible that the error for B2 is caused by not calling rclone with |
|
Looks like it, running the backend tests for the same directory without Shall we maybe enable this by default in rclone for the restic server? Or just hard-code this into the backend test call for Are there any other backends which have a concept of "versions" of files for which we should do the same? |
|
@fd0 It should make no difference whether Apparently the test is indicating that there is an extra Looking at that function I'm wondering if there might be some sort of eventual consistency thing going on as that function seems to collect up as many list entries as it can, rather than just returning the last list which had the desired entry in it. I wonder whether something like this might fix the problem... --- a/internal/backend/test/tests.go
+++ b/internal/backend/test/tests.go
@@ -673,9 +673,10 @@ func (s *Suite) delayedRemove(t testing.TB, be restic.Backend, handles ...restic
}
func delayedList(t testing.TB, b restic.Backend, tpe restic.FileType, max int, maxwait time.Duration) restic.IDs {
- list := restic.NewIDSet()
+ var list restic.IDSet
start := time.Now()
for i := 0; i < max; i++ {
+ list = restic.NewIDSet()
err := b.List(context.TODO(), tpe, func(fi restic.FileInfo) error {
id := restic.TestParseID(fi.Name)
list.Insert(id)
@@ -783,7 +784,7 @@ func (s *Suite) TestBackend(t *testing.T) {
list := delayedList(t, b, tpe, len(IDs), s.WaitForDelayedRemoval)
if len(IDs) != len(list) {
- t.Fatalf("wrong number of IDs returned: want %d, got %d", len(IDs), len(list))
+ t.Fatalf("wrong number of IDs returned: want %d, got %d\n want:\n %v\n got:\n%v\n", len(IDs), len(list), IDs, list)
}
sort.Sort(IDs)I tried it - it didn't! Any more ideas? |
|
@ncw I've managed to reproduce it while writing debug logs for restic and rclone, see here: Restic was started like this: and rclone: Maybe you can see something. What I discovered so far was: restic creates and deletes the file It was found before and for other file types. So, a file/key is missing, instead of one additional key. I can see in the rclone log file that listing the subdirs of Any idea what's going on? |
|
So, what's still to do before this can be merged? From my point of view, we need find a solution for the deletion issue on B2. I'd like to avoid users ending up with plenty of invisible files stored at B2 that they're not aware of. When restic deletes files via rclone, are they eventually deleted on B2? Is there some delayed remove for hidden files? Or do users need to manually run |
This makes it easier for rclone.
When `/` is requested, rclone returns the list of all files in the remote, which is not what we want (and it can take quite some time).
For details see #1657 (comment)
|
I've added the two parameters for now, we can remove them from restic once the next version of rclone is released. Thanks a lot for your work @ncw! Once the integration tests complete I'll merge this! |
What is the purpose of this change? What does it change?
These commits add a backend based on rclone (or, an unreleased version of rclone). The corresponding PR for rclone is here: rclone/rclone#2116
Basically, restic starts an instance of rclone, talks to it via HTTP2 on stdin/stdout, and then uses the REST protocol.
Was the change discussed in an issue or in the forum before?
Closes #1561
Checklist
changelog/unreleased/that describes the changes for our users (template here)gofmton the code in all commitsTODO: