Backup fails after about 2GB

[JaCoB1123]

Output of restic version

restic 0.5.0 (v0.5.0-0-gf678c97)
compiled with go1.8 on windows/amd64

Expected behavior

Backing up any file to minio/s3 continues without errors.

Actual behavior

Backup fails after about 2GB of a single 8GB file.

Steps to reproduce the behavior

> restic backup "D:\Backup\..."
scan [D:\Backup\...]
scanned 1 directories, 1 files in 0:00
panic: client.PutObject: Put https://example.com/restic-ubs/restic/data/7629894652bfee34a27ae6b27033224716de4a2f63c1a50fe12b62f0f5aecc7b: http2: Transport: peer server initiated graceful shutdown after some of Request.Body was written; define Request.GetBody to avoid this error

goroutine 4583 [running]:
restic/archiver.(*Archiver).saveChunk(0xc042161310, 0x9079c5d2, 0x93ff7, 0x0, 0x
c05b484000, 0x93ff7, 0x300000, 0xc042074bb0, 0xaef280, 0xc042004360, ...)
        /tmp/restic-build-310554648/src/restic/archiver/archiver.go:163 +0x277
created by restic/archiver.(*Archiver).SaveFile
        /tmp/restic-build-310554648/src/restic/archiver/archiver.go:237 +0x388

[fd0]

Uh, that's oddy. Thanks for the report! What backend did you use when the error in the paste occurred? Can you provide the minio server log (if it was minio)? Is there anything in between you and the backend, e.g. a load balancer or proxy?

@harshavardhana have you seen this error before?

[JaCoB1123]

The client-side is secured by a Sophos Firewall. I think it acts as a proxy as well. EDIT: HTTPS is not proxied
On the server-side there's a nginx instance running as a reverse proxy in front of minio.

minio was not logging when the error happened. I'll try to reproduce the error with logging enabled.

[JaCoB1123]

minio log only contains lines like this one:

time="2017-04-06T13:52:04+02:00" level=error msg="Unable to fetch object info." cause="Object not found: restic-ubs#restic/data/b7cc414349a34abccc64581b4cbb92f8c6aeef39908a21417f365a27e803a5f5" stack="fs-v1.go:322:fsObjects.getObjectInfo fs-v1.go:367:fsObjects.GetObjectInfo <autogenerated>:145:(*fsObjects).GetObjectInfo object-handlers.go:233:objectAPIHandlers.HeadObjectHandler api-router.go:37:(objectAPIHandlers).HeadObjectHandler-fm" 

[fd0]

Sounds like this: https://groups.google.com/forum/#!topic/golang-codereviews/Z3PJkpbTH0o

It sounds to me like the minio-go library should retry the request (at least that's what I interpret from the error message and the code linked in the mailing list thread). We've seen the same issue in #849 with nginx, but Go 1.8 has changed the http client library slightly.

You can try disabling http2 as a workaround with GODEBUG=http2client=0. Please report back :)

[JaCoB1123]

Yep, that seems to be the problem.
Using GODEBUG=http2client=0 I was able to complete a 25GB Backup without problems.

[fd0]

You can use that as a workaround until we figure out what's going on here.

[harshavardhana]

This is the same issue reported twice. Will open a bug on our end and debug it further. Btw is restic using HTTP2 forcefully ?

[fd0]

Ok, then I'll close this issue as a duplicate of #849. restic doesn't do anything to force http2. Thanks for looking into it!