Error: Backend down when using gateway to aws s3

[tomerzel87]

When writing multiple objects to a minio acting as a gateway to AWS with local cache, getting the following errors:

API: PutObject(bucket=data, object=agents/group-ADCDPL/agent-S0W1/resources/chunks/full_vol:DISK01:2018-12-06_114645.676Z:2805a94d-be66-47a9-b726-df3w82f57188:00001911.chunk.bin)
Time: 12:01:46 UTC 12/06/2018
RequestID: 156DBCAEBF30D7AF
RemoteHost: 192.1.1.2
UserAgent: jclouds/2.1.0 java/1.8.0_181
Error: Backend down
1: cmd/fs-v1-helpers.go:349:cmd.fsCreateFile()
2: cmd/disk-cache-fs.go:396:cmd.(*cacheFSObjects).PutObject()
3: cmd/disk-cache-fs.go:277:cmd.(*cacheFSObjects).Put()
4: cmd/disk-cache.go:696:cmd.cacheObjects.PutObject.func2()

The local cache is not close to being full.

Your Environment

• Version used (minio version): Version: 2018-11-22T02:51:56Z
• Using minio docker container
• Operating System and version (uname -a): Linux ubuntu16 4.4.0-139-generic

[kannappanr]

@tomerzel87 We will try to reproduce this issue and get back to you with more questions.

[poornas]

@tomerzel87, looks like you lost network connectivity to s3 backend while this object was being written. Cache writes occur simultaneously with backend writes - if there was a network issue, it would cause cache to fail as well. Is this happening consistently on all uploads or just a one-off issue?

[tomerzel87]

@poornas we wrote several blobs to AWS which caused our internet connection to become a bit slow.
I'd assume if there is a internet problem or if it's jammed, minio would write the blobs async to AWS when the line is clear. Does that sound reasonable?

[poornas]

@tomerzel87, the underlying minio-go client by default is configured to retry the upload upto 5 times before it returns the Backend Down error

[tomerzel87]

@poornas I can tell you that we tested with minio as gateway to AWS and without.
When we ran without minio the writes to AWS were ok and no timeouts occurred.
Only when we setup minio to check if it can help us as gateway we got the timeouts. Also what are the default timeouts?

[poornas]

@tomerzel87, retry starts at 30 seconds with exponential backoff. You mentioned that writes to AWS without minio as gateway had no timeouts - was this the case when you turn off cache? Can you paste your cache settings and how you are starting gateway with docker to help debug further

[tomerzel87]

@poornas our cache settings:
MINIO_CACHE_EXPIRY=30
MINIO_CACHE_MAXUSE=75

I've mounted an external directory as the caching directory and it had ~150GB of free space to utilize.

[poornas]

@tomerzel87 , tried to repro this issue starting gateway with docker container like so,

docker run -p 9000:9000  -e "MINIO_ACCESS_KEY=XXXXX" -e "MINIO_SECRET_KEY=XXXXXXX"  -e "MINIO_CACHE_DRIVES=/var/log/testcache" -v ~/testcache:/var/log/testcache -e "MINIO_CACHE_EXPIRY=30" -e "MINIO_CACHE_MAXUSE=75" minio/minio gateway s3

and ran some automated tests that upload objects to the gateway. I was able to simulate the error you see when I killed my network connection - after the retries, the client fails with Object storage backend is unreachable.

-- server log --
API: PutObject(bucket=aws-sdk-ruby-bucket-00499837ac00, object=datafile-1-MB)
Time: 07:05:05 UTC 12/13/2018
RequestID: 156FD2875255414E
RemoteHost: 172.17.0.3
UserAgent: aws-sdk-ruby3/3.43.0 ruby/2.3.1 x86_64-linux-gnu aws-sdk-s3/1.30.0
Error: Backend down
       1: cmd/fs-v1-helpers.go:349:cmd.fsCreateFile()
       2: cmd/disk-cache-fs.go:396:cmd.(*cacheFSObjects).PutObject()
       3: cmd/disk-cache-fs.go:277:cmd.(*cacheFSObjects).Put()
       4: cmd/disk-cache.go:696:cmd.cacheObjects.PutObject.func2()


-- client log --
(1/3) Running aws-sdk-ruby tests ... FAILED in 2 minutes and 37 seconds
{
  "name": "aws-sdk-ruby",
  "function": "putObject(bucket_name, file)",
  "args": {
    "bucket_name": "aws-sdk-ruby-bucket-00499837ac00",
    "file": "/mint/data/datafile-1-MB"
  },
  "duration": 153298,
  "error": "Object storage backend is unreachable",
  "status": "FAIL"
}

Executed 0 out of 3 tests successfully.

Can you run tcpdump while uploading to gateway with cache on, and attach packet trace? Also, which client is being used for uploading to the gateway?

[tomerzel87]

@poornas It will take me awhile to reproduce this on our lab due to other responsibilities.
Regarding the client it's part of a test program that uses Apache JClouds.

[poornas]

@tomerzel87, were you able to reproduce this with tcpdump enabled?

[deekoder]

PUTs will no longer be cached. Hence we will close this as a won't fix. @tomerzel87 let us know your thoughts.

[harshavardhana]

We discussed again, this time we are going to cache PUTs but we will be doing it with a new backend style.