cat: tries to do listing

[krishnasrinivas]

Is list operation needed for cat command?

krishna@escape:~$ mc --debug cat myminio/test/passwd
mc: <DEBUG> GET /test/?location= HTTP/1.1
Host: localhost:9000
User-Agent: Minio (linux; amd64) minio-go/2.0.4 mc/DEVELOPMENT.GOGET
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20170516/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20170516T185647Z
Accept-Encoding: gzip

mc: <DEBUG> HTTP/1.1 403 Forbidden
Transfer-Encoding: chunked
Accept-Ranges: bytes
Content-Type: application/xml
Date: Tue, 16 May 2017 18:56:47 GMT
Server: Minio/DEVELOPMENT.GOGET (linux; amd64)
Vary: Origin
X-Amz-Bucket-Region: us-east-1
X-Amz-Request-Id: 14BF2B1D84DCC1FF

e1
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied.</Message><Key></Key><BucketName></BucketName><Resource>/test/</Resource><RequestId>3L137</RequestId><HostId>3L137</HostId></Error>
0

mc: <DEBUG> Response Time:  2.075711ms

mc: <DEBUG> GET /test/?delimiter=%2F&max-keys=1000&prefix=passwd HTTP/1.1
Host: localhost:9000
User-Agent: Minio (linux; amd64) minio-go/2.0.4 mc/DEVELOPMENT.GOGET
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20170516/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20170516T185647Z
Accept-Encoding: gzip

mc: <DEBUG> HTTP/1.1 403 Forbidden
Transfer-Encoding: chunked
Accept-Ranges: bytes
Content-Type: application/xml
Date: Tue, 16 May 2017 18:56:47 GMT
Server: Minio/DEVELOPMENT.GOGET (linux; amd64)
Vary: Origin
X-Amz-Bucket-Region: us-east-1
X-Amz-Request-Id: 14BF2B1D84FAE241

e1
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied.</Message><Key></Key><BucketName></BucketName><Resource>/test/</Resource><RequestId>3L137</RequestId><HostId>3L137</HostId></Error>
0

mc: <DEBUG> Response Time:  2.703904ms

mc: <ERROR> Unable to read from `myminio/test/passwd`. Access Denied.
 (3) cat-main.go:168 cmd.mainCat(..) Tags: [myminio/test/passwd]
 (2) cat-main.go:88 cmd.catURL(..) Tags: [myminio/test/passwd]
 (1) client-url.go:181 cmd.url2Stat(..) Tags: [myminio/test/passwd]
 (0) client-s3.go:880 cmd.(*s3Client).Stat(..)
 Release-Tag:DEVELOPMENT.GOGET | Commit:DEVELOPMENT. | Host:escape | OS:linux | Arch:amd64 | Lang:go1.7.5 | Mem:1.0MB/8.4MB | Heap:1.0MB/4.8MB
krishna@escape:~$ 

[harshavardhana]

Yes it is expected.. @krishnasrinivas url2Stat looks for an object by listing .

[krishnasrinivas]

Can it be avoided? (I haven't checked the code flow) because I was testing gateway-caching when the backend is down and mc cat was failing.

From a user experience perspective mc cat should have worked because gateway has cached the contents

[harshavardhana]

Can it be avoided? (I haven't checked the code flow) because I was testing gateway-caching when the backend is down and mc cat was failing.

From a user experience perspective mc cat should have worked because gateway has cached the contents

It can do definitely but it was a design choice we validate all input arguments in each of the commands as checkCmdSyntax() .

[vadmeste]

Theoretically we can, but that probably it needs a lot of changes because it wasn't expected that we will ever download a nonexistent object :)

Generally speaking, mc sometimes makes some additional REST calls to provide better user experience and error messages. For example in mirror command, mc checks first if source & target are directories (otherwise, mirror won't make any sense)

For mc cat, I added the Stat() to check if the downloaded object size is equal to what claimed in the server.

[harshavardhana]

why is Gateway Caching failing? shouldn't the List be allowed @krishnasrinivas ? and what do you mean by backend was down?

[krishnasrinivas]

@harshavardhana backend-down means when GCS/AWS-S3 is not available (ex. network problem). In such a situation the minio gateway just serves files from cache. Listing can't be done as caching directory looks as described here: https://github.com/krishnasrinivas/wikinotes/wiki/Caching-gateway-proposal

@vadmeste if mc needs too many changes and compromises on user experience then we need not fix it. But if both this situation and UX can be taken care of then we can fix it.

@deekoder Priority can be lowered to low as mc was not written for minio-gateway which does not work support all S3 apis when the backend is down.

[krisis]

@krishnasrinivas for mc cat to work with gateway caching feature, you don't want List call to be made. How about HEAD request on the object in mc cat ALIAS/bucket/object?

[vadmeste]

The current PR modifies the behavior of mc cat, it ignores HEAD 404 response and moves forward to GET object.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.