mc ls no longer work with GCS since 2019-12-17T23-26-28Z

[wsxiaoys]

Expected behavior

mc ls google/... worked.

Actual behavior

mc: Unable to list folder. A header or query you provided requested a function that is not implemented.
(1) ls.go:132 cmd.doList(..) Tags: [https://storage.googleapis.com/ps-gcs/]
(0) client-s3.go:1906 cmd.(*s3Client).listInRoutine(..)
Commit:b70a79e39bbd | Release-Tag:RELEASE.2020-01-25T03-02-19Z | Host:jupyter-meng | OS:linux | Arch:amd64 | Lang:go1.13.6 | Mem:2.1 MB/72 MB | Heap:2.1 MB/66 MB

mc --version

mc version RELEASE.2020-01-25T03-02-19Z

[kannappanr]

@wsxiaoys Thanks for reporting the issue, we will take a look and get back to you.

[kannappanr]

@wsxiaoys It is working as expected with this release. Can you please verify that mc config host list gcs prints the accesskey and secret key that you expect. Most probably those creds are not part of the alias. Based on the executable name, mc creates a folder in the home folder and each binary name will have it's own config.json file. You might have to add those creds to that alias again.

[lsde]

I can confirm this bug, which appeared in RELEASE.2019-12-17T23-26-28Z.
Since this version, accessing data on GCS is impossible (upload is possible though).
Just try it for yourself:
(both releases have their respective configs with the same access credentials)

RELEASE.2019-10-09T22-54-57Z

$  ./mc.RELEASE.2019-10-09T22-54-57Z ls --json gcs/test1582795099
{
 "status": "success",
 "type": "file",
 "lastModified": "2020-02-27T10:19:12.098+01:00",
 "size": 9,
 "key": "testfile",
 "etag": "blabla"
}

RELEASE.2019-12-17T23-26-28Z

$ ./mc.RELEASE.2019-12-17T23-26-28Z ls --json gcs/test1582795099
{
 "status": "error",
 "error": {
  "message": "Unable to list folder.",
  "cause": {
   "message": "A header or query you provided requested a function that is not implemented.",
   "error": {
    "Code": "NotImplemented",
    "Message": "A header or query you provided requested a function that is not implemented.",
    "BucketName": "",
    "Key": "",
    "RequestID": "",
    "HostID": "",
    "Region": ""
   }
  },
  "type": "error",
  "sysinfo": {
   "host.arch": "amd64",
   "host.cpus": "32",
   "host.lang": "go1.13.5",
   "host.name": "blabla",
   "host.os": "linux",
   "mem.heap.total": "66 MB",
   "mem.heap.used": "2.5 MB",
   "mem.total": "72 MB",
   "mem.used": "2.5 MB"
  }
 }
}

[harshavardhana]

This is because GCS doesn't implement encoding--type=url for their ListObjectsV2 implementation or may be metadata=true field is ignored.

[manigandham]

We've run into the same issue, latest release doesn't list files on GCS.

[harshavardhana]

@lsde can you provide --debug output?

[manigandham]

Here's our debug output if this helps:

mc ls --json --debug gs/dbfiles-1

mc: <DEBUG> GET /dbfiles-1/?location= HTTP/1.1
Host: storage.googleapis.com
User-Agent: MinIO (linux; amd64) minio-go/v6.0.49 mc/2020-02-25T18:10:03Z
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20200303/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20200303T074535Z
Accept-Encoding: gzip
mc: <DEBUG> HTTP/1.1 200 OK
Content-Length: 81
Alt-Svc: quic=":443"; ma=2592000; v="46,43",h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000
Cache-Control: private, max-age=0
Content-Type: application/xml; charset=UTF-8
Date: Tue, 03 Mar 2020 07:45:35 GMT
Expires: Tue, 03 Mar 2020 07:45:35 GMT
Server: UploadServer
X-Goog-Metageneration: 9
X-Guploader-Uploadid: AEnB2UqQx_KfFFpTG4a05k6l5Vrttp6PiD1LU0gYVeRrFNeURLRpOvWNbc6THp3_wIoloUobmU_JIqTiFiqttj5w8TlFIUo5FogHKjrQVo0guM_lGbftq9E
mc: <DEBUG> Response Time:  144.900019ms
mc: <DEBUG> HEAD / HTTP/1.1
Host: dbfiles-1.storage.googleapis.com
User-Agent: MinIO (linux; amd64) minio-go/v6.0.49 mc/2020-02-25T18:10:03Z
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20200303/US/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20200303T074535Z
mc: <DEBUG> HTTP/1.1 200 OK
Content-Length: 1198
Alt-Svc: quic=":443"; ma=2592000; v="46,43",h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000
Cache-Control: private, max-age=0
Content-Type: application/xml; charset=UTF-8
Date: Tue, 03 Mar 2020 07:45:35 GMT
Expires: Tue, 03 Mar 2020 07:45:35 GMT
Server: UploadServer
X-Goog-Metageneration: 9
X-Guploader-Uploadid: AEnB2Uq2ZNFhVqsIXiqG6353lcudosw-fwXGZYK_SlaM-sUc14F6HtaBbiGI_5eMAtI3TZlqAKEQ2G-VTgTHN1wG-E7XS5tOUA
mc: <DEBUG> Response Time:  138.03758ms
mc: <DEBUG> HEAD / HTTP/1.1
Host: dbfiles-1.storage.googleapis.com
User-Agent: MinIO (linux; amd64) minio-go/v6.0.49 mc/2020-02-25T18:10:03Z
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20200303/US/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20200303T074535Z
mc: <DEBUG> HTTP/1.1 200 OK
Content-Length: 1198
Alt-Svc: quic=":443"; ma=2592000; v="46,43",h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000
Cache-Control: private, max-age=0
Content-Type: application/xml; charset=UTF-8
Date: Tue, 03 Mar 2020 07:45:35 GMT
Expires: Tue, 03 Mar 2020 07:45:35 GMT
Server: UploadServer
X-Goog-Metageneration: 9
X-Guploader-Uploadid: AEnB2Uro83A7EztRZOAlsPs7We1192_0rGzqIgVZKvN-o5zPErf0G4GE2JeHmzXMBVj_c2S3P_ilP-GbcpCX4OZWeu5TpZx2i4CSLYXIPtg1D56alhg1Oqk
mc: <DEBUG> Response Time:  65.256384ms
mc: <DEBUG> GET /?delimiter=%2F&encoding-type=url&fetch-owner=true&list-type=2&prefix= HTTP/1.1
Host: dbfiles-1.storage.googleapis.com
User-Agent: MinIO (linux; amd64) minio-go/v6.0.49 mc/2020-02-25T18:10:03Z
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20200303/US/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20200303T074535Z
Accept-Encoding: gzip
mc: <DEBUG> HTTP/1.1 400 Bad Request
Content-Length: 239
Alt-Svc: quic=":443"; ma=2592000; v="46,43",h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000
Cache-Control: private, max-age=0
Content-Type: application/xml; charset=UTF-8
Date: Tue, 03 Mar 2020 07:45:35 GMT
Expires: Tue, 03 Mar 2020 07:45:35 GMT
Server: UploadServer
X-Guploader-Uploadid: AEnB2UrHpj2NX4Au24NGE608QejC3rglpOFmRyTGCi1jWER_0AILJmiTeEGkrTX8oSd4lUp-WXeA1AwDUW-4ajOL2041RWnedSvm0duLi6f2UTC-D6vMpAM
<?xml version='1.0' encoding='UTF-8'?><Error><Code>NotImplemented</Code><Message>A header or query you provided requested a function that is not implemented.</Message><Details>GET ?list-type is not implemented for buckets</Details></Error>mc: <DEBUG> Response Time:  46.834087ms
{"status":"error","error":{"message":"Unable to list folder.","cause":{"message":"A header or query you provided requested a function that is not implemented.","error":{"Code":"NotImplemented","Message":"A header or query you provided requested a function that is not implemented.","BucketName":"","Key":"","RequestID":"","HostID":"","Region":"","Server":"UploadServer"}},"type":"error","trace":[{"line":1922,"file":"client-s3.go","func":"cmd.(*s3Client).listInRoutine"},{"line":132,"file":"ls.go","func":"cmd.doList","env":{"Tags":["https://storage.googleapis.com/dbfiles-1/"]}}],"sysinfo":{"host.arch":"amd64","host.cpus":"16","host.lang":"go1.13.7","host.name":"d2f8943f5255","host.os":"linux","mem.heap.total":"66 MB","mem.heap.used":"2.3 MB","mem.total":"72 MB","mem.used":"2.3 MB"}}}

[harshavardhana]

This looks like GCS S3 layer doesn't implement AWS S3 compatible ListObjectsV2 implementation

[harshavardhana]

FWIW #3101 should address this - NOTE: this is not a bug in mc as such - it is just that GCS S3 layer has some catching up to do regarding their S3 compatibility.

[manigandham]

Here are the GCS docs for reference.

List Objects via the S3 Interoperability: https://cloud.google.com/storage/docs/xml-api/get-bucket-list