Unable to perform aptly publish drop to repo published in GCS (S3-compatible cloud storage)

[smstone]

Hello,

I am unable to perform aptly publish drop as I receive a 400 Bad Request - Cannot create buckets using a POST. on a repo that was published via the Google Cloud Storage XML API (https://cloud.google.com/storage/docs/xml-api/overview)

Detailed Description

Commands run:

$ aptly publish list
Published repositories:
  * s3:testing/unstable << REMOVED >>
# debug mode on
$ aptly publish drop unstable s3:testing:bionic
<< SNIP >>
2019/10/30 21:39:47 <?xml version='1.0' encoding='UTF-8'?><Error><Code>InvalidArgument</Code><Message>Invalid argument.</Message><Details>Cannot create buckets using a POST.</Details></Error>
ERROR: unable to remove: published files removal failed, use -force-drop to override: error deleting multiple paths from S3: my-region-name-here:my-awesome-aptly-bucket/: InvalidArgument: Invalid argument.
    status code: 400, request id: , host id:

Verbose request/response from command above

---[ REQUEST POST-SIGN ]-----------------------------
POST /my-awesome-aptly-bucket?delete= HTTP/1.1
Host: storage.googleapis.com
User-Agent: aws-sdk-go/1.25.18 (go1.10.4; linux; amd64)
Content-Length: 43473
Authorization: AWS4-HMAC-SHA256 Credential=REMOVED, SignedHeaders=content-length;content-md5;host;x-amz-content-sha256;x-amz-date, Signature=REMOVED
Content-Md5: REMOVED
X-Amz-Content-Sha256: REMOVED
X-Amz-Date: 20191030T200743Z
Accept-Encoding: gzip

Request body is large XML document with list of objects/keys to delete.

Response headers and body:

2019/10/30 20:07:43 DEBUG: Response s3/DeleteObjects Details:
---[ RESPONSE ]--------------------------------------
HTTP/2.0 400 Bad Request
Content-Length: 171
Content-Type: application/xml; charset=UTF-8
Date: Wed, 30 Oct 2019 20:07:43 GMT
Server: UploadServer
Vary: Origin
X-Guploader-Uploadid: REMOVED
-----------------------------------------------------
2019/10/30 20:07:43 <?xml version='1.0' encoding='UTF-8'?><Error><Code>InvalidArgument</Code><Message>Invalid argument.</Message><Details>Cannot create buckets using a POST.</Details></Error>
ERROR: unable to remove: published files removal failed, use -force-drop to override: error deleting multiple paths from S3: my-region-name-here:my-awesome-aptly-bucket/: InvalidArgument: Invalid argument.
    status code: 400, request id: , host id:

I was able to get around this issue by adding -force-drop but the bucket is not cleaned up.

I did come across this issue minio/mc#2398 which seems exactly what I am hitting, so perhaps this command is incompatible if using Google Cloud Storage XML API?

Context

I'd like to use this to properly clean and un-publish unneeded/unused repos.

Your Environment

aptly version: 1.4.0

Example Aptly config I am using:

{
  "rootDir": "/aptly",
  "downloadConcurrency": 4,
  "downloadSpeedLimit": 0,
  "architectures": [],
  "dependencyFollowSuggests": false,
  "dependencyFollowRecommends": false,
  "dependencyFollowAllVariants": false,
  "dependencyFollowSource": false,
  "dependencyVerboseResolve": false,
  "gpgDisableSign": false,
  "gpgDisableVerify": false,
  "gpgProvider": "gpg",
  "downloadSourcePackages": false,
  "skipLegacyPool": true,
  "ppaDistributorID": "ubuntu",
  "ppaCodename": " ",
  "skipContentsPublishing": false,
  "FileSystemPublishEndpoints": {},
  "S3PublishEndpoints": {
    "production": {
      "region": "my-region-name-here",
      "bucket": "my-awesome-aptly-bucket",
      "endpoint": "https://storage.googleapis.com",
      "awsAccessKeyID": "REMOVED",
      "awsSecretAccessKey": "REMOVED",
      "acl": " ",
    }
  },
  "SwiftPublishEndpoints": {}
}

[lneva-fastly]

Try setting this in your S3 endpoint configuration: "disableMultiDel": true. I was getting 400s with GCS and that fixed it.

[smstone]

@lneva-fastly that did the trick! Thank you very much.

[lneva-fastly]

Glad I could help!

Note to the devs: this one was a bit tricky to debug. I turned on debug output, but all that gave me was the request headers, not the body of the request, so I was unable to reproduce the request with curl to see the full error. I got lucky that the above fix worked.

It would be really cool if error messages like InvalidArgument from the S3 endpoint included the full body of the error message. That would have helped me debug another GCS problem: aptly is incompatible with buckets with bucket-wide permissions enabled.

[smstone]

@lneva-fastly I encountered the same issue as well and had to recompile aptly with a different debug logging option to view the response body: #303 (comment)