Unable to connect to Runpod.io S3 buckets

[karthik-picologic]

When trying to connect to a Runpod.io S3 bucket, Cyberduck consistently fails, initially this was due to region settings not matching the bucket, but once this was resolved with a custom profile Cyberduck continues to fails after listing the bucket alone. This is due to the directory listing requests including a trailing slash on the request URL, which it appears Runpod does not support.

Steps to reproduce the behavior:

1. Create an S3 bookmark for a runpod.io S3 bucket
2. Try to browse the bucket
3. See the error about invalid path

[dkocher]

Previously discussed in #17723.

[karthik-picologic]

Thanks. I have prototyped a fix locally and verified. Patch is below

Fixes #17901

Introduces a config value "s3.bucket.name.trailing.delimiter" with defaults to true (maintaining existing behaviour).
When set to false, removes the trialing slash on directory listings.

---
 defaults/src/main/resources/default.properties              | 1 +
 .../cyberduck/core/s3/RequestEntityRestStorageService.java  | 6 ++++--
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/defaults/src/main/resources/default.properties b/defaults/src/main/resources/default.properties
index f1ef70f9a7..e788034699 100644
--- a/defaults/src/main/resources/default.properties
+++ b/defaults/src/main/resources/default.properties
@@ -294,6 +294,7 @@ s3.signature.headers.exclude=Connection Authorization Proxy-Authorization Proxy-
 # Default bucket location
 s3.location=us-east-1
 s3.bucket.virtualhost.disable=false
+s3.bucket.name.trailing.delimiter=true
 s3.bucket.requesterpays=true
 s3.domain=amazonaws.com
 s3.hostname.default=s3.amazonaws.com
diff --git a/s3/src/main/java/ch/cyberduck/core/s3/RequestEntityRestStorageService.java b/s3/src/main/java/ch/cyberduck/core/s3/RequestEntityRestStorageService.java
index d162a8c85a..9146751eaf 100644
--- a/s3/src/main/java/ch/cyberduck/core/s3/RequestEntityRestStorageService.java
+++ b/s3/src/main/java/ch/cyberduck/core/s3/RequestEntityRestStorageService.java
@@ -197,6 +197,8 @@ public class RequestEntityRestStorageService extends RestS3Service {
         // Virtual host style endpoint including bucket name
         String hostname = endpoint;
         String resource = String.valueOf(Path.DELIMITER);
+        final String bucketDelimiter = objectKey != null || preferences.getBoolean("s3.bucket.name.trailing.delimiter")
+                ? String.valueOf(Path.DELIMITER) : StringUtils.EMPTY;
         if(!this.getDisableDnsBuckets()) {
             // Virtual host style requests enabled in connection profile
             if(StringUtils.isNotBlank(bucketName)) {
@@ -205,14 +207,14 @@ public class RequestEntityRestStorageService extends RestS3Service {
                 }
                 else {
                     // Add bucket name to path
-                    resource += URIEncoder.encode(bucketName) + Path.DELIMITER;
+                    resource += URIEncoder.encode(bucketName) + bucketDelimiter;
                 }
             }
         }
         else {
             if(StringUtils.isNotBlank(bucketName)) {
                 // Add bucket name to path
-                resource += URIEncoder.encode(bucketName) + Path.DELIMITER;
+                resource += URIEncoder.encode(bucketName) + bucketDelimiter;
             }
         }
         final HttpUriRequest request;
-- 
2.46.0

[karthik-picologic]

I am able to connect, browse, download files, with the above patch and a .cyberduckprofile that includes the following (Region in my case is eu-ro-1)

        <key>Region</key>
        <string>eu-ro-1</string>
        <key>Properties</key>
        <dict>
            <key>s3.bucket.virtualhost.disable</key>
            <true/>
            <key>s3.bucket.name.trailing.delimiter</key>
            <false/>
        </dict>

[dkocher]

I do not see how this would work when the objectKey parameter is a non-empty value.

[karthik-picologic]

If objectKey is non-empty or If s3.bucket.name.trailing.delimiter is true (default), the behaviour reverts to the current behaviour which defaults to adding the trailing / in the bucket name.

The objectKey non-empty check is required since when addressing specific files the trailing slash is required in the path. The main issue affects directory ls alone it appears. Hence the check for objectKey != null

[dkocher]

It would be great if you can confirm the fix running the latest snapshot build.