Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support S3 interface endpoints (AWSPrivateLink for Amazon S3) #11735

Closed
cyberduck opened this issue Jul 18, 2021 · 10 comments
Closed

Support S3 interface endpoints (AWSPrivateLink for Amazon S3) #11735

cyberduck opened this issue Jul 18, 2021 · 10 comments
Assignees
Labels
enhancement fixed s3 AWS S3 Protocol Implementation
Milestone

Comments

@cyberduck
Copy link
Collaborator

cyberduck commented Jul 18, 2021

068bb52 created the issue

S3 interface endpoints enable to connect to Amazon S3 using a private IP address:

I am unable to connect to Amazon S3 using the interface endpoint URL (e.g. vpce-0971cacd1f2xxxxxxxxx.s3.eu-west-1.vpce.amazonaws.com) as the server hostname. Cyberduck continously tries to authenticate (I see thousands of packets in Wireshark) and fails a few minutes later. The issue comes from how Cyberduck generates the SigV4 signature, because it considers that "vpce" is the region (e.g. HTTP header Authorization is AWS4-HMAC-SHA256 Credential=AKIASFI36Y5VXXXXXXX/20210702/vpce/s3/aws4_request which fails).

I think that two things should be corrected in Cyberduck:

  • Fetch the region differently from the server endpoint URL
  • Consider S3 interface endpoint URL as "special URL" and use this URL only (don't use dualstack or North Virginia as the default region to list existing S3 buckets)

As a workaround, I was able to connect to a S3 interface endpoint by:

  • Resolving s3.eu-west-1.amazonaws.com to one of the private IP addresses of the interface endpoint (added an entry in the hosts file)
  • Applying the default parameters:
s3.bucket.virtualhost.disable=true
s3.endpoint.dualstack.enable=false
s3.endpoint.format.ipv4=s3.eu-west-1.amazonaws.com

However, it would be great if Cyberduck could natively support S3 interface endpoints, without all these tricks.

@cyberduck
Copy link
Collaborator Author

cyberduck commented Jul 30, 2021

@dkocher commented

We already check the hostname using the regular expression ([a-z0-9\-]+\.)?s3(\.dualstack)?(\.[a-z0-9\-]+)?\.amazonaws\.com and thus no usage of the dual stack endpoints should apply.

@cyberduck
Copy link
Collaborator Author

cyberduck commented Jul 30, 2021

@dkocher commented

Dependent on upstream changeset to detect region.

@cyberduck
Copy link
Collaborator Author

cyberduck commented Jul 30, 2021

@dkocher commented

Existing documentation to disable the use of virtual host style requests in connection profile.

@cyberduck
Copy link
Collaborator Author

cyberduck commented Jul 30, 2021

@dkocher commented

Regional DNS names include a unique VPC endpoint ID, a service identifier, the AWS Region, and vpce.amazonaws.com in its name. For example, for VPC endpoint ID vpce-1a2b3c4d, the DNS name generated might be similar to vpce-1a2b3c4d-5e6f.s3.us-east-1.vpce.amazonaws.com.

@cyberduck
Copy link
Collaborator Author

cyberduck commented Aug 3, 2021

@dkocher commented

In 4ef31c5 with upstream fix. Please try the latest snapshot build with this connection profile.

@cyberduck
Copy link
Collaborator Author

cyberduck commented Aug 5, 2021

068bb52 commented

Hello,

Many thanks for the quick update! I tested with Access Key and Secret and it works. However I tried to create a mix profile with STS credentials and Private Link, and it does not work. The profile is:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "_Apple_DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
    <dict>
        <key>Protocol</key>
        <string>s3</string>
        <key>Vendor</key>
        <string>s3-sts-privatelink</string>
        <key>Scheme</key>
        <string>https</string>
        <key>Description</key>
        <string>S3 (STS and Private Link)</string>
        <key>Default Nickname</key>
        <string>S3 (STS and Private Link)</string>
        <key>Username Placeholder</key>
        <string>Profile Name in ~/.aws/credentials</string>
        <key>Password Configurable</key>
        <false/>
        <key>Token Configurable</key>
        <false/>
        <key>Anonymous Configurable</key>
        <false/>
        <key>Properties</key>
        <array>
            <string>s3service.disable-dns-buckets=true</string>
            <string>s3.endpoint.dualstack.enable=false</string>
        </array>
    </dict>
</plist>

It works when the server is s3.amazonaws.com, but I get an error message "The AWS Access Key Id you provided does not exist in our records". It seems like it does not use STS credentials.

Is that the expected behavior?
Thanks!

@cyberduck
Copy link
Collaborator Author

cyberduck commented Aug 5, 2021

@dkocher commented

Replying to [comment:13 malaval]:

It works when the server is s3.amazonaws.com, but I get an error message "The AWS Access Key Id you provided does not exist in our records". It seems like it does not use STS credentials.

Is that the expected behavior?
Thanks!
This requires an additional fix as we currently handle the VPC endpoints as non AWS S3.

@cyberduck
Copy link
Collaborator Author

cyberduck commented Aug 6, 2021

@dkocher commented

Additional fix in a910429.

@cyberduck
Copy link
Collaborator Author

cyberduck commented Aug 10, 2021

068bb52 commented

Thank you! I don't manage to get the last release work. With the latest snapshot build (35295), I tried to both approaches and none of them worked for me. What should be the cyberduckprofile to use STS credentials with VPC endpoints?

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "_Apple_DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
    <dict>
        <key>Protocol</key>
        <string>s3</string>
        <key>Vendor</key>
        <string>s3-privatelink</string>
        <key>Scheme</key>
        <string>https</string>
        <key>Description</key>
        <string>AWS PrivateLink for Amazon S3 (VPC endpoint)</string>
        <key>Hostname Configurable</key>
        <true/>
        <key>Port Configurable</key>
        <true/>
        <key>Username Placeholder</key>
        <string>Profile Name in ~/.aws/credentials</string>
        <key>Password Configurable</key>
        <false/>
        <key>Token Configurable</key>
        <false/>
        <key>Anonymous Configurable</key>
        <false/>
        <key>Properties</key>
        <array>
            <string>s3service.disable-dns-buckets=true</string>
            <string>s3.endpoint.dualstack.enable=false</string>
        </array>
    </dict>
</plist>

Or

<!DOCTYPE plist PUBLIC "_Apple_DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
    <dict>
        <key>Protocol</key>
        <string>s3</string>
        <key>Vendor</key>
        <string>s3-token</string>
        <key>Description</key>
        <string>S3 (Credentials from AWS Security Token Service)</string>
        <key>Default Nickname</key>
        <string>S3 (Credentials from AWS Security Token Service)</string>
        <key>Username Placeholder</key>
        <string>Profile Name in ~/.aws/credentials</string>
        <key>Password Configurable</key>
        <false/>
        <key>Token Configurable</key>
        <false/>
        <key>Anonymous Configurable</key>
        <false/>
        <key>Properties</key>
        <array>
            <string>s3service.disable-dns-buckets=true</string>
            <string>s3.endpoint.dualstack.enable=false</string>
        </array>
    </dict>
</plist>

@cyberduck
Copy link
Collaborator Author

cyberduck commented Aug 12, 2021

@dkocher commented

Can you please attach the debug log output for the connection attempt. To enable debug logging open a Terminal.app window and enter

defaults write ~/Library/Preferences/ch.sudo.cyberduck.plist logging debug

Log output can be found in the cyberduck.log file in~/Library/Logs/Cyberduck. You can easily reach this file in Console.app (Open from /Applications/Utilities) under Reports → Log Reports → cyberduck.log.

@iterate-ch iterate-ch locked as resolved and limited conversation to collaborators Nov 27, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement fixed s3 AWS S3 Protocol Implementation
Projects
None yet
Development

No branches or pull requests

2 participants