[BUG][Functional] S3 Interface Endpoint missing wildcard A record

[ColinL2021]

Bug reports which fail to provide the required information will be closed without action.

Required Basic Info

• Accelerator Version: v1.5.5 (Affects All)
• Install Type: All
• Upgrade from version: N/A

Describe the bug
After enabling the Amazon S3 Interface Endpoint in the SharedNetwork account as you would add other interface endpoints. Access to Amazon S3 buckets will fail due to a DNS no such host error. This will prevent you from creating a Systems Manager Session Manager session to an EC2 instance because of logging enabled to an Amazon S3 bucket that can no longer be located. You can still access the Amazon S3 buckets by explicitly specifying the S3 interface endpoint as the endpoint-url.

Failure Info

• What error messages have you identified, if any:
• When trying to access Amazon S3 buckets you will get a "no such host" or "server can't find "
• What symptoms have you identified, if any:
• DNS lookups to Amazon S3 buckets in the region will fail

Required files

• Please provide a copy of your config.json file (sanitize if required)
  "interface-endpoints": { "subnet": "Endpoint", "endpoints": [ "ec2", "ec2messages", "ssm", "ssmmessages", "secretsmanager", "cloudformation", "kms", "logs", "monitoring", "s3" ] },

Steps To Reproduce

1. Add the "s3" endpoint to "interface-endpoints" section in the SharedNetwork account
2. Run the State Machine
3. Shortly after the execution when DNS is refreshed and starting to use your Private Hosted Zone for your Amazon S3 Interface Endpoint you will get the error when trying to access Amazon S3 buckets.
4. You can try to connect to a EC2 instance via Session Manager, or login to an EC2 instance and try to access a Amazon S3 bucket.

Expected behavior
A clear and concise description of what you expected to happen.
Expect to be able to access Amazon S3 buckets per normal via Gateway Endpoints, and now also via Interface Endpoints

Screenshots
If applicable, add screenshots to help explain your problem.
Private Hosted Zone missing the Wildcard A record to reference buckets within the domain.

Additional context
Solution is to add a Wildcard A record pointing to the CNAME of the Amazon S3 Interface endpoint. More information can be found on this blog: https://aws.amazon.com/blogs/networking-and-content-delivery/secure-hybrid-access-to-amazon-s3-using-aws-privatelink/

Specifically this image describes what is missing from the Private Hosted Zone:

Expected Private hosted zone would look like this:

[Brian969]

Did you remove the S3 gateway endpoint before attempting this?

[ColinL2021]

I did not, but now I realize this change will circumvent the Gateway Endpoints entirely and only use the S3 Interface endpoints. With only Gateway Endpoints the resolution of Amazon S3 buckets will be to the S3 public IP addresses then the routing prefix of the vpc will direct the traffic to S3 over the Gateway Endpoints. With the Private Hosted Zone the resolution of Amazon S3 buckets will be the S3 interface endpoints. Since the routing prefix for these private S3 interface endpoints will not match, the traffic within your vpc will go through the S3 Interface endpoints. Not the intended solution. Intended purpose was to use Gateway endpoints for traffic within a VPC and S3 interface endpoints for on-prem.

Couple of possible options would be:

1. Not create the Private hosted zone and have onprem DNS resolve to the S3 interface endpoint directly
2. Have some resolution condition on source
3. Update the prefix for Gateway Endpoint routing

[archikierstead]

Documented workaround in Issue.