data/aws/vpc: Add an S3 endpoint to new VPCs

[wking]

As suggested by @cuppett, this allows registry <-> S3 transfers to bypass the (NAT) gateways. Traffic over the NAT gateways costs money, so the new endpoint should make S3 access from the cluster cheaper (and possibly more reliable). This also allows for additional security policy flexibility, although I'm not taking advantage of that in this commit. Docs for VPC endpoints are here, here, here, and here.

Endpoints do not currently support cross-region requests. And based on discussion with @cuppett, adding an endpoint may break access to S3 on other regions. But I can't find docs to back that up, and the docs say:

We use the most specific route that matches the traffic to determine how to route the traffic (longest prefix match). If you have an existing route in your route table for all internet traffic (0.0.0.0/0) that points to an internet gateway, the endpoint route takes precedence for all traffic destined for the service, because the IP address range for the service is more specific than 0.0.0.0/0. All other internet traffic goes to your internet gateway, including traffic that's destined for the service in other regions.

which suggests that access to S3 on other regions may be unaffected. In any case, our registry buckets, and likely any other buckets associated with the cluster, will be living in the same region.

concat is documented here.

[abhinavdahiya]

https://www.terraform.io/docs/providers/aws/r/vpc_endpoint.html#subnet_ids vs https://www.terraform.io/docs/providers/aws/r/vpc_endpoint.html#route_table_ids

[wking]

Only route_table_ids applies to Gateway mode.

[abhinavdahiya]

Only route_table_ids applies to Gateway mode.

Oh, and Gateway is deafult https://www.terraform.io/docs/providers/aws/r/vpc_endpoint.html#vpc_endpoint_type 👍

[abhinavdahiya]

Is there a precedent for this kind of setup in OpenShift @cuppett ?

[cuppett]

Not in our hosted environments. Those and our reference architectures have historically taken a much more vanilla approach to each cloud.

Using ALBs, NAT gateways and VPC endpoints is new and applies optimizations and competencies to this cloud. Excited to bring more maturity to our offering. All are relatively new within AWS (last couple years) and is helping to catch us up.

[abhinavdahiya]

@dgoodwin not sure who keeps track of all these unmanaged things. heads up.

/approve

[dgoodwin]

Will this come out tagged with the expected tags?

And I assume we need to add VPC endpoint cleanup to hiveutil's code.

[wking]

Will this come out tagged with the expected tags?

At least Terraform does not seem to allow tagging for this resource. It may be untaggable in AWS.

And I assume we need to add VPC endpoint cleanup to hiveutil's code.

Maybe it doesn't block VPC deletion and is automatically removed when the VPC is deleted (in which case we would not need Hive changes). I'll see if I can tell from the CI logs.

[wking]

I'll see if I can tell from the CI logs.

The logs won't be particularly helpful until #730 lands. I'll circle back to this after that happens.

[abhinavdahiya]

The logs won't be particularly helpful until #730 lands. I'll circle back to this after that happens.

deletion doesn't need that right?

[wking]

The logs won't be particularly helpful until #730 lands. I'll circle back to this after that happens.

deletion doesn't need that right?

No, but we're not going to see anything about this endpoint in the deletion logs. What I want is to see creation (and ideally an ID) for the new resource, and then I'll check to make sure it's gone after CI wraps up.

[abhinavdahiya]

/hold
need 👍 from hive team @joelddiaz

[joelddiaz]

I looked at this a bit, and adding the endpoint does case panics in the uninstall due to some assumptions about the kinds of default fields found in a route table (easy fix).

You can't leave the endpoint hanging around as that will block the VPC from ever being deleted. Unfortunately, the endpoints don't appear to be taggable objects, so we'll need to add a dependency in the VPC deletion path to find any endpoints associated with the VPC being deleted.

[joelddiaz]

FWIW, we've got the endpoint deletion merged now. so we're ready for this on the uninstall side of things openshift/hive#122

[wking]

I've rebased onto master and pulled in openshift/hive#122 with 1a1dc49 -> e205898.

[openshift-ci-robot]

@wking: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
ci/prow/e2e-libvirt	1a1dc49	link	/test e2e-libvirt

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[abhinavdahiya]

I've rebased onto master and pulled in openshift/hive#122 with 1a1dc49 -> e205898.

/lgtm

/retest

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abhinavdahiya, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [abhinavdahiya,wking]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[abhinavdahiya]

/hold cancel