provide support for private rack servers

[cleblanc87]

Proposal for private rack hosts, this PR adds the --private-resources to install, which when present will:

• require 3 additional subnet parameters, --subnet0-private-cidr --subnet0-private-cidr --subnet0-private-cidr
• place the convox ecs hosts in the specified private subnets
• provision a private elb for the convox api
• provision 3 additional subnets, route tables, & routetableassociates
• provision 3 nat instances, one per az
• configure private networking according to http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Scenario2.html
• cause all elbs created to be internal

example usage:

convox install  \
--vpc-cidr "10.0.5.0/24" \
--subnet0-public-cidr "10.0.5.192/28" \
--subnet1-public-cidr "10.0.5.208/28" \
--subnet2-public-cidr "10.0.5.224/28”\ 
--private-resources \
--subnet0-private-cidr "10.0.5.0/26" \
--subnet1-private-cidr "10.0.5.64/26" \
--subnet2-private-cidr "10.0.5.128/26"

The smaller /28 public subnets will be used to host the nats for ecs hosts in the private /26 networks.

Still left todo:

• work on security groups
• wait for private network connection before logging in to rack during install (Currently the installation will hang waiting for the elb, until the vpc is peered or some other access is configured (vpn etc))
• access options (opposed to peering - vpn or bastion host) I have been using https://docs.openvpn.net/how-to-tutorialsguides/virtual-platforms/amazon-ec2-appliance-ami-quick-start-guide/ for access, but will be relying on vpc peering for actual deployment

[csquared]

You guys think we can close this for the time being?

[cleblanc87]

I would love to keep this alive and get this feature added to Convox if possible. Are there any design changes you would like to see, or is this not something that you see being merged?

@ddollar did mention refactoring the nat setup to use VPC Nat Gateways, however that would require a custom cloudformation resource at this time, something I haven't had time to tackle.

[csquared]

ok yeah so this PR works, it just doesn't use NAT boxes?

[nzoschke]

This is a really promising PR, I definitely see getting it into the project.

+1 on the Nat Gateway too. Custom CloudFormation resource is the way to go there. I can help with that part.

[nzoschke]

Private Rack Servers

[nzoschke]

First tactical question... Can we split this up into smaller chunks to get through somehow? Some strawmen sub projects:

• Parameter to make all ELBs internal
• Parameters to provision the private subnets, route tables, and associations
• Custom resource for VPC NAT Gateways
• ...

Ideally the smaller steps will have some value in their own. But even if they don't, smaller PRs are much easier to manage.

[nzoschke]

Here's some WIP for the Nat Gateway Custom Handler: #338

I'd like some review first, but it'll be safe to get the custom handler bit into master and released first.

[nzoschke]

The NAT handler is released.

I tried to rebase this PR onto master and had some challenges / questions...

One is around ELB public/private logic. Currently we use the manifest port mapping to decide if an ELB is public (80:80) or private (6379). In this mode should all ELBs be private or will there still be some way to make one public?

I also had a general question around CF dependencies. Some resouces explcitly depend on the subnet or gateway to guarantee that we remove things in the correct order on uninstall. How does DependsOn work with conditional resources?

[cleblanc87]

In regards to ELBs, the intent was to make them all private, as our use case is a purely private rack using a VPN connection for access.

I think the all of the resources that are depending on conditional resources are conditional themselves using the same condition logic, was there a specific resource you were referring to?

[rgarcia]

FWIW in our environment we have a mix of public and private ELBs. Public ELBs for services we expose to the internet, like our website and public APIs, and private ELBs for internal services.

[cleblanc87]

Our intent is to run both a fully private rack and a fully public rack instead of sharing one rack for both, but perhaps convox should provide the flexibility for both approaches?

[rgarcia]

I should also mention that the instances that back a public ELB are still in the private subnet, which gives us some peace of mind security-wise (the only things in the public subnet are ELBs).

[arikfr]

My use case is similar to @rgarcia's.

[nzoschke]

I'm still learning about all the VPC options, but a picture is coming into focus.

--private adds a 2nd network but retains the public network for internet-facing ELBs.

Rack

• --private flag creates an additional 3 private subnets, gateways and gateway attachments. Default values are 10.0.4.0/24, 10.0.5.0/24, 10.0.6.0/24
• --subnet{0,1,2}-private-cidr option to customize private subnets values
• --private flag creates 3 NAT gateways, routes, and elastic IPs
• when --private, instances and convox ELB are placed in the private subnet

Apps

• create and set Private Yes/No Parameter on app template
• always create ELB in public subnet if service port mapping indicates public (80:80)
• conditionally create ELB in private subnet if service port mapping indicates private (80) and Private=Yes

Docs

• document VPC peering
• document VPN access
• document Grid access?

[nzoschke]

@cleblanc87 can you use the existing app private port mapping option (single port 80, vs 80:80) to tell your apps to use the private network? If so we continue to use 80:80 to signify making public ELBs

And would you be ok with the public network still being there?

[ddollar]

looks good, though maybe consider using --private and Private and drop the resources noun

[nzoschke]

References:

http://stackoverflow.com/questions/22541895/amazon-elb-for-ec2-instances-in-private-subnet-in-vpc
http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/elb-internet-facing-load-balancers.html

[cleblanc87]

The checklist looks perfect, and the public networking is required by the original PR for the NAT instances, so no concerns there :)

The single port mapping will work great for us, and using the 80:80 syntax to denote public makes sense as well.

[nzoschke]

Moved this to #350 so I can work off the convox private-resources branch. I am probably doing something not right on GitHub here, apologies.