New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multi region formation #95

Open
wants to merge 17 commits into
base: master
from

Conversation

Projects
None yet
3 participants
@rjocoleman
Copy link
Contributor

rjocoleman commented Aug 2, 2015

This is matched with convox-archive/cli#47

Lambda isn't supported in all Regions. Cloudformation CustomResource's must be in the same region as the stack. This is a workaround to allow Convox creation in multiple regions - Lambda will stay in us-east-1

It:

  • Decouples Lambda Cloudformation bridge from the main Cloudformation Stack (so they can be created in different regions).
  • Adds an Input Parameter in the kernel.json cloudformation stack for the Lambda Cloudformation Bridge ARN.
  • Support SNS as a notification source for the Lambda bridge function

TODO:

@rjocoleman rjocoleman referenced this pull request Aug 2, 2015

Closed

WIP: Non Lambda Regions #47

@rjocoleman

This comment has been minimized.

Copy link
Contributor Author

rjocoleman commented Aug 2, 2015

@ddollar

Supporting arbitrary numbers of AZs is proving to be a bit of a pain.

I assume that Custom::EC2AvailabilityZones exists because creating subnets is unreliable and inconsistent so they need to be tested before creation rather than using things like FN::getAZs.

Conditions should be the way forward but they don't support the output of Resources like Custom::EC2AvailabilityZones.

How about me removing the subnet creation from CFN and putting it over in the Lambda bridge i.e.
Custom::EC2Subnet
Return values: Array of created subnet ids and comma separated list.

Custom::SubnetRouteTableAssociation
Return value: Array of resource names

I'd also add an array return value to Custom::EC2AvailabilityZones that can be used inplace of the "hard" coded number of values now.

The goal would be support of subnets in 2-many AZs.

Here are the problematic lines I'm looking to replace with the above output values:
https://github.com/convox/kernel/blob/master/dist/kernel.json#L301
https://github.com/convox/kernel/blob/master/dist/kernel.json#L339
https://github.com/convox/kernel/blob/master/dist/kernel.json#L456-L465
https://github.com/convox/kernel/blob/master/dist/kernel.json#L598-L602
https://github.com/convox/kernel/blob/master/dist/kernel.json#L803

@rjocoleman

This comment has been minimized.

Copy link
Contributor Author

rjocoleman commented Aug 2, 2015

Update: I've got the subnets and route associations creating, updating and deleting dynamically via the Lambda bridge as they should.

I'm having trouble dealing with the variable length input coming back for usage when creating the ELB and the ASG.

I've got a ticket in with AWS support to see if they can shed some light, I've got some other half formed ideas here too so I'll finish it off in the morning.

(I haven't pushed the latest commits because they may change drastically with the solution to this issue)

@rjocoleman

This comment has been minimized.

Copy link
Contributor Author

rjocoleman commented Aug 2, 2015

this is done, workable solution in f7b0b89

Here's what's up:

  • I can't find a way to pass back in an JSON array from the custom resource [ "us-east-1a", "us-east-1b", "us-east-1c" ]
  • Cloudformation doesn't like null or empty values.
  • Cloudformation's conditional features only work with Conditions
  • Conditions can't be used with resources.

So my workaround is:

  1. Specify at least the number of results we want, in this case:
        "Subnets": [
          { "Fn::GetAtt": [ "Subnets", "SubnetId0" ] },
          { "Fn::GetAtt": [ "Subnets", "SubnetId1" ] },
          { "Fn::GetAtt": [ "Subnets", "SubnetId2" ] },
          { "Fn::GetAtt": [ "Subnets", "SubnetId3" ] },
          { "Fn::GetAtt": [ "Subnets", "SubnetId4" ] },
          { "Fn::GetAtt": [ "Subnets", "SubnetId5" ] },
          { "Fn::GetAtt": [ "Subnets", "SubnetId6" ] },
          { "Fn::GetAtt": [ "Subnets", "SubnetId7" ] },
          { "Fn::GetAtt": [ "Subnets", "SubnetId8" ] },
          { "Fn::GetAtt": [ "Subnets", "SubnetId9" ] }
        ]
  1. Have formation return the same number of subnets. Cloudformation deals with duplication but not empty, null or not present values so I set all of these results to the first.
  2. Return the comma separated lists rather than generating them in CFN template.

I've also:

  • Using a more flexible value type on the Response map map[string]interface{} over map[string]string
  • Switched Custom::EC2AvailabilityZones for a Custom::EC2Subnets that creates the subnets based on the AZs that can have subnets created in them.
  • Added Custom::EC2SubnetRouteTableAssociation to allocate those subnets dynamically.
  • Both of these new custom resources also handle deletion but nothing on update (I think this is correct behaviour)
  • Removed AvailabilityZones from AWS::AutoScaling::AutoScalingGroup (it's not required https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-group.html#cfn-as-group-vpczoneidentifier)
  • Dealt with a race condition deleting subnets that didn't appear using CFN directly.

This paired with the CLI PR this works for me deploying (and undeploying) clusters into non-us-east-1 regions (and us-east-1 of course!).

@rjocoleman rjocoleman changed the title WIP: Multi region formation Multi region formation Aug 2, 2015

@ddollar

This comment has been minimized.

Copy link
Contributor

ddollar commented Aug 4, 2015

This is looking really good! I'm going to hold off for 1-2 more days until we get an answer back from our AWS contacts about wider Lambda availability.

@nzoschke

This comment has been minimized.

Copy link
Contributor

nzoschke commented Aug 5, 2015

Word is that there is no specific timeline for Lambda in other regions. So we should proceed with the SNS based workaround to add support for other regions.

@rjocoleman

This comment has been minimized.

Copy link
Contributor Author

rjocoleman commented Aug 5, 2015

Ok, in that case its important to note that this is all held up by: convox-archive/app#11

@rjocoleman rjocoleman changed the title Multi region formation WIP: Multi region formation Aug 6, 2015

@rjocoleman

This comment has been minimized.

Copy link
Contributor Author

rjocoleman commented Aug 6, 2015

@ddollar @nzoschke

Now that multi-region for Lambda regions has hit I'm keen to refactor my PRs to simplify and support non-lambda regions too. (I'm in ap-southeast-2!)

There are two major things to support these:

  1. making custom resources via CFN work in regions we can't access lambda
  2. supporting < 3 subnets (some regions have less than 3 https://aws.amazon.com/about-aws/global-infrastructure/) (this is already in my PRs, just needs the conflicts resolved)

How do you feel about this workflow?

Install:

  • Create a second bootstrap stack (first).
  • This stack is conditionally created in the deploy region if lambda is supported else in us-east-1.
  • This stack makes a bootstrap lambda function in with the same formation.zip.
  • It has a custom resource that returns StackCustomTopic.
  • StackCustomTopic is an ARN to the lambda function, or it's an SNS topic with the bootstrap lambda function subscribed.
  • All the permissions setting and subscriptions happen in this custom resource not the CLI.
  • StackCustomTopic is passed to the main CFN stack and install happens in the selected deployment region.

Applications:

  • CustomTopic as currently implemented is removed from the app template.
  • Replaced with a custom resource that calls StackCustomTopic and returns AppCustomTopic.
  • AppCustomTopic creates a lambda function for the app and conditionally a SNS topic and returns the appropriate ARN.

Installs will end up with 2 cloudformation stacks.
Apps get one each.

@nzoschke

This comment has been minimized.

Copy link
Contributor

nzoschke commented Aug 10, 2015

What is the simplest thing possible for now? Lambda will be available someday.

Can we go back to when installs had an SNS / lambda handler for all Apps?

Can we use deletion policy retain to keep things around through a kernel delete?

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-attribute-deletionpolicy.html

@nzoschke

This comment has been minimized.

Copy link
Contributor

nzoschke commented Aug 11, 2015

To summarize this PR and some chats:

The installer will make two stacks: a Lambda and SNS stack in a supported region (e.g. us-east-1), and a VPC in the desired region (e.g. ap-southeast-2).

The Lambda / SNS topic will be set as a parameter for the VPC stack to use as a custom resource topic.

The Lambda / SNS stack will not be deleted on an convox uninstall, so that app stacks can still use the custom resource handler for deletes through cloudformation.

When Lambda is available in every region, we can re-evaluate a strategy that uses a Lambda function in every stacks.

@rjocoleman rjocoleman force-pushed the rjocoleman:multi-region-formation branch from d475b6a to f1f1d5b Aug 12, 2015

@rjocoleman rjocoleman changed the title WIP: Multi region formation Multi region formation Aug 12, 2015

@rjocoleman

This comment has been minimized.

Copy link
Contributor Author

rjocoleman commented Aug 12, 2015

@nzoschke

I've done the following:

Details

$ convox install:

  1. creates bootstrap stack containing lambda function called bootstrap.
  2. creates SNS topic called BootstrapTopic
  3. subscribes bootstrap to BootstrapTopic.
  4. Creates permissions for BootstrapTopic to invoke bootstrap.

If it's a rack install region is lambda supported then these all exist in that region.
If the rack install region is not lambda supported then bootstrap is created in us-east-1 and BootstrapTopic is created in the rack install region.

This is done outside of CFN and in the cli because a) AddPermission is not supported in CFN and b) the subscription needs to exist before the CFN is run.

There is no teardown handling for bootstrap and BootstrapTopic. I have not changed the name in the later kernel CFN template from CustomTopic but they're the same thing.

kernel

  • kernel.json has had the lambda extracted from it and put into a new bootstrap.json.
  • bootstrap.json has been added to the Makefile.
  • formation has had processing for SNS events added.
  • CustomResources have been created to create subnets and relations to cope with regions that have less than 3 AZs. kernel.json has been updated to fit this.

app

  • Use the single rack BootstrapTopic/CustomTopic` rather than a discrete one per app.

I haven't done any docs but it should be as simple as $ convox install --region ap-southeast-2 where ap-southeast-2 is a non-lambda region.

@nzoschke

This comment has been minimized.

Copy link
Contributor

nzoschke commented Aug 12, 2015

Thanks @rjocoleman. I'll start testing this, convox-archive/app#11, and convox-archive/cli#82 now.

@rjocoleman rjocoleman force-pushed the rjocoleman:multi-region-formation branch from bbd9942 to b4eb570 Aug 22, 2015

}
},
"Bootstrap": {
"Type": "AWS::Lambda::Function",

This comment has been minimized.

@nzoschke

nzoschke Aug 22, 2015

Contributor

I think we want a DeletionPolicy: Retain to not totally orphan app stacks when we remove the kernel.

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-attribute-deletionpolicy.html

@nzoschke

This comment has been minimized.

Copy link
Contributor

nzoschke commented Aug 22, 2015

In trying to install in us-west-1 I got the following error:

Installing Convox...
Lambda is not supported in us-west-1, creating bootstrap in us-east-1:
Created Lambda Function: convox-bootstrap-Bootstrap-1W88F05LOM8RS
Created CloudFormation Stack: convox-bootstrap
Continuing normal install...
Failed AWS::IAM::Role: Resource creation cancelled
Failed AWS::DynamoDB::Table: Resource creation cancelled
Failed AWS::S3::Bucket: Resource creation cancelled
Failed AWS::EC2::VPC: Resource creation cancelled
Failed AWS::DynamoDB::Table: Resource creation cancelled
Failed AWS::IAM::Role: Resource creation cancelled
Failed AWS::S3::Bucket: Resource creation cancelled
Failed Custom::KMSKey: Resource creation cancelled
Failed AWS::Kinesis::Stream: Resource creation cancelled
Failed AWS::EC2::InternetGateway: Resource creation cancelled
Created IAM User: convox-RegistryUser-CQGBQJ9HC7IP
Created IAM User: convox-LogsUser-RV72RP6Z8V7Q
Created IAM User: convox-KernelUser-1X5ELNJ2TQ1KG
Failed AWS::ECS::Cluster: Resource is not supported in this region
Deleted IAM User: convox-RegistryUser-CQGBQJ9HC7IP
Deleted IAM User: convox-LogsUser-RV72RP6Z8V7Q
Deleted IAM User: convox-KernelUser-1X5ELNJ2TQ1KG
ERROR: stack deletion failed

I am reaching out to AWS to see why CloudFormation in us-west-1 doesn't support ECS.

@nzoschke nzoschke referenced this pull request Sep 3, 2015

Open

Multi Region #149

@rjocoleman rjocoleman referenced this pull request Jun 23, 2016

Closed

Support ap-southeast-2 region #806

0 of 11 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment