final boss: defeating the 200 resource limit of CloudFormation

[brianleroux]

Architect 5.x and lower applications utilized resource namespacing for discovery and, as such, was only limited by each individual AWS service quota. Architect 6.x and higher is based on CloudFormation which only allows 200 top level resources. This has made upgrading impossible for some folks and we need to bring them with us! This problem is compounding with time and as we add more pragmas to Architect hitting the 200 resource limit is becoming very easy to do. Said plainly, having one deployment document/artifact isn't scaling.

There is one way to solve the 200 resource limit: keep your stacks under 200 resources! You can achieve this goal by separating stacks and passing parameters into them at authortime OR nesting stacks and passing resource references at during deployment 'runtime'.

Both approaches will manifest as extra calls to aws cloudformation package and aws cloudformation deploy.

Stack separation is pretty commonly done by loosely grouping resources by lifecycle: network, compute, storage and governance. Compute changes all the time. Database tables and URLS less so. You also probably want different retention policies on storage. SNS topics aren't that important but DNS records sure are. Etc.

An early prototype of this functionality separated stacks by pragma. This worked reasonably well but created a completely new code path for deployment which predictably fell by the wayside as new functionality came on board. There should be only one way to do this and predictably the simple way won out when there wasn't. So regardless of nesting stacks or separating stacks it will have to be the only way this works (or the same thing will happen again).

So this is a rewrite of architect/package and architect/deploy which isn't daunting but worth planning very carefully! The first decision, I think, is to figure out if we are separating stacks (and doing resource discovery via the SDK and passing values at authortime) or nesting stacks (and doing everything by implicit reference). I'm totally unsure which direction is better…both are fine and both have tradeoffs. So this issue is to solicit feedback and input before we start working on this. Thx!

[ryanblock]

Thoughts / tradeoffs to consider:

Speed
Between nested stacks, separate stacks, single stack, how fast will common operations take by comparison to each other? For example, if using separate stacks makes deploys go from ~30s to ~300s+ (perhaps because we have to call into the stack after it's finished deploying to get refs for other stacks to use), that would be a strong reason to avoid.

Non-fungible resources
Many resources are completely fungible – we can destroy and spin up Lambdas all day long, because they're referenced internally to the services they're wired to. But if an existing app were to have its API Gateway redeployed in a new stack, that Gateway would get a new URL. Arc resources I can think of that are non-fungible in this way:

• @http - breaking out into a new template would create a new APIG URL
• @ws - breaking out into a new template would create a new APIG URL
• @static - breaking out into a new template would create a new S3 bucket/URL
• @tables - breaking out into a new template would create a new table, and presumably lose the existing table data
• @indexes? - I'm actually not sure if indexes would be impacted
• Any others?

So anyhow, ideally whatever changes we make will have no impact on non-fungible resources for existing applications (kind of like how REST API users automatically had their API type detected to prevent us from deploying over them with HTTP APIs in Arc 7).

Macros
How will Macros work if, in theory, we are now shipping multiple CloudFormation templates?

[brianleroux]

Role and SSM params for resource name discovery are top level too. Tho SSM params can go anywhere. Ideally whatever strategy we use to defeat this limit isn't too specialized and itself can scale horizontally. That said, each pragma type has completely different wiring requirements so maybe we just bite that bullet.

sam.json parent stack (retaining top level references to static, ws, http, tables, indexes and Role)

• sam-http.json (all the lambda resources up to 100 api gateway route lambdas and permissions)
• sam-events.json (ssm, topics and lambdas/permissions)
• sam-queues.json (ssm, queues and lambdas/permissions)
• sam-scheduled.json (ssm, eventbridge rule, lambda and permission)
• sam-domains.json (route53, cloudfront, acm)

A second iteration of this could allow each type to horizontal scale out. So if you have an API Gateway with 250 routes you'd end up with sam.json, sam-http-0.json and sam-http-1.json smearing the routes over multiple nested stacks. This blows out the limits for the other pragmas to their account quota max too. Ideally we'd get tables into a nested stack but this would blow up folks using Architect >=6.x so we don't want that.

WDYT?

[ryanblock]

Update: last week CloudFormation unexpectedly upped the resource limit to 500 resources! This is the thing we've been waiting for, and should cover us in almost all apps. I'm going to close this issue (and destroy the long out of date nested cfn code path in package next time I'm in there).

Should this still be an issue for folks in the future, please give us a shout out and we'll reopen!