thin-egress-app-IamPolicyDownload with AWS SourceIPs is too large in us-east-1

[philvarner]

I apologize in advance for what may be a confusing or incorrect bug report, as I'm pretty new to working with all of these software components, but I think my analysis of this is correct.

The problem we're seeing is this error when the lambda that updates the role policy runs:

There was a problem updating policy my-app-thin-egress-app-IamPolicyDownload for Role my-app-thin-egress-app-DownloadRoleInRegion in region us-east-1: An error occurred (LimitExceeded) when calling the PutRolePolicy operation: Maximum policy size of 10240 bytes exceeded for role my-app-thin-egress-app-DownloadRoleInRegion

This is part of a deployment of https://github.com/nasa/cumulus

I believe the issue is that the policy condition to allow all AWS IPs is too large in us-east-1. Looking at the number of ip ranges for each of these regions (there are 6 lines in this output for each CIDR block), us-east-1 has nearly 1000, while the others have less than half as many. I think this results in a policy that's more than 10k characters and exceeds the AWS limit.

$ jq '.prefixes[] | select(.region=="us-east-1")' < Downloads/ip-ranges.json | wc
    5262    8770  110088
~ via  v14.17.5 on :a:
$ jq '.prefixes[] | select(.region=="us-west-2")' < Downloads/ip-ranges.json | wc
    2292    3820   48248
~ via  v14.17.5 on :a:
$ jq '.prefixes[] | select(.region=="us-east-2")' < Downloads/ip-ranges.json | wc
    1890    3150   39043

[philvarner]

I manually compared the size of the policy we have from a Cumulus instance that was created 2021-04-19, though I don't know the last time it was updated by the lambda. That policy is 9001 characters, and a policy with all of the IPs listed for us-east-1 in https://ip-ranges.amazonaws.com/ip-ranges.json would be 16585, so well above the limit of 10k.

[bbuechler]

I'll have to look into this more, but I know we don't just filter by region, we also filter by service... We select only the root AMAZON level since all the other services would by definition fall under those.

Selecting only AMAZON cuts the number of CIDR's by ~50%:

$ curl -s https://ip-ranges.amazonaws.com/ip-ranges.json  |  jq '.prefixes[] | select(.region=="us-east-1") | select(.service=="AMAZON").ip_prefix' | wc -l
     570
     
$ curl -s https://ip-ranges.amazonaws.com/ip-ranges.json  |  jq '.prefixes[] | select(.region=="us-east-1").ip_prefix' | wc -l
     877
     

If 570's CIDR's genuinely don't fit into a single IAM policy, and we can't trim any more fat from cidr list (nothing jumps out at me right now), suppose the only solution to get TEA into us-east-1 would be to split up the policy up. OTOH, its my understanding that all EOSDIS data supposed to be in us-west-2?

[philvarner]

I wasn't filtering AMAZON when i did this the first time, but I just re-executed in and replaced the policy with the one we created in April 2021, and it's 10935 bytes (ignoring whitespace). "Role policy size cannot exceed 10,240 characters." from https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_iam-quotas.html

[philvarner]

OTOH, its my understanding that all EOSDIS data supposed to be in us-west-2?

I believe so. I'm not sure why we deployed this in us-east-1 to begin with, but it may have just been that most of our work was deployed there by default, as now we're moving most of it to us-west-2. So, the workaround was originally to deploy this to us-west-2, but I think that's what we really want to do anyway even if this wasn't a problem.