Adding VPN restriction capability to MCR

[dwhitestratiform]

Description

This pull request is to lock down non production environments for MCR to be only accessible behind CMS VPN. In addition to locking down the environments to be accessible behind VPN changes were required to temporarily enable access for GitHub Actions to access the environment to run tests.

Because the serverless environment uses AWS Cloudfront we cannot place Cloudfront inside of a private subnet. Therefore, to restrict traffic the AWS WAF is being utilized to lock down access. To accomplish this we're beginning to utilize the CMS WAF Serverless plugin to apply standard rule sets and extend that by applying custom IP sets.

This PR was also needed
#11602

This pull request does the following:

Implements the CMS WAF Plugin for both the cloudfront distro as well as the api
Creates a prioritized rule set for the WAF
Creates a custom IP set per stack
Associates a manually created global IP (for both ipv4 and ipv6) set that houses all CMS VPN IP addresses to the stacks WAF
Looks up GitHub Runner IP's and adds them to the stacks custom IP set (that is only applied to that stack)
Upon successful run of all tests clears out the GitHub Runner IP's from the stacks custom IP set

Related ticket(s)

https://jiraent.cms.gov/browse/CMDCT-3242)

How to test

To test locally connect to Zscaler or Cisco Any Connect:
In an incognito window navigate to the endpoint of the environment and you should see the login page for MCR and should be able to login and use the app as expected
Disconnect from Zscaler and navigate to the same endpoint in a new incognito window and you should receive a 403.

Important updates

When creating new projects or migrating this work to other products we will need to manually create the CMS VPN IP set for both ipv4 and ipv6. In addition SSM parameters will need to be created containing the arn of each IP set so that they can be pulled into the serverless.yml at deploy time.

• Worth noting: when creating an ipset the ip set gets created by going Into the aws console and going to the waf service -> ipsets -> change region to cloudfront -> create ip set

Additional Details

The script that registers IPs against the IP set assigned to the WAF for the application stack, is designed to be able to be run concurrently while managing race and optimistic concurrency control locks. The new implementation of the web application firewall (version 2) requires setting the entirety of the array in each operation, and does not support append. To avoid concurrency write contention, AWS requires a lock token be supplied in the update operation, such that a read must occur before a write. Console actions will also invalidate a token. Read operations are also throttled, so this script implements an exponential backoff strategy with jitter, such that a random time between min, and the lesser of min^try or max, is used to sleep between each each read, each write, and the entirety of the loop, with short circuits should the number of attempts of any operation exceed the circuit breaker number of attempts. This allows for a relatively high level of concurrency, at the cost of performance with creating the registry. For other implementation patterns, the not fully realized branch, skipci-whitelist-gha-cidrs can be used an example.

Also worth noting the reason that a global vpn ip set and a per stack GitHub runner ip set was chosen was to allow a single place with ideally few updates to control VPN access. Then, for the individual stacks IP set we can simply add the GitHub runner ip's, run tests, then clear out those IP's at the end of the run ensuring that we only apply this to one stack and ensure security across all the stacks.

The script is called as follows:
./waf-controller.sh [set|append] [ip_set name] [ip_set id] [space delimited list of ips or CIDRs]

---

Author checklist

• I have performed a self-review of my code
• I have added thorough tests, if necessary
• I have updated relevant documentation, if necessary

---

convert to a different template: test → val | val → prod

[gmrabian]

didn't fully read the code changes but I know they are being ported.
branch url didn't work in incognito for me while on zscaler
https://d33o8hg6uk6hv9.cloudfront.net/

[dwhitestratiform]

Today @BearHanded and I met with the CMS networking/zscaler team. They let us know that ipv6 is not currently turned on for Zscaler which is why we're seeing our ISP IP come into the waf when having an IPV6 IP. Merging these changes will enable both IPV4 and IPV6 IP whitelisting for MCR. This only affects ephemeral environments for the time being. When the networking team turns on ipv6 for Zscaler we would anticipate this just working .

[gmrabian]

approving knowing that this will be re-verified later

[codeclimate]

Code Climate has analyzed commit ba53a02 and detected 0 issues on this pull request.

The test coverage on the diff in this pull request is 100.0% (90% is the threshold).

This pull request will bring the total coverage in the repository to 96.8% (0.0% change).

View more on Code Climate.