Add support for AWS GovCloud region and add missing ap-northeast-2c AZ

[JoshRosen]

This PR adds support for the AWS GovCloud region and adds the ap-northeast-2c AZ (which was missing).

GovCloud uses completely separate credentials from standard AWS, so the reservation capacity / utilization pollers will not be able to access it. GovCloud accounts are associated with standard AWS accounts for billing purposes: their usage appears as line items in a standard AWS account's bill and thus can show up in ICE reports, so it's necessary to add this region in order to be able to properly group costs by region.

See https://docs.aws.amazon.com/govcloud-us/latest/UserGuide/usage-and-payment.html for more details on GovCloud billing and payment.

[nfonrose]

Hi Josh. We haven't finalized the Travis setup yet; this is the cause of the build failure (not your code)

[nfonrose]

Regarding the PR itself, can you confirm that the intended use-case is to

• run an Ice instance outside GovCloud
• feed it with a mix of GovCloud and non-GovCloud billing files (all located outside GovCloud)

[JoshRosen]

That's correct. Our GovCloud account's billing data is reported via a standard AWS account. That standard AWS account, in turn, is a member of an AWS Organization which contains other standard AWS accounts, so our billing data buckets in S3 capture billing data from both standard and GovCloud regions. We run ICE outside of GovCloud.

[nfonrose]

Ok, great. Can you contribute a small anonymized extract of your GovCloud billing file, with the file name unchanged, expect for the account number that you can obviously hide with 123456...9 ? This is to enable a more complete test of the PR (FilePoller, ...).

[JoshRosen]

In my case I am processing programmatic billing reports produced by the root account in a consolidated billing hierarchy, so in S3 we have files from a single account, e.g. <ROOT-ACCOUNT-ID>-aws-billing-csv-2017-06.csv.

Within the billing file, the GovCloud usage appears as usage under the associated standard AWS account. For example, if I have a staging account in standard AWS and a staging-govcloud account in GovCloud then usage incurred in my staging-govcloud will appear as line items claiming that the usage occurred in my standard staging account and the details of those usage items will claim that the usage occurred in a GovCloud region.

As a concrete example, consider data transfer from standard AWS to an EC2 instance running in GovCloud. In this case, it shows up as a AWSDataTransfer billing entry, the exact same as it would be for transfer between regular AWS regions, except it will appear as USW2-UGW1-AWS-Out-Bytes (note UGW1 region short code here) and the rate will be something like "$0.02 per GB - US West (Oregon) data transfer to AWS GovCloud (US)".

Similarly, for KMS you see things like us-gov-west-1-KMS-Keys instead of ap-southeast-1-KMS-Keys, but otherwise the formats of the billing records are the same.

I hope this provides enough detail. I'm not sure that I can provide any more specifics, such as anonymized billing data extracts, but I don't think that will be necessary because this should be equivalent to the changes for adding a new standard AWS region.

[nfonrose]

Thanks for the details!

Here is why I was asking for an anonymized data line: we plan to add an automated test suite inside Ice. The way it works is be pretty classic:

• controlled inputs in the form of 'artificial' billing files illustrating various situations
• expected outputs for various requests, in the form of the CSV file that you get when you click the Download button inside Ice after you've executed that request.
• and a 'run in self-check mode' option that will run Ice and let it auto check itself using the controlled inputs and expected output.

This way, people will be free to make contributions with a good insurance that nothing has been broken outside the specific area where the modification has been made.

[JoshRosen]

Got it. Are there any examples of that style of test data in the current repository? I'm thinking that it might be easiest to take a GovCloud line from my bill, note the GovCloud-specific parts (e.g. USW2-UGW1-AWS-Out-Bytes, as mentioned above), and then copy and modify the existing standard AWS test data already present in ICE to produce GovCloud test data.

[nfonrose]

Are there any examples of that style of test data in the current repository?

There aren't any yet. We're working on it.

It takes time because we want to make this integrated with our own internal test suite for the SaaS service. And there are many differences between "OSS Ice" and our own internal fork, which makes this integration not so obvious.

Making these two tests suites work in the same way has the obvious advantage of ensuring the test suite for OSS is going to be well taken care of in the long run.