Migrate all ISRG owned storage to GCP cloud storage

[tgeoghegan]

We need to move our entire deployment, including all the buckets we use as mailboxes, to GCP for budget reasons. This will entail some protocol changes because we will have to figure out how Apple will authenticate, and what parameters we need to discover from them to be able to configure the ingestion buckets (probably a GCP service account).

[winstrom]

Apple can provide a GCP account on its global manifest.

Our ingestor global manifest will have both an AWS account and a GCP account.

Maybe we can make it generic if we add to the data share processor specific manifests to include fields that details the protocol to use (AWS or GCP) to access the bucket and the account to use (AWS or GCP)

For Example:

{
	"format": 0,
	"ingestion-bucket": "us-west-1/prio-demo-gcp-test-pha-1-ingestor-1-ingestion",
	"ingestion-account":"aws-iam-entity" or "google-service-account",
	"ingestion-protocol":"AWS_S3" or "GCP_Cloud_Storage",
	"peer-validation-bucket": "us-west-1/prio-demo-gcp-test-pha-1-ingestor-1-peer-validation",
	"batch-signing-public-keys": {
    		"demo-gcp-test-pha-1-ingestor-1-batch-signing-key": {
        		"public-key": "-----BEGIN PUBLIC KEY-----\nMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEAlTcegCjPeFj4S1jHTDqHf8Zk56/\nWGVRHpx4wp6h/cRDq9JO+CTjuot/Z1S1bgcmpeB2ab6OKfiQpuFIcz/aqA==\n-----END PUBLIC KEY-----\n",
        	"expiration": "2021-01-15T18:53:20Z"
    		}
	},
	"packet-encryption-certificates": {
    		"demo-gcp-test-pha-1-ingestion-packet-decryption-key": {
        		"certificate": "TODO get certificate"
    		}
	}
}

Copied from https://docs.google.com/document/d/1MdfM3QT63ISU70l63bwzTrxr93Z7Tv7EDjLfammzo6Q/edit?ts=5f90e408#heading=h.f7uu5u7i5wfc with two extra keys: {"ingestion-account","ingestion-protocol"}

[tgeoghegan]

I agree that the data share processor needs to advertise what kind of storage it uses. I might nest the structure a little. I don't think we need ingestion-account on the data share processor side, because the way the ingestion server authenticates will be implied by ingestion-protocol. If an ingestor sees that a data share processor manifest advertises an AWS_S3 bucket, Apple knows it needs to use its AWS credential, and Google knows it needs to use the service account that has been allowed to federate. If the manifest advertises GCP_Cloud_Storage, Apple needs to use its GCP credential and Google still uses its service account.

Correspondingly, we need to amend the schema for the ingestion server global manifest so its server-identity structure can contain google-service-account-email, alongside aws-iam-entity (in the Apple case where you use your AWS account to write to NCI's S3 buckets) or alongside google-service-account (in the Google case where they use OIDC Web Identity to write to NCI's S3 buckets). It seems silly but I'm pretty sure the Google ingestor manifest needs both the GCP SA email (google-service-account-email) and the numeric account ID (google-service-account), even though both refer to the same entity.

[tgeoghegan]

#181 mostly addresses this, but we need to make sure the manifest format changes that PR implemented are OK with everyone. I will drive that discussion when everyone is back from Thanksgiving break next week.

[tgeoghegan]

We did a successful integration test today and Apple signed off on the format changes. This is donezo.