bulkio: support passing key in query param for google cloud storage

[dt]

Motivation/Context

Currently interacting with files on google cloud storage in backup/restore/import/export requiring using either the implicit "instance" auth (where the file access is done using the vm's auth) or using a key stored in a the cloudstorage.gs.default.key key (identified by auth=default). This setting was devised and named with the intent that we could define other settings to save multiple keys, though support for arbitrary, user-defined settings names does not yet exist. We went with this approach because "key" for gcs access was a large, cumbersome JSON blob that we judged would be clunky to put in a query parameter.

However using settings has some drawbacks -- only the root user can read and write them, and they are global to a cluster -- anyone who can use the cluster can use them. This is less than ideal -- ideally a user who can access a specific GCS bucket should be able to backup or restore to it independently of what a different user, on the same cluster, can do. Making auth per-statement avoids this issue. Most of the time a human isn't running a BACKUP by hand anyway -- a script composes the query by combining variables, so it doesn't matter how large a key is (up to the max URL length).

Plan

1. Allow supplying the gcs key json bundle in as a base64 encoded blob in a query param to bulk IO ops
2. Deprecate the gcs key setting.
3. Add a settings to enable "implicit" / "instance" auth for each SDK where use allow using it, defaulting to disabled, so that simply being allowed to run bulk io ops doesn't automatically allow using the buckets that the machine's account has access to.

[dt]

cc @mberhault @rolandcrosby @kannanlakshmi

[rolandcrosby]

This seems like a good plan. Does 3) mean you're also considering implementing implicit auth for AWS S3 when CockroachDB is running on AWS EC2? We got some questions about this recently from a prospective customer, and it sounded like a relatively simple addition using the ec2rolecreds package in the Go SDK for S3.

[rolandcrosby]

We should also consider how to allow users to pass additional parameters to our various cloud storage options. For example, S3 allows users to configure server-side encryption for PutObject requests via three x-amz-server-side-encryption headers; I'm sure other cloud storage services have similar options that customers will ask us to implement at some point. Do you have any thoughts about how to extensibly support these kinds of cloud-service-specific options?

[maddyblue]

We can add more of these options as requested. I think they are pretty straightforward.

[dt]

Let's file them individually though -- I've like to be able to close this one once we implement the described auth params.

[rolandcrosby]

Sure, filed:

• EC2 instance creds: bulkio: enable EC2 instance-based temporary credentials for S3 access #31611
• S3 server-side encryption: bulkio: allow user to request S3 server-side encryption #31612