[datasets][annotationdb] specify cloud storage platform (gcp or aws)

[pwc2]

This allows the user to specify the cloud platform ('gcp' or 'aws') they are using when accessing datasets via the datasets API and annotation DB. A user running hail on AWS would read from the s3 bucket, and a user running on GCP would read from the gs bucket (can also still read locally from gs bucket with cloud storage connector installed). Not intended for cross-platform use like running a dataproc cluster and trying to access the s3 bucket, or trying to access the gs bucket on an EMR cluster. Will assume user on AWS has their configuration set with their credentials.

Did not have permissions to set up a cluster or EC2 instance on AWS to test, but was able to access all the datasets without issue on a dataproc cluster when using s3a:// prefixes and providing my AWS credentials in the spark config. So these changes should (hopefully) work fine on an EMR cluster with the s3:// client. Everything worked as expected on GCP.

Overview of changes:

• Added missing type hints to load_dataset() function and methods in db.py.
• In datasets.json:
  • Added AWS urls so now for each version of a dataset in dataset["versions"], the url entry looks like:

"url": {
  "aws": {
    "us": "s3://hail-datasets-us-east-1/..."
  },
  "gcp": {
    "eu": "gs://hail-datasets-eu/...",
    "us": "gs://hail-datasets-us/...
  }

• In load_dataset() function:
  • Added cloud parameter, set default values to region='us' and cloud='gcp.
• In DB class:
  • Added cloud parameter to constructor, set default values to region='us' and cloud='gcp.
  • All datasets in datasets.json currently end up in the _DB__by_name dictionary, even if not annotation datasets. Added line 279 in db.py to fix this and filter out datasets that are not annotation datasets (datasets missing "annotation_db" key).
• In Dataset class:
  • Added cloud and custom_config parameters to Dataset.from_name_and_json() to pass to DatasetVersion.from_json() to grab correct urls for platform.
  • Refactored Dataset.from_name_and_json(). Will require users passing a custom config to be of same format as what is in datasets.json for now. So each key: value pair in a user provided config should be as below:

"dataset_name": {
"annotation_db": {"key_properties": ["..."]},
"description": "...",
"url": "...",
"versions": [
  {
    "reference_genome": "...",
    "url": {"aws": {"eu": "...", "us": "..."},
            "gcp": {"eu": "...", "us": "..."}},
    "version": "..."
  }]}

• In DatasetVersion class:
  • Added reference_genome attribute, now that the version and reference genome are two separate fields in datasets.json.
  • Modified DatasetVersion.from_json() to handle cloud parameter to grab correct version url when using checked-in config file.

[danking]

This is awesome, I love the type annotations. I've just one nit on custom configs.

[danking]

I'm inclined to require custom configs to have the same format as the publicly supported one. AFAIK, nobody is using this functionality. Moreover, this is in the experimental module, so we have no obligation to maintain compatibility.

Do you have an argument in favor of different formats for the custom vs non-custom? I suppose a custom config is probably used by one lab which will exist in one cloud. I can see its a bit more annoying, but I prefer the consistency.

[pwc2]

Not really any argument for different formats, and the consistency would make things nicer. With that in mind, I think I'll also modify Dataset.from_name_and_json(), as it currently doesn't check for region with a custom config. That way the expected formatting for a custom config will be consistent with whats in datasets.json.

fixed custom configs