read_hubmeta

[elray1]

It might be helpful to have a convenience function to load hub metadata files. Some requirements/desired behaviors:

• support both json and yaml
• support this standard for references in json documents: https://datatracker.ietf.org/doc/html/draft-pbryan-zyp-json-ref-03 Briefly, the idea is that a reference has the form { "$ref": "path/to/json/object" }, and when that reference is resolved, it is replaced by the object that was referenced. For example, suppose we have the following input document:

{
  "round-1": {
    "model_tasks": [
      {
        "task_ids": {
          "origin_date": {
            "required": ["2022-09-03"],
            "optional": null
          },
          "location": {
            "required": {
              "$ref": "#/$defs/task_ids/location/us_states"
            },
            "optional": ["US"]
          },
        },
        "output_types": {
          "mean": {
            "type_id": {
              "required": null,
              "optional": ["NA"]
            },
            "value": {
              "type": "integer",
              "minimum": 0
            }
          }
        }
      },
    ],
    "submissions_due": {
        "start": "2022-09-01",
        "end": "2022-09-05"
    }
  },
  "$defs": {
    "task_ids": {
      "location": {
        "us_states": ["01", "02", "04", "05", "06", "08", "09", "10", "11", "12", "13", "15", "16", "17", "18", "19", "20", "21", "22", "23", "24", "25", "26", "27", "28", "29", "30", "31", "32", "33", "34", "35", "36", "37", "38", "39", "40", "41", "42", "44", "45", "46", "47", "48", "49", "50", "51", "53", "54", "55", "56"]
      }
    },
  ]
}

After resolving the reference, we would obtain the following document:

{
  "round-1": {
    "model_tasks": [
      {
        "task_ids": {
          "origin_date": {
            "required": ["2022-09-03"],
            "optional": null
          },
          "location": {
            "required": ["01", "02", "04", "05", "06", "08", "09", "10", "11", "12", "13", "15", "16", "17", "18", "19", "20", "21", "22", "23", "24", "25", "26", "27", "28", "29", "30", "31", "32", "33", "34", "35", "36", "37", "38", "39", "40", "41", "42", "44", "45", "46", "47", "48", "49", "50", "51", "53", "54", "55", "56"],
            "optional": ["US"]
          },
        },
        "output_types": {
          "mean": {
            "type_id": {
              "required": null,
              "optional": ["NA"]
            },
            "value": {
              "type": "integer",
              "minimum": 0
            }
          }
        }
      },
    ],
    "submissions_due": {
        "start": "2022-09-01",
        "end": "2022-09-05"
    }
  },
  "$defs": {
    "task_ids": {
      "location": {
        "us_states": ["01", "02", "04", "05", "06", "08", "09", "10", "11", "12", "13", "15", "16", "17", "18", "19", "20", "21", "22", "23", "24", "25", "26", "27", "28", "29", "30", "31", "32", "33", "34", "35", "36", "37", "38", "39", "40", "41", "42", "44", "45", "46", "47", "48", "49", "50", "51", "53", "54", "55", "56"]
      }
    },
  ]
}

Question: should the function also delete a "$defs" block, if there is one? Maybe as an option that defaults to TRUE?

[annakrystalli]

Noting this here for now as needing to be included in the data metadata documentation that NAs need to be encoded as .na in yml formats. Not sure where the best place in the Hub Data Formats document to record this currently would be. Happy for suggestions.

Also dropping this useful blogpost that compares JSON & yaml in R https://coolbutuseless.bitbucket.io/2018/02/03/yaml-vs-json-for-saving-human-readable-r-objects--tldr-use-yaml/

[elray1]

Due to technical hurdles that we don't want to get stuck on, we made working decisions to:

• Not support yaml for now
• Not support json references for now

I am filing separate issues for these two items.