Clarify ownership and provenance for datasets listed in data.json

[philipashlock]

As @JoshData and @lilybradley have mentioned, the data.json from HHS includes data aggregated from State governments as well (here's an example from ny.gov). Does Project Open Data already have a clear requirement that datasets should only be those produced by the agency? If not, should that requirement be better specified?

I'm sure there are a lot of gray areas here where a local government has produced some data in partnership with a federal agency, but if we can provide better guidance on those scenarios (eg the data.json should only include data hosted on a federal .gov) then that would be helpful.

If other sources can be included, we'll want a better method to identify the source of these datasets. If each federal dataset listed the programCode and bureauCode as required it would be easy to filter out those that don't have them, but we'd still want a consistent and detailed way to identify those other sources.

[haleyvandyck]

Interesting point--thanks for raising it. Do we think this is something we could issue guidance on for the Publisher field? or are you suggesting we consider a new field in the schema?

[lilybradley]

@philipashlock This is the other example: http://hub.healthdata.gov/dataset/supplemental-nutrition-assistance-program-snap-data-system
@feomike Do you have thoughts from FGDC/GIS world, on a standard for referencing state and local governments (legal entities) in the current schema? Potentially as Publishers as @haleyvandyck mentions above?

[feomike]

right great question. the short answer is no, a standard nomenclature/vocabulary for state/local government does not exist. there isn't really an authority to maintain such a lexicon (e.g. dept of ed would have lists of school districts, HHS have local health agencies etc, but not an comprehensive thing). i can't even think of an association which would have the current understanding (perhaps american planning association?). all this being said, i can see this as an outreach opportunity if someone wants to dive in. strategy might be (a) just use Census geography as the nomenclature (eg Census Places as local cities, census counties as all counties etc). this method is not comprehensive (would mis some local tax areas and miss council of governments etc, but a good start), (b) assign the task to FGDC to form a committee and make such a thing (this would be great for a standards body to take on, and FGDC could be one to ask (there could be others, but FGDC is a federal entity and i assume OMB might be able to direct them to do something), (c) ask an association for some in kind help. in the geo world, NSGIC (national states GIS council) could be one, the american planning association could be one, might be some others. the short is, there isn't one today and building it is likely not a small task, depending on exactly how deep we want to go. sorry.

[gbinal]

@haleyvandyck - to your question:

Do we think this is something we could issue guidance on for the Publisher field? or are you suggesting we consider a new field in the schema?

I'd suggest the former. I think the schema is robust enough at the moment, so the next stage would be to outline a proposal for guidance on this topic.

[rebeccawilliams]

I am adding notes from Thursday's Common-Core Metadata Schema Review (see #325) where the group agreed there was value in accommodating more information about hierarchy (as well as primary source guidance) in the Publisher field.

The discussion noted that some Publisher information is less likely to change (e.g. Bureaus) and that due to potential Publisher changes flexibility should be favored. The discussion also noted that hierarchical information might be best collected with additional designated fields.

The group outlined three possible resolutions:

• 1. Including guidance that called for additional hierarchical information in the Publisher field to be separated by commas (a string, but still human-readable)
• 2. Collecting additional Publisher information through additional fields (as an array)
• 3. A combination of both, with optional fields for additional information

[gbinal]

One option for clarifying guidance would be to provide very specific guidance about what to put in the string, including a clear vision for how to provide multiple levels of publisher, separated by commas.

I'll take a pass at this.

[philipashlock]

If we were to align the publisher field to be compatible with DCAT it's actually defined there in a way that allows this additional structure. In DCAT this property uses the foaf:Agent class which includes an Organization class with properties like unitOf and subOrganizationOf see http://www.w3.org/TR/vocab-org/

[gbinal]

Here's a suggestion for updated Usage Notes for the publisher field:

The plaintext name of the entity publishing this dataset. Where greater specificity is desired, include as many levels of publisher as is useful, in ascending order, separated by commas. For instance: team, office, bureau, agency, department.

The example can be updated to be:

{"publisher":"U.S. Department of Education"} or {"publisher":"Office of Duty Operations, International Trade Administration, U.S. Department of Commerce"}

[gbinal]

Ah - sorry for missing your comment, Phil. I'd assumed that it had to stay a string. What would my above example (Office of Duty Operations) look like in that case?

[jpmckinney]

@gbinal Having a longer string for publisher is allowed by DCAT, since it uses the old Dublin Core publisher term which has no specific range. However, as @philipashlock points out, it's probably more maintainable to have the value of publisher be a foaf:Agent. That way, we don't end up in a situation with dozens of variations on the same publisher, like:

• Office of Duty Operations
• Office of Duty Operations, International Trade Administration
• Office of Duty Operations, U.S. Department of Commerce
• Office of Duty Operations, International Trade Administration, U.S. Department of Commerce

In fully expanded form, it can look like:

{
  "publisher": {
    "name": "Office of Duty Operations",
    "subOrganizationOf": {
      "name": "International Trade Administration",
      "subOrganizationOf": {
        "name": "U.S. Department of Commerce"
      }
    }
  }
}

However, you would probably never do that, as it has the same issues of the string version, in that there can be inconsistency in how the publisher value is serialized. Ideally, you would abbreviate to a URL:

{
  "publisher": "http://example.org/publishers/1.json"
}

Or include a name but use a URL for subOrganizationOf:

{
  "publisher": {
    "name": "Office of Duty Operations",
    "subOrganizationOf": "http://example.org/publishers/2.json"
  }
}

[jpmckinney]

@feomike The UK did your option (b) and produced the RDF that powers this visualization: http://data.gov.uk/organogram/cabinet-office

For option (a), I'd look at OpenCivicData's Division Identifiers, which establish stable identifiers for geographies. Examples:

• ocd-division/country:us
• ocd-division/country:us/state:nc
• ocd-division/country:us/state:nc/county:wake
• ocd-division/country:us/state:nc/place:cary