Creating URLs based on the base URL of another CSV file

[JeniT]

The example with table group comprising three interrelated tables highlights a usability problem in the current specification.

The basis for the example is that two interrelated schemas can be published by a central authority and be used multiple times. Naturally this means that within the schemas themselves, there can't be any hard-coding of the URLs of the CSV files.

This works fine for the foreign keys because there is a mechanism there (the schemaReference property) to ensure that the key references whichever CSV file adheres to a known schema. But it falls down with the valueUrl or aboutUrl because those are always interpreted relative to the location of the CSV file the schema is being applied to; there is no mechanism to interpret those URLs relative to the base URL of a different CSV file.

You can see this in the example where the schema file junior-roles.json includes:

"valueUrl": "senior-roles.csv#post-{reportsToSenior}"

This hard-codes the name of the CSV file that adheres to the senior-roles.json schema in the paired publication. I don't think this is acceptable.

The only thing I can think of here, and it's really ugly, is to introduce some built-in variables to URL templates that look like _schema:encoded-schema-URL which would resolve to the base URL of the table that used that schema, so that you could do:

"valueUrl": "{_schema:http%3A%2F%2Fexample.org%2Fsenior-roles.json}#post-{reportsToSenior}"

and it would set the value URL property on the cell to something like "http://example.com/senior-roles.csv#post-1".

Alternatively, we can make the assumption that anyone creating data of this sort is going to want to create URLs that aren't based on the locations of CSV files and leave this as deliberately undefined. In that case we'll need to change the example in the RDF (and JSON) conversion documents accordingly.

[gkellogg]

The valueUrl can have the form of an absolute URL, in which it will not be considered relative to the CSV.

Note that a variable can also be specified as {+variable} can be used with reserved characters and will not bus subject to % substitution, but that's not really necessary for this example. So, you could specify "valurUrl" as follows:

"valueUrl": "http://example.org/senior-roles.json#post-{reportsToSenior}"

This would create the "http://example.com/senior-roles.csv#post-1" you're looking for, without regard to the actual location of the senior-roles.json file, as URI joining with an absolute URL yields that absolute URL as defined in Relative Resolution in RFC 3986.

Perhaps I'm missing your concern, though.

[6a6d74]

@gkellogg - I think @JeniT was concerned about hardcoding the name of the CSV file in the centrally published schema. In your snippet, you have referenced metadata file senior-roles.json but are trying to create a valueURL based on the CSV file senior-roles.cv.

Also, in this case, the centrally published schema does not have any indication of where the actual CSV files will be published - in fact they will be published by each government department and agency at different locations, so the shared schema can't use absolute URLs.

[6a6d74]

@JeniT - when putting this example together I ended up with the same concern as you've raised here but concluded that the way forward was to require use of specific names for the senior- and junior-roles CSV files to be provided in the same directory as the metadata file (which must be downloaded from a central location). I included the following note to that effect:

[...] the metadata description resource metadata.json would be also be centrally published. Government departments seeking to validate their role and salary data would download a copy of this metadata description and place it, without modification, in the same directory as their tabular data files whose names MUST match those specified in the metadata description; senior-roles.csv and junior-roles.csv.

A bit ugly, but it does work.

I am guessing that you want to be able to work without needing to download the metadata.json resource? To do this you need to reference the CSV file that uses a given tableSchema?

[iherman]

Alternatively, we can make the assumption that anyone creating data of this sort is going to want to create URLs that aren't based on the locations of CSV files and leave this as deliberately undefined. In that case we'll need to change the example in the RDF (and JSON) conversion documents accordingly.

@JeniT, I would prefer going with this for now. We may want to flag this as a question to the community in the drafts seeking input on whether this is acceptable or not. The other option is not only ugly, but there are also a number of (edge) cases that we should go through. E.g., are we sure, in all settings, that the processor knows which CSV files use a particular schema? Eg, in the case where only one document is converted?

[6a6d74]

[cont]

The name of the CSV files is also hardcoded in the resources property of the metadata.json file; e.g.

{
  "@type": "TableGroup",
  "@context": ["http://www.w3.org/ns/csvw", {"@language": "en"}],
  "resources": [
  ...
  {
    "url": "senior-roles.csv",
    "tableSchema": "senior-roles.json"
  }, {
    "url": "junior-roles.csv",
    "tableSchema": "junior-roles.json"
  }]
}

Are you assuming that the CSV publisher would create their own top-level metadata file that references the remotely published senior-roles.json and junior-roles.json resources?

[6a6d74]

@iherman - I agree.

In the real example (see Use Case 4) you can see that the data includes a column for the Organisation; e.g. Higher Education Funding Council for England.

If we imagine that the name of this column is organisation, then the valueUrl could be explicitly set without naming clashes ...

"valueUrl": "http://example.org/{organisation}#post-{reportsToSenior}"

which would produce the URL "http://example.org/Higher%20Education%20Funding%20Council%20for%20England#post-1".

OK; the PCT-encoded name of the organisation is a bit ugly but it illustrates how unique names could be created. In such a case, I would recommend including a code for the organisation name as a column (e.g. HEFCE) that would be used to make more compact URLs (e.g. "http://example.org/HEFCE#post-1").

Like with the profession column, this could be validated against a centrally published controlled vocabulary to make sure that publishers are using the correct abbreviation for their organisation.

[iherman]

Discussed at telco: http://www.w3.org/2015/04/08-csvw-irc#T14-55-47

@6a6d74 will update the examples