CSV download of all terms

[cygri]

This is one part of #208, for CSV only. My goal here is to provide a replacement for the outdated CSV export on http://schema.rdfs.org/ as a step towards retiring that site. Does the following sound okay?

• Two files /docs/all-classes.csv and /docs/all-properties.csv
• Columns for the class file: id, label, comment, ancestors, supertypes, subtypes, properties
• Columns for the property file: id, label, comment, domains, ranges
• Simple space-separated lists for the multi-valued columns like subtypes
• Use DataCache like done for the full tree page
• Write the CSV using csv.writer (no external dependencies)

Should I work off of sdo-gozer or sdo-scripts?

[danbri]

Thanks for looking into this :) sdo-scripts is pretty volatile right now as I'm implementing our extensions model, which adds a named layer to each triple. The changes there should be pretty straightforward though, so that is probably the best place to start.

#208 has some rough code which might still work and help get you started. What you sketch sounds fine for now. Eventually we need to rationalize the machine views into a release/snapshot process. However having a simple "this is a live view of the data via CSV" would be useful right away to many parties I'm sure.

Besides classes and properties, there are also the enumerated values btw.

[cygri]

Ok, so I will work off sdo-scripts, and also look into providing a CSV version of the enumerated values.

The CSV writer code used on schema.rdfs.org is here:
https://github.com/mhausenblas/schema-org-rdf/blob/master/scrapers/schema2csv.py

So the plan is to retrieve the data with a variant of the code from #208, generate CSV with a variant of the schema2csv.py code, and integrate that into api.py, including caching, in a similar way as done for the JSON-LD tree.

[danbri]

@cygri - FYI sdo-scripts was merged into sdo-gozer which then shipped as schema.org v2.0. The current default branch is sdo-ganymede which will be our next release. #1 has links to plans for that...

[RichardWallis]

I am in the process of building on the vocabulary definition file downloads released in 3.1 and @cygri's previous useful work, to introduce CSV formats that will address this issue and part of (#208).

I have 3 prototype .csv files (for Types, Properties, Enumeration values) for those more versed in things CSV than me, to view for legality of format, terms included (or not), file naming, and general “quality”.

Gist: https://gist.github.com/RichardWallis/5ab417fd01b3fd3e9b7ef2e845bff13c - view in 'raw' to see the actual file formats.

The functionality to produce these was only run against the bib extension just to keep things small for now.

Comments, criticisms, etc. eagerly sort.

~Richard

[danbri]

@gkellogg any thoughts?

[danbri]

I wonder whether it is preferable to have list-valued cells, versus break those columns out into further tables (i.e. one for domainIncludes, with rows that pair one property ID to one type ID, etc.).

A commonsense / healthy sanity check could be making sure it can be parsed easily by e.g. Pandas in Python, and by D3 in JS.

[RichardWallis]

If possible I think it would be good to keep as few files as possible - with one for domains, one for range, one for properties, one for subtypes, etc. it would get very confusing.

Not a comprehensive test but Excel (except for utf8 chars) and the gist importer seems to be happy with them.

Also noting an issue with these prototype files - properties file includes properties defined outside of the bib extension but referenced from within it.

[danbri]

See http://webschemas.org/docs/developers.html for what we have currently. I just noticed the download form expands when you pick "CSV" to give Types, Properties and Enums as options.

[danbri]

@RichardWallis - from a quick look at the current types CSV (didn't check the others) it seems as if newline characters are leaking into the CSV:

grep "Related actions:" ~/Desktop/all-layers-types.csv

.... shows a lot of lines whose content is just <p>Related actions:</p>. One of these lines also has commas and quotes which don't seem to be escaped - "http://schema.org/SendAction","SendAction","<p>The act of physically/electronically dispatching an object for transfer from an origin to a destination.Related actions:</p>. Are we publishing wellformed CSV?

[RichardWallis]

We are publishing CSV using standard Python CSV libraries:
writer = csv.writer(out,quoting=csv.QUOTE_ALL,lineterminator='\n')

Apparently newline characters are valid in fields so long as the field is surrounded by double quotes, which ours are.

Test loaded all-layers-types.csv into Excel and it read OK.

[danbri]

Do we have an open issue along lines of "separate our descriptions into full blurb (including markup etc.) and a short purely-textual paragraph"? I know there is a Python function that approximates this (with '...' truncation) but perhaps in the managed schema files we also need to evolve and make such a distinction explicit?