Resource format not guessed automatically #1350

kindly · 2013-12-04T03:21:07Z

[WIP] The resource format isn't guessed automatically, but CKAN claims to guess it automatically. I recommend removing the text for for 2.2 and fix guessing for 2.3.

nigelbabu · 2013-12-03T12:42:42Z

At the dev meeting, we decided to do this guessing in a custom validator.

kindly · 2013-12-05T10:25:24Z

This has turned into a major rewrite of how to handle resource formats.

It consists of a new json file which maps input formats to canonical we formatted formats. This is configurable by the instance by specifying an alternative path. See https://github.com/okfn/ckan/blob/9fadea50e08379164ab1b1070544f9795c80a669/ckan/config/resource_formats.json
At the moment only the second field in the list is actually used for display.
All logic concerning manipulating formats has been moved to a validator. Helpers have also been, added which the validators call, that do the work of loading the file.
Adding formats by default if the user leaves blank based on the mimetype, this is also a validator.

ISSUES
The format field not get converted to its clean form when saving and showing in the database. This could result in extra revisions when the data underneath has not changes. This also means we will be converting to uppercase more resources than we do currently. This is all fairly easily override nonetheless.

amercader · 2013-12-05T12:21:36Z

To be decided for 2.2 or 2.3 pending review

@davidread you are probably interested in this

davidread · 2013-12-05T13:23:21Z

ckan/logic/validators.py

+        url = data.get(key[:-1] + ('url',), '')
+        type, encoding = mimetypes.guess_type(url)
+        if type:
+            data[key] = type


"value = data[key]" and "data[key] = type" Can you be more specific with your variable names?

I guess you can't get away from data and key for a validator, but value and type?

value is used everywhere in the validators as the value of that key. So it may be bad but its a least consistent.

Also type, encoding is what is the documented return types.
http://docs.python.org/3.4/library/mimetypes.html#module-mimetypes

Good call on value.

In the context of the mimetype docs "type" might be a useful abbreviation of "mimetype". But here I think 'mimetype' would be clearer. 'type' is way too overloaded in coding everywhere... I'm just nitpicking sorry

will change it to mimetype! I agree its better.

wardi · 2013-12-05T20:42:37Z

My understanding from the call on Tuesday was that the format field is going from what was usually a file extension value (or almost any value a user might like to fill in) to a field that will contain a mime type some of the time instead.

Reading the code it looks like the the format being stored is the file extension, not the mime type. Did I misunderstand the intent during call or am I misreading the code?

kindly · 2013-12-05T21:03:46Z

This pull request is not the one that changes the format field to be a mimetype even though I wanted to prepare for the time if we decide to move that way... I would not try and squeeze that in version 2.2, as I do not think we have settled on this decision.

I have tried to make the new behaviour as backwards compatible as possible (except for the fact it fills in a format if left empty and removes the nasty "unified_format" stuff).

amercader · 2013-12-09T10:05:03Z

@kindly can we have a small separate pull request with just the template notice string change, so at least we have this fixed before the string freeze? Cheers

wardi · 2013-12-10T01:19:45Z

what's missing is a way to update all the resources based on the new validator. This depends on the schema so it doesn't feel right to make it a migration. What about a paster command that does package_update(package_show(id)) on all the packages? paster dataset touch-all or something

[#1350] Remove misleading resource format notice

davidread · 2013-12-17T12:34:20Z

David to come back to looking at this in a couple of weeks

amercader · 2014-03-11T12:17:53Z

ckan/config/resource_formats.json

+  ["MXD", "ESRI ArcGIS project file", "application/x-mxd", []],
+  ["TAR", "TAR Compressed File", "application/x-tar", []],
+  ["PNG", "PNG Image File", "image/png", []],
+  ["RSS", "RSS feed", "application/rss+xml", []]


Can we add this line?

["GeoJSON", "Geographic JavaScript Object Notation", "application/json", []]

davidread · 2014-03-25T11:25:51Z

ckan/config/resource_formats.json

+  ["SVG", "SVG vector image", "image/svg+xml", ["svg"]],
+  ["PPT", "Powerpoint Presentation", "application/vnd.ms-powerpoint", []],
+  ["JPEG", "JPG Image File", "image/jpeg", ["jpeg", "jpg"]],
+  ["Sparql Results", "Sparql Results", "application/sparql-results+xml", []],


I'm not sure it is useful to split SPARQL endpoints into two. This is a list of resource formats, and people recognize a SPARQL URL as an API. I don't think it is going to happen that someone publishes a text file with a SPARQL query or a SPARQL response in it, are they? Can we just get rid of the "Sparql query" line and rename "Sparql Results" to ["SPARQL", "SPARQL endpoint", "application/sparql-results+xml", []],?

davidread · 2014-03-25T11:33:30Z

ckan/config/resource_formats.json

+  ["NetCDF", "NetCDF File", "application/netcdf", []],
+  ["ArcGIS Map Service", "ArcGIS Map Service", "ArcGIS Map Service", ["arcgis map service"]],
+  ["TSV", "Tab Separated Values File", "text/tsv", []],
+  ["WFS", "Web Feature Service", "text/xml", []],


The mimetype for WFS should be "application/xml" i.e. the same as for "XML".

I can see that setting it to the same as for XML would spoil the if_empty_guess_format as it might see "file.xml" and may say it is a WFS file. Better to have no mimetype for WFS?

Actually for WFS (at least 1.1.0) the mime type defined by the OGC standard is text/xml. I think is probably fine to leave it like this for the time being because of what you mention regarding if_empty_guess

[#1350] Resource format not guessed automatically

[#1350] resource format change

9fadea5

davidread reviewed Dec 5, 2013
View reviewed changes

amercader added a commit that referenced this pull request Dec 12, 2013

[#1350] Remove misleading resource format notice

cf3a97e

amercader mentioned this pull request Dec 12, 2013

[#1350] Remove misleading resource format notice #1383

Merged

kindly added a commit that referenced this pull request Dec 12, 2013

Merge pull request #1383 from okfn/1350-resource-format-remove-text

19496ce

[#1350] Remove misleading resource format notice

davidread mentioned this pull request Dec 13, 2013

Use DCAT standard for format field #1336

Closed

kindly added 2 commits March 11, 2014 10:55

[#1350] change format of input file and fix some typos

36513ad

[#1350] merge conflicts

f295486

kindly assigned davidread Mar 11, 2014

amercader reviewed Mar 11, 2014
View reviewed changes

kindly added 2 commits March 18, 2014 09:08

[#1350] Fix typos and add geojson type

c241483

Merge branch 'master' into 1350-resource-format-change

3e16411

davidread reviewed Mar 25, 2014
View reviewed changes

[#1350] Added uniqueness constraint to mimetypes to avoid ambiguity.

e18a24d

davidread pushed a commit that referenced this pull request Mar 26, 2014

Merge pull request #1350 from ckan/1350-resource-format-change

7587c6e

[#1350] Resource format not guessed automatically

davidread merged commit 7587c6e into master Mar 26, 2014

davidread deleted the 1350-resource-format-change branch March 26, 2014 15:52

This was referenced Jun 27, 2014

Map Distribution formats to something that CKAN understands ckan/ckanext-dcat#18

Closed

Distribution's format type in DCAT XML ckan/ckanext-dcat#17

Closed

This was referenced Oct 21, 2014

Check resource format on resource create/update #1047

Closed

Resource 'Format' field is opaque #1371

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Resource format not guessed automatically #1350

Resource format not guessed automatically #1350

kindly commented Dec 4, 2013

nigelbabu commented Dec 3, 2013

kindly commented Dec 5, 2013

amercader commented Dec 5, 2013

davidread Dec 5, 2013

davidread Dec 5, 2013

kindly Dec 5, 2013

davidread Dec 5, 2013

kindly Dec 5, 2013

wardi commented Dec 5, 2013

kindly commented Dec 5, 2013

amercader commented Dec 9, 2013

wardi commented Dec 10, 2013

davidread commented Dec 17, 2013

amercader Mar 11, 2014

davidread Mar 25, 2014

davidread Mar 25, 2014

davidread Mar 25, 2014

amercader Apr 9, 2014

Resource format not guessed automatically #1350

Resource format not guessed automatically #1350

Conversation

kindly commented Dec 4, 2013

nigelbabu commented Dec 3, 2013

kindly commented Dec 5, 2013

amercader commented Dec 5, 2013

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wardi commented Dec 5, 2013

kindly commented Dec 5, 2013

amercader commented Dec 9, 2013

wardi commented Dec 10, 2013

davidread commented Dec 17, 2013

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment