Resource creation fails with IDatasetForm extra fields with names sorting before and after 'extra_validation'

[wardi]

Steps to reproduce:

1. Enable an IDatasetForm plugin that has extra fields with names that sort both before and after the string 'extra_validation', such as the one in wardi@eeb4267
2. Create a dataset (adding resources will still work if your plugin accepts its extra fields in the package_metadata_fields.html as the one above does)
3. Click Edit package, then Add resource
4. Try to save the resource

What happens:

A traceback is issued with an IndexError in ckan.lib.navl.dictization_functions.flatten()

Workaround:

I wrote a patch wardi@fa99298 that avoids the problem. This fix is rather inelegant, better would be to execute the 'extra_validation' rule before or after all the other rules. I leave that decision to the experts.

[tobes]

@wardi could you replace the links so that they point to the specific file/patch you are referring to. Also if you could add the traceback in a comment on this issue that would be really helpful

thanks

[wardi]

@tobes links updated and:

Module ckan.controllers.package:645 in new_resource          view
>>  get_action('resource_create')(context, data)
Module ckan.logic:324 in wrapped          view
>>  return _action(context, data_dict, **kw)
Module ckan.logic.action.create:251 in resource_create          view
>>  pkg_dict = _get_action('package_update')(context, pkg_dict)
Module ckan.logic:324 in wrapped          view
>>  return _action(context, data_dict, **kw)
Module ckan.logic.action.update:251 in package_update          view
>>  data, errors = _validate(data_dict, schema, context)
Module ckan.lib.navl.dictization_functions:228 in validate          view
>>  converted_data, errors = _validate(flattened, schema, context)
Module ckan.lib.navl.dictization_functions:289 in _validate          view
>>  convert(converter, key, converted_data, errors, context)
Module ckan.lib.navl.dictization_functions:186 in convert          view
>>  converter(key, converted_data, errors, context)
Module ckan.logic.validators:306 in duplicate_extras_key          view
>>  unflattened = unflatten(data)
Module ckan.lib.navl.dictization_functions:393 in unflatten          view
>>  current_pos = current_pos[key]
IndexError: list index out of range

[wardi]

From the mailing list http://lists.okfn.org/pipermail/ckan-dev/2013-March/004242.html

Fabian says this applies to another issue as well

I was also able to fix my old issue: http://lists.okfn.org/pipermail/ckan-dev/2013-January/003852.html
by adding your fix to the lib.navl.dictization_functions.py validate method:

    data_no_extras = dict(converted_data)
    if ('extras',) in converted_data:
        del data_no_extras[('extras',)]
    converted_data = unflatten(data_no_extras) 

    #converted_data = unflatten(converted_data)

[tobes]

@seanh have you any thoughts on this?

[seanh]

Yes, I think I need a drink

[tobes]

:)

[seanh]

The schema is a dict so I don't know how we can make it run the "extra_validation" rule before or after all the others, unless we special-case that rule, If @wardi's fix fixes this problem and doesn't break any of the tests (either the core tests or my IDatasetForm tests) then I would be tempted to take it as-is, on the grounds that validate() is best treated as a black box.

I want to merge my pending IDatasetForm pull request before I look at this one

[amercader]

@seanh Assigning this to you just so you are aware of it and in case is related to the IDatasetForm stuff you are working on now. Feel free to ignore it if not.

[seanh]

I think we need to get my other idatasetform pull requests merged first, then we can see if this bug still occurs and if this fix still fixes it

[seanh]

Confirmed this is still happening on master

[seanh]

Confirmed @wardi's commit fixes the problem, and all the tests (including example_idatasetform tests) still pass with this commit. I'm tempted to just take this commit but I'll have a look after lunch and see if there's a more elegant way...