Admin: add schema field to resource

[AntoineAugusti]

• Link data.gouv.fr to the schema catalog of schema.data.gouv.fr
• Add a schema field to the resource model
• Let producers select a schema from a list on resources

I left some comments/questions at various places.

To discuss

• Configuration for the schema catalog URL
• The schema catalog of schema.data.gouv.fr only lists TableSchema schemas for now, should we change it? It's because we needed this endpoint for Validata, therefore it's made to include only TableSchema schemas. This can change.

[abulte]

nice work 🌈

[abulte]

I think we need a smarter caching strategy and exception handling.

Something like that https://github.com/etalab/udata-gouvfr/pull/486/files#diff-1c9abf3241dab91b0e60c554f8bcd67aR72.

This would lead to removing the cache on the API and handle it here.

[abulte]

The schema catalog of schema.data.gouv.fr only lists TableSchema schemas for now, should we change it? It's because we needed this endpoint for Validata, therefore it's made to include only TableSchema schemas. This can change.

No need to change that for now. I don't think we'll do anything very interesting with other schemas for quite some time.

[AntoineAugusti]

Thanks for your guidance and your help @abulte!

I:

• tackled your PR comments
• reverted the commit introducing indexing - it'll be done later
• added a double caching mechanism to fetch schemas from the schema catalog
• cached at the model level instead of the API level
• enforced an enumeration for the schema property at the form level

[AntoineAugusti]

Added custom exceptions as you suggested @abulte. Thanks for your review.

[abulte]

One last suggestion, add a CHANGELOG entry (under "Current in progress") and 🚀

[AntoineAugusti]

I had to do an extra commit this morning to fix dataset/forms.py. When starting my local server, I had a stracktrace regarding the absence of the app context in this file, because of choices=[schema['id'] for schema in ResourceSchema.objects()] which needs the app context under the hood. I guess the app context is not available yet, hence we can't access the current config at this point.

I switched to a string field with a validator. The SelectField relies on a list of options, through the choices keyword. You can't leave it empty, that's why I switched to the String class.

This pattern is used elsewhere in the very same file

udata/udata/core/dataset/forms.py

Lines 33 to 48 in 32b0903

	def enforce_filetype_file(form, field):
	'''Only allowed domains in resource.url when filetype is file'''
	if form._fields.get('filetype').data != RESOURCE_FILETYPE_FILE:
	return
	domain = urlparse(field.data).netloc
	allowed_domains = current_app.config['RESOURCES_FILE_ALLOWED_DOMAINS']
	allowed_domains += [current_app.config.get('SERVER_NAME')]
	if current_app.config.get('CDN_DOMAIN'):
	allowed_domains.append(current_app.config['CDN_DOMAIN'])
	if '*' in allowed_domains:
	return
	if domain and domain not in allowed_domains:
	message = _('Domain "{domain}" not allowed for filetype "{filetype}"')
	raise validators.ValidationError(message.format(
	domain=domain, filetype=RESOURCE_FILETYPE_FILE
	))

I made sure the validator was working by editing the HTML