Support YAML in metadata - metadata.yaml

[simonw]

I was originally going to do this with a plugin - see #357 - but the more I work with metadata.json the more I want it to just accept YAML as an optional alternative to JSON.

The best example why is still this one: https://github.com/simonw/russian-ira-facebook-ads-datasette/blob/master/russian-ads-metadata.yaml

YAML is just SO much better than JSON for multi-line strings - in particular HTML and SQL, both of which are common in metadata.json files.

[simonw]

I can add https://pypi.org/project/PyYAML/ as a dependency for this.

One open question: how best to tell the difference between a JSON and a YAML file? I'd rather not do it based on a file extension.

Since I'm going to read the whole file into memory anyway (rather than try to stream it) and I only load it once at startup, maybe I try to parse with one and, if there's a parsing error, try the other one before giving up?

[simonw]

I'm going to depend on ~=5.3 https://github.com/yaml/pyyaml/blob/master/CHANGES - though
I could probably depend on a much wider set of versions (or maybe even no pinned minimum version at all).

[simonw]

I'm going to try to parse as JSON first, then fall back to YAML, then error.

[simonw]

Don't forget to fix the code in publish that loads and modifies metadata, e.g.:

datasette/datasette/utils/__init__.py

Lines 361 to 367 in 2aaad72

	if metadata:
	metadata_content = json.load(metadata)
	else:
	metadata_content = {}
	for key, value in extra_metadata.items():
	if value:
	metadata_content[key] = value

And:

datasette/datasette/publish/heroku.py

Lines 156 to 168 in 2aaad72

	if metadata:
	metadata_content = json.load(metadata)
	else:
	metadata_content = {}
	for key, value in extra_metadata.items():
	if value:
	metadata_content[key] = value
	try:
	os.chdir(tmp.name)
	if metadata_content:
	open("metadata.json", "w").write(json.dumps(metadata_content, indent=2))

[simonw]

datasette/datasette/utils/__init__.py

Lines 795 to 803 in 6717c71

	def parse_metadata(content):
	# content can be JSON or YAML
	try:
	return json.loads(content)
	except json.JSONDecodeError:
	try:
	return yaml.safe_load(content)
	except yaml.YAMLError:
	raise BadMetadataError("Metadata is not valid JSON or YAML")

[simonw]

Documentation: https://datasette.readthedocs.io/en/latest/metadata.html#using-yaml-for-metadata