Plugin hook for dynamic metadata

[simonw]

@brandonrobertz contributed an implementation of this in PR #1368, which I just merged. Opening this ticket to track further work on this before it goes out in a Datasette release (likely preceded by an alpha).

[simonw]

Documentation for the new hook is now live at https://docs.datasette.io/en/latest/plugin_hooks.html#get-metadata-datasette-key-database-table-fallback

Link to the current snapshot of that documentation: https://github.com/simonw/datasette/blob/05a312caf3debb51aa1069939923a49e21cd2bd1/docs/plugin_hooks.rst#get-metadata-datasette-key-database-table-fallback

[simonw]

The documentation doesn't describe the fallback argument at the moment.

[simonw]

The documentation says:

datasette: You can use this to access plugin configuration options via datasette.plugin_config(your_plugin_name), or to execute SQL queries.

That's not accurate: since the plugin hook is a regular function, not an awaitable, you can't use it to run await db.execute(...) so you can't execute SQL queries.

I can fix this with the await-me-maybe pattern, used for other plugin hooks: https://simonwillison.net/2020/Sep/2/await-me-maybe/

BUT... that requires changing the ds.metadata() function to be awaitable, which will affect every existing plugn that uses that documented internal method!

[simonw]

Hmmm... that's tricky, since one of the most obvious ways to use this hook is to load metadata from database tables using SQL queries.

@brandonrobertz do you have a working example of using this hook to populate metadata from database tables I can try?

[simonw]

Here's where the plugin hook is called, demonstrating the fallback= argument:

datasette/datasette/app.py

Lines 426 to 472 in 05a312c

	def metadata(self, key=None, database=None, table=None, fallback=True):
	"""
	Looks up metadata, cascading backwards from specified level.
	Returns None if metadata value is not found.
	"""
	assert not (
	database is None and table is not None
	), "Cannot call metadata() with table= specified but not database="
	metadata = {}
	for hook_dbs in pm.hook.get_metadata(
	datasette=self, key=key, database=database, table=table, fallback=fallback
	):
	metadata = self._metadata_recursive_update(metadata, hook_dbs)
	# security precaution!! don't allow anything in the local config
	# to be overwritten. this is a temporary measure, not sure if this
	# is a good idea long term or maybe if it should just be a concern
	# of the plugin's implemtnation
	metadata = self._metadata_recursive_update(metadata, self._metadata_local)
	databases = metadata.get("databases") or {}
	search_list = []
	if database is not None:
	search_list.append(databases.get(database) or {})
	if table is not None:
	table_metadata = ((databases.get(database) or {}).get("tables") or {}).get(
	table
	) or {}
	search_list.insert(0, table_metadata)
	search_list.append(metadata)
	if not fallback:
	# No fallback allowed, so just use the first one in the list
	search_list = search_list[:1]
	if key is not None:
	for item in search_list:
	if key in item:
	return item[key]
	return None
	else:
	# Return the merged list
	m = {}
	for item in search_list:
	m.update(item)
	return m

I'm not convinced of the use-case for passing fallback= to the hook here - is there a reason a plugin might care whether fallback is True or False, seeing as the metadata() method already respects that fallback logic on line 459?

[simonw]

The await thing is worrying me a lot - it feels like this plugin hook is massively less useful if it can't make it's own DB queries and generally do asynchronous stuff - but I'd also like not to break every existing plugin that calls datasette.metadata(...).

One solution that could work: introduce a new method, maybe await datasette.get_metadata(...), which uses this plugin hook - and keep the existing datasette.metadata() method (which doesn't call the hook) around. This would ensure existing plugins keep on working.

Then, upgrade those plugins separately - with the goal of deprecating and removing .metadata() entirely in Datasette 1.0 - having upgraded the plugins in the meantime.

[simonw]

Just realized I already have an issue open for this, at #860. I'm going to close that and continue work on this in this issue.

[simonw]

The other alternative is to finish the work to build a _metadata internal table, see #1168. The idea there was that if we want to support efficient pagination and search across the metadata for thousands of attached tables powering it with a plugin hook doesn't work well - we don't want to call the hook once for every one of 1,000+ tables just to implement the homepage.

So instead, all metadata for all attached databases would be loaded into an in-memory database called _metadata. Plugins that want to modify stored metadata could then do so by directly writing to that table.

[simonw]

Hmmm... that's tricky, since one of the most obvious ways to use this hook is to load metadata from database tables using SQL queries.

@brandonrobertz do you have a working example of using this hook to populate metadata from database tables I can try?

Answering my own question: here's how Brandon implements it in his datasette-live-config plugin: https://github.com/next-LI/datasette-live-config/blob/72e335e887f1c69c54c6c2441e07148955b0fc9f/datasette_live_config/__init__.py#L50-L160

That's using a completely separate SQLite connection (actually wrapped in sqlite-utils) and making blocking synchronous calls to it.

This is a pragmatic solution, which works - and likely performs just fine, because SQL queries like this against a small database are so fast that not running them asynchronously isn't actually a problem.

But... it's weird. Everywhere else in Datasette land uses await db.execute(...) - but here's an example where users are encouraged to use blocking calls instead.