Datasette Plugins

[simonw]

It would be neat if additional functionality could be opted-in to the system in the form of easy-to-add plugins, hosted as separate packages. First example: a Google Analytics plugin, which adds GA tracking code with your tracking ID to the web interface for your dataset.

This may be an opportunity to experiment with entry points: http://amir.rachum.com/blog/2017/07/28/python-entry-points/

[simonw]

The plugin system can also allow alternative providers for the publish command - e.g. maybe hook up hyper.sh as an option for publishing containers.

[simonw]

Plugins should be able to interact with the build step. This would give plugins an opportunity to modify the SQL databases and help prepare them for serving - for example, a full-text search plugin might create additional FTS tables, or a mapping plugin might pre-calculate a bunch of geohashes for tables that have latitude/longitude values. Plugins could really take advantage of the immutable nature of the dataset here.

[simonw]

For visualizations, Google Maps should be made available as a plugin. The default visualizations can use Leaflet and Open Street Map, but there's no reason to not make Google Maps available as a plugin, especially if the plugin can provide a mechanism for configuring the necessary API key.

I'm particularly excited in the Google Maps heatmap visualization https://developers.google.com/maps/documentation/javascript/heatmaplayer as seen on http://mochimachine.org/wasteland/

[simonw]

http://setuptools.readthedocs.io/en/latest/setuptools.html#dynamic-discovery-of-services-and-plugins Is pretty good

[jacobian]

I'd also suggest taking a look at stevedore, which has a ton of tools for doing plugin stuff. I've had good luck with it in the past.

[simonw]

Oh thanks, that definitely looks like an interesting option.

[simonw]

I started a thread on Twitter asking people for good examples of Python projects with a strong plugin ecosystem: https://twitter.com/simonw/status/985377670388105216

The most impressive example that came back was pytest - which now has nearly 400 plugins: https://plugincompat.herokuapp.com/

The pytest plugin infrastructure is available as an independent package called pluggy - which appears to offer everything I need for Datasette. I'm going to give that a go and see how well it works: https://pluggy.readthedocs.io/en/latest/

[simonw]

Datasette 1.0 will be the release of Datasette that attempts to provide a stable plugin API: https://github.com/simonw/datasette/milestone/7

There's a lot of work to be done before then, but as a starting point I'm going to support two very simple extension mechanisms:

• Template system plugins - where the hook gets passed the Jinja environment and can freely register new template tags and filters
• SQLite connection plugins - where the hook gets passed a new SQLite connection and can register custom SQLite functions

The template system hook will go near here:

datasette/datasette/app.py

Lines 1225 to 1228 in efbb4e8

	self.jinja_env.filters['escape_css_string'] = escape_css_string
	self.jinja_env.filters['quote_plus'] = lambda u: urllib.parse.quote_plus(u)
	self.jinja_env.filters['escape_sqlite'] = escape_sqlite
	self.jinja_env.filters['to_css_class'] = to_css_class

The SQLite connection hook will go near here:

datasette/datasette/app.py

Lines 1094 to 1098 in efbb4e8

	def prepare_connection(self, conn):
	conn.row_factory = sqlite3.Row
	conn.text_factory = lambda x: str(x, 'utf-8', 'replace')
	for name, num_args, func in self.sqlite_functions:
	conn.create_function(name, num_args, func)

These two feel simple enough that I'm not worried that I might design an API that I later regret.

[simonw]

Tox is a good example of a project that uses pluggy in the way I want to use it (function hooks rather than classes): https://github.com/tox-dev/tox/blob/master/tox/hookspecs.py

[simonw]

OK, from that prototype in f2720b0 it looks like pluggy provides a solid path forward.

Next steps:

• Build a demo plugin that uses setuptools entrypoints to register with the datasette plugin manager via pluggy
• Figure out a mechanism for registering plugins without first needing to publish them to PyPI. Can I load plugins from a special plugins/ directory similar to the --template-dir=templates/ option already supported by Datasette? Load plugins from a --plugins-dir=plugins/ directory #211

[simonw]

Here's a demo of the convert_units() SQL function I prototyped in f2720b0

[simonw]

Once I've got the plugins mechanism stable and people start releasing plugins it would be useful to have a dedicated Trove classifier on PyPI for Datasette plugins - Framework :: Datasette for example.

This would help me build a Datasette equivalent of the http://plugincompat.herokuapp.com/ site, which works by scanning PyPI for items with the Framework :: Pytest classifier:

https://github.com/pytest-dev/plugincompat/blob/8bdf1a6fb82807091ece0c68c196103ee8270194/update_index.py#L52-L53

It looks like the mechanism for requesting new PyPI classifiers is to file a ticket against warehouse, like these ones: pypi/warehouse#3570 and pypi/warehouse#2881

[simonw]

I created https://github.com/simonw/datasette-plugin-demos which is now published to PyPI and can be installed with pip install datasette-plugin-demos - I've confirmed that if you DO install it my Datasette plugins branch picks up the plugins, and select random_integer(1, 4) works as it should.

[simonw]

Slight code design problem... when I tried installing my branch in a fresh virtual environment I got this error, because setup.py now depends on pluggy (from importing __version__):

      File "/private/var/folders/jj/fngnv0810tn2lt_kd3911pdc0000gp/T/pip-req-build-dftqdezt/setup.py", line 2, in <module>
        from datasette import __version__
      File "/private/var/folders/jj/fngnv0810tn2lt_kd3911pdc0000gp/T/pip-req-build-dftqdezt/datasette/__init__.py", line 2, in <module>
        from .hookspecs import hookimpl # noqa
      File "/private/var/folders/jj/fngnv0810tn2lt_kd3911pdc0000gp/T/pip-req-build-dftqdezt/datasette/hookspecs.py", line 1, in <module>
        from pluggy import HookimplMarker
    ModuleNotFoundError: No module named 'pluggy'

Looks like I've run into point 6 on https://packaging.python.org/guides/single-sourcing-package-version/ :

[simonw]

I should check if it's possible to have two template registration function plugins in a single plugin module. If it isn't maybe I should use class plugins instead of module plugins.

[simonw]

Annoyingly, the following only results in the last of the two prepare_connection hooks being registered:

from datasette import hookimpl
import pint
import random

ureg = pint.UnitRegistry()


@hookimpl
def prepare_connection(conn):
    def convert_units(amount, from_, to_):
        "select convert_units(100, 'm', 'ft');"
        return (amount * ureg(from_)).to(to_).to_tuple()[0]
    conn.create_function('convert_units', 3, convert_units)


@hookimpl
def prepare_connection(conn):
    conn.create_function('random_integer', 2, random.randint)

[simonw]

I think that's OK. The two plugins I've implemented so far (prepare_connection and prepare_jinja2_environment) both make sense if they can only be defined once-per-plugin. For the moment I'll assume I can define future hooks to work well with the same limitation.

The syntactic sugar idea in #220 can help here too.

[simonw]

I just shipped Datasette 0.19 with where I'm at so far: https://github.com/simonw/datasette/releases/tag/0.19

[simonw]

I added a mechanism for plugins to serve static files and define custom CSS and JS URLs in #214 - see new documentation on http://datasette.readthedocs.io/en/latest/plugins.html#static-assets and http://datasette.readthedocs.io/en/latest/plugins.html#extra-css-urls

[simonw]

I released everything we have so far in Datasette 0.20 and built and released an example plugin, datasette-cluster-map. Here's my blog entry about it: https://simonwillison.net/2018/Apr/20/datasette-plugins/

[simonw]

Here's a link demonstrating my new plugin: https://datasette-cluster-map-demo.now.sh/polar-bears-455fe3a/USGS_WC_eartags_output_files_2009-2011-Status

[simonw]

We've grown a bunch of plugin hooks over the past two years: https://datasette.readthedocs.io/en/latest/plugins.html#plugin-hooks

Since the plugin system will never be 100% "finished", I'm closing this in favor of the label: plugins