New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Datasette Plugins #14

Open
simonw opened this Issue Oct 23, 2017 · 21 comments

Comments

Projects
None yet
2 participants
@simonw
Owner

simonw commented Oct 23, 2017

It would be neat if additional functionality could be opted-in to the system in the form of easy-to-add plugins, hosted as separate packages. First example: a Google Analytics plugin, which adds GA tracking code with your tracking ID to the web interface for your dataset.

This may be an opportunity to experiment with entry points: http://amir.rachum.com/blog/2017/07/28/python-entry-points/

@simonw simonw added this to the V2: visualization edition milestone Oct 23, 2017

@simonw

This comment has been minimized.

Owner

simonw commented Nov 11, 2017

The plugin system can also allow alternative providers for the publish command - e.g. maybe hook up hyper.sh as an option for publishing containers.

@simonw simonw changed the title from Some kind of plugin system to Plugin system Nov 14, 2017

@simonw

This comment has been minimized.

Owner

simonw commented Nov 14, 2017

Plugins should be able to interact with the build step. This would give plugins an opportunity to modify the SQL databases and help prepare them for serving - for example, a full-text search plugin might create additional FTS tables, or a mapping plugin might pre-calculate a bunch of geohashes for tables that have latitude/longitude values. Plugins could really take advantage of the immutable nature of the dataset here.

@simonw

This comment has been minimized.

Owner

simonw commented Nov 16, 2017

For visualizations, Google Maps should be made available as a plugin. The default visualizations can use Leaflet and Open Street Map, but there's no reason to not make Google Maps available as a plugin, especially if the plugin can provide a mechanism for configuring the necessary API key.

I'm particularly excited in the Google Maps heatmap visualization https://developers.google.com/maps/documentation/javascript/heatmaplayer as seen on http://mochimachine.org/wasteland/

@simonw

This comment has been minimized.

@jacobian

This comment has been minimized.

Contributor

jacobian commented Nov 22, 2017

I'd also suggest taking a look at stevedore, which has a ton of tools for doing plugin stuff. I've had good luck with it in the past.

@simonw

This comment has been minimized.

Owner

simonw commented Nov 22, 2017

Oh thanks, that definitely looks like an interesting option.

@simonw simonw changed the title from Plugin system to Datasette Plugins Apr 15, 2018

@simonw

This comment has been minimized.

Owner

simonw commented Apr 15, 2018

I started a thread on Twitter asking people for good examples of Python projects with a strong plugin ecosystem: https://twitter.com/simonw/status/985377670388105216

The most impressive example that came back was pytest - which now has nearly 400 plugins: https://plugincompat.herokuapp.com/

The pytest plugin infrastructure is available as an independent package called pluggy - which appears to offer everything I need for Datasette. I'm going to give that a go and see how well it works: https://pluggy.readthedocs.io/en/latest/

@simonw

This comment has been minimized.

Owner

simonw commented Apr 15, 2018

Datasette 1.0 will be the release of Datasette that attempts to provide a stable plugin API: https://github.com/simonw/datasette/milestone/7

There's a lot of work to be done before then, but as a starting point I'm going to support two very simple extension mechanisms:

  • Template system plugins - where the hook gets passed the Jinja environment and can freely register new template tags and filters
  • SQLite connection plugins - where the hook gets passed a new SQLite connection and can register custom SQLite functions

The template system hook will go near here:

datasette/datasette/app.py

Lines 1225 to 1228 in efbb4e8

self.jinja_env.filters['escape_css_string'] = escape_css_string
self.jinja_env.filters['quote_plus'] = lambda u: urllib.parse.quote_plus(u)
self.jinja_env.filters['escape_sqlite'] = escape_sqlite
self.jinja_env.filters['to_css_class'] = to_css_class

The SQLite connection hook will go near here:

datasette/datasette/app.py

Lines 1094 to 1098 in efbb4e8

def prepare_connection(self, conn):
conn.row_factory = sqlite3.Row
conn.text_factory = lambda x: str(x, 'utf-8', 'replace')
for name, num_args, func in self.sqlite_functions:
conn.create_function(name, num_args, func)

These two feel simple enough that I'm not worried that I might design an API that I later regret.

@simonw

This comment has been minimized.

Owner

simonw commented Apr 15, 2018

Tox is a good example of a project that uses pluggy in the way I want to use it (function hooks rather than classes): https://github.com/tox-dev/tox/blob/master/tox/hookspecs.py

simonw added a commit that referenced this issue Apr 15, 2018

First working prototype of plugins, refs #14
Uses pluggy: https://pluggy.readthedocs.io/

Two example plugins - an uppercase template filter and a convert_units() SQL function.
@simonw

This comment has been minimized.

Owner

simonw commented Apr 15, 2018

OK, from that prototype in f2720b0 it looks like pluggy provides a solid path forward.

Next steps:

  • Build a demo plugin that uses setuptools entrypoints to register with the datasette plugin manager via pluggy
  • Figure out a mechanism for registering plugins without first needing to publish them to PyPI. Can I load plugins from a special plugins/ directory similar to the --template-dir=templates/ option already supported by Datasette? #211
@simonw

This comment has been minimized.

Owner

simonw commented Apr 15, 2018

Here's a demo of the convert_units() SQL function I prototyped in f2720b0

2018-04-15 at 4 23 pm

@simonw

This comment has been minimized.

Owner

simonw commented Apr 15, 2018

Once I've got the plugins mechanism stable and people start releasing plugins it would be useful to have a dedicated Trove classifier on PyPI for Datasette plugins - Framework :: Datasette for example.

This would help me build a Datasette equivalent of the http://plugincompat.herokuapp.com/ site, which works by scanning PyPI for items with the Framework :: Pytest classifier:

https://github.com/pytest-dev/plugincompat/blob/8bdf1a6fb82807091ece0c68c196103ee8270194/update_index.py#L52-L53

It looks like the mechanism for requesting new PyPI classifiers is to file a ticket against warehouse, like these ones: pypa/warehouse#3570 and pypa/warehouse#2881

@simonw

This comment has been minimized.

Owner

simonw commented Apr 16, 2018

I created https://github.com/simonw/datasette-plugin-demos which is now published to PyPI and can be installed with pip install datasette-plugin-demos - I've confirmed that if you DO install it my Datasette plugins branch picks up the plugins, and select random_integer(1, 4) works as it should.

@simonw

This comment has been minimized.

Owner

simonw commented Apr 16, 2018

Slight code design problem... when I tried installing my branch in a fresh virtual environment I got this error, because setup.py now depends on pluggy (from importing __version__):

      File "/private/var/folders/jj/fngnv0810tn2lt_kd3911pdc0000gp/T/pip-req-build-dftqdezt/setup.py", line 2, in <module>
        from datasette import __version__
      File "/private/var/folders/jj/fngnv0810tn2lt_kd3911pdc0000gp/T/pip-req-build-dftqdezt/datasette/__init__.py", line 2, in <module>
        from .hookspecs import hookimpl # noqa
      File "/private/var/folders/jj/fngnv0810tn2lt_kd3911pdc0000gp/T/pip-req-build-dftqdezt/datasette/hookspecs.py", line 1, in <module>
        from pluggy import HookimplMarker
    ModuleNotFoundError: No module named 'pluggy'

Looks like I've run into point 6 on https://packaging.python.org/guides/single-sourcing-package-version/ :

2018-04-15 at 5 34 pm

simonw added a commit that referenced this issue Apr 16, 2018

Import version datasette.version to avoid dependency error
Running `from datasette import __version__` in `setup.py` was throwing
an error `ModuleNotFoundError: No module named 'pluggy'`

See https://packaging.python.org/guides/single-sourcing-package-version/

Refs #14

simonw added a commit that referenced this issue Apr 16, 2018

Start of the plugin system, based on pluggy (#210)
Uses https://pluggy.readthedocs.io/ originally created for the py.test project

We're starting with two plugin hooks:

prepare_connection(conn)

This is called when a new SQLite connection is created. It can be used to register custom SQL functions.

prepare_jinja2_environment(env)

This is called with the Jinja2 environment. It can be used to register custom template tags and filters.

An example plugin which uses these two hooks can be found at https://github.com/simonw/datasette-plugin-demos or installed using `pip install datasette-plugin-demos`

Refs #14
@simonw

This comment has been minimized.

Owner

simonw commented Apr 16, 2018

I should check if it's possible to have two template registration function plugins in a single plugin module. If it isn't maybe I should use class plugins instead of module plugins.

@simonw

This comment has been minimized.

Owner

simonw commented Apr 16, 2018

Annoyingly, the following only results in the last of the two prepare_connection hooks being registered:

from datasette import hookimpl
import pint
import random

ureg = pint.UnitRegistry()


@hookimpl
def prepare_connection(conn):
    def convert_units(amount, from_, to_):
        "select convert_units(100, 'm', 'ft');"
        return (amount * ureg(from_)).to(to_).to_tuple()[0]
    conn.create_function('convert_units', 3, convert_units)


@hookimpl
def prepare_connection(conn):
    conn.create_function('random_integer', 2, random.randint)
@simonw

This comment has been minimized.

Owner

simonw commented Apr 16, 2018

I think that's OK. The two plugins I've implemented so far (prepare_connection and prepare_jinja2_environment) both make sense if they can only be defined once-per-plugin. For the moment I'll assume I can define future hooks to work well with the same limitation.

The syntactic sugar idea in #220 can help here too.

@simonw simonw changed the title from Datasette Plugins to First working version of Datasette Plugins Apr 17, 2018

@simonw

This comment has been minimized.

Owner

simonw commented Apr 17, 2018

I just shipped Datasette 0.19 with where I'm at so far: https://github.com/simonw/datasette/releases/tag/0.19

@simonw

This comment has been minimized.

Owner

simonw commented Apr 18, 2018

I added a mechanism for plugins to serve static files and define custom CSS and JS URLs in #214 - see new documentation on http://datasette.readthedocs.io/en/latest/plugins.html#static-assets and http://datasette.readthedocs.io/en/latest/plugins.html#extra-css-urls

simonw added a commit that referenced this issue Apr 19, 2018

@simonw simonw changed the title from First working version of Datasette Plugins to Datasette Plugins Apr 20, 2018

@simonw

This comment has been minimized.

Owner

simonw commented Apr 20, 2018

I released everything we have so far in Datasette 0.20 and built and released an example plugin, datasette-cluster-map. Here's my blog entry about it: https://simonwillison.net/2018/Apr/20/datasette-plugins/

@simonw

This comment has been minimized.

Owner

simonw commented Apr 20, 2018

simonw added a commit that referenced this issue Jul 26, 2018

Extract publish heroku/now implementations into default plugins
This change introduces a new plugin hook, publish_subcommand, which can be
used to implement new subcommands for the "datasette publish" command family.

I've used this new hook to refactor out the "publish now" and "publish heroku"
implementations into separate modules. I've also added unit tests for these
two publishers, mocking the subprocess.call and subprocess.check_output
functions.

As part of this, I introduced a mechanism for loading default plugins. These
are defined in the new "default_plugins" list inside datasette/app.py

Closes #217 (Plugin support for datasette publish)
Closes #348 (Unit tests for "datasette publish")
Refs #14, #59, #102, #103, #146, #236, #347

simonw added a commit that referenced this issue Jul 26, 2018

publish_subcommand hook + default plugins mechanism, used for publish…
… heroku/now (#349)

This change introduces a new plugin hook, publish_subcommand, which can be
used to implement new subcommands for the "datasette publish" command family.

I've used this new hook to refactor out the "publish now" and "publish heroku"
implementations into separate modules. I've also added unit tests for these
two publishers, mocking the subprocess.call and subprocess.check_output
functions.

As part of this, I introduced a mechanism for loading default plugins. These
are defined in the new "default_plugins" list inside datasette/app.py

Closes #217 (Plugin support for datasette publish)
Closes #348 (Unit tests for "datasette publish")
Refs #14, #59, #102, #103, #146, #236, #347
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment