Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unit test that checks that all plugin hooks have corresponding unit tests #771

simonw opened this issue May 27, 2020 · 5 comments


Copy link

simonw commented May 27, 2020

Turns out some hooks are missing unit test coverage: #581 (comment)_

@simonw simonw added the tests label May 27, 2020
Copy link
Owner Author

simonw commented May 27, 2020

This seems to work:

diff --git a/tests/ b/tests/
index 8b6a6b4..e9a40aa 100644
--- a/tests/
+++ b/tests/
@@ -7,7 +7,7 @@ from .fixtures import (
     TestClient as _TestClient,
 )  # noqa
 from import Datasette
-from datasette.plugins import get_plugins, DEFAULT_PLUGINS
+from datasette.plugins import get_plugins, DEFAULT_PLUGINS, pm
 from datasette.utils import sqlite3
 import base64
 import json
@@ -20,6 +20,21 @@ import pytest
 import urllib
+def test_plugin_hooks_have_tests():
+    "Every plugin hook should be referenced in this test module"
+    hooks = [name for name in dir(pm.hook) if not name.startswith("_")]
+    tests_in_this_module = [t for t in globals().keys() if t.startswith('test_')]
+    untested = []
+    for hook in hooks:
+        ok = False
+        for test in tests_in_this_module:
+            if hook in test:
+                ok = True
+        if not ok:
+            untested.append(hook)
+    assert not untested, 'These plugin hooks are missing tests: {}'.format(untested)
 def test_plugins_dir_plugin_prepare_connection(app_client):
     response = app_client.get(

Based on how the documentation unit tests work.

Currently fails with:

AssertionError: These plugin hooks are missing tests:
['prepare_jinja2_environment', 'publish_subcommand', 'register_facet_classes', 'register_output_renderer']

Copy link
Owner Author

simonw commented May 27, 2020

I'll do the work for this in the pull request #772.

Copy link
Owner Author

simonw commented May 27, 2020

Actually I'll land this using @pytest.mark.xfail.

Copy link
Owner Author

simonw commented May 27, 2020

$ pytest -k test_plugin_hooks_have_tests -vv
====================================== test session starts ======================================
platform darwin -- Python 3.7.7, pytest-5.2.4, py-1.8.1, pluggy-0.13.1 -- /Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/bin/python
cachedir: .pytest_cache
rootdir: /Users/simon/Dropbox/Development/datasette, inifile: pytest.ini
plugins: asyncio-0.10.0
collected 486 items / 475 deselected / 11 selected                                              

tests/[asgi_wrapper] XPASS                   [  9%]
tests/[extra_body_script] XPASS              [ 18%]
tests/[extra_css_urls] XPASS                 [ 27%]
tests/[extra_js_urls] XPASS                  [ 36%]
tests/[extra_template_vars] XPASS            [ 45%]
tests/[prepare_connection] XPASS             [ 54%]
tests/[prepare_jinja2_environment] XFAIL     [ 63%]
tests/[publish_subcommand] XFAIL             [ 72%]
tests/[register_facet_classes] XFAIL         [ 81%]
tests/[register_output_renderer] XFAIL       [ 90%]
tests/[render_cell] XPASS                    [100%]

========================= 475 deselected, 4 xfailed, 7 xpassed in 1.70s =========================

Copy link
Owner Author

simonw commented May 27, 2020

Closed in da87e96

@simonw simonw closed this as completed May 27, 2020
@simonw simonw added this to the Next planned release milestone May 27, 2020
simonw added a commit that referenced this issue May 28, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet

No branches or pull requests

1 participant