Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

.execute_write() and .execute_write_fn() methods on Database #683

Merged
merged 10 commits into from Feb 25, 2020
Merged

Conversation

@simonw
Copy link
Owner

simonw commented Feb 24, 2020

See #682

  • Come up with design for .execute_write() and .execute_write_fn()
  • Build some quick demo plugins to exercise the design
  • Write some unit tests
  • Write the documentation
@simonw

This comment has been minimized.

Copy link
Owner Author

simonw commented Feb 24, 2020

Next steps are from comment #682 (comment)

I'm going to move ahead without needing that ability though. I figure SQLite writes are fast, and plugins can be trusted to implement just fast writes. So I'm going to support either fire-and-forget writes (they get added to the queue and a task ID is returned) or have the option to block awaiting the completion of the write (using Janus) but let callers decide which version they want. I may add optional timeouts some time in the future.

I am going to make both execute_write() and execute_write_fn() awaitable functions though, for consistency with .execute() and to give me flexibility to change how they work in the future.

I'll also add a block=True option to both of them which causes the function to wait for the write to be successfully executed - defaults to False (fire-and-forget mode).

@simonw

This comment has been minimized.

Copy link
Owner Author

simonw commented Feb 24, 2020

I've been testing this out by running one-off demo plugins. I saved the following in a file called write-plugins/log_asgi.py (it's a hacked around copy of asgi-log-to-sqlite) and then running datasette data.db --plugins-dir=write-plugins/:

from datasette import hookimpl
import sqlite_utils
import time


class AsgiLogToSqliteViaWriteQueue:
    lookup_columns = (
        "path",
        "user_agent",
        "referer",
        "accept_language",
        "content_type",
        "query_string",
    )

    def __init__(self, app, db):
        self.app = app
        self.db = db
        self._tables_ensured = False

    async def ensure_tables(self):
        def _ensure_tables(conn):
            db = sqlite_utils.Database(conn)
            for column in self.lookup_columns:
                table = "{}s".format(column)
                if not db[table].exists():
                    db[table].create({"id": int, "name": str}, pk="id")
            if "requests" not in db.table_names():
                db["requests"].create(
                    {
                        "start": float,
                        "method": str,
                        "path": int,
                        "query_string": int,
                        "user_agent": int,
                        "referer": int,
                        "accept_language": int,
                        "http_status": int,
                        "content_type": int,
                        "client_ip": str,
                        "duration": float,
                        "body_size": int,
                    },
                    foreign_keys=self.lookup_columns,
                )
        await self.db.execute_write_fn(_ensure_tables)

    async def __call__(self, scope, receive, send):
        if not self._tables_ensured:
            self._tables_ensured = True
            await self.ensure_tables()

        response_headers = []
        body_size = 0
        http_status = None

        async def wrapped_send(message):
            nonlocal body_size, response_headers, http_status
            if message["type"] == "http.response.start":
                response_headers = message["headers"]
                http_status = message["status"]

            if message["type"] == "http.response.body":
                body_size += len(message["body"])

            await send(message)

        start = time.time()
        await self.app(scope, receive, wrapped_send)
        end = time.time()

        path = str(scope["path"])
        query_string = None
        if scope.get("query_string"):
            query_string = "?{}".format(scope["query_string"].decode("utf8"))

        request_headers = dict(scope.get("headers") or [])

        referer = header(request_headers, "referer")
        user_agent = header(request_headers, "user-agent")
        accept_language = header(request_headers, "accept-language")

        content_type = header(dict(response_headers), "content-type")

        def _log_to_database(conn):
            db = sqlite_utils.Database(conn)
            db["requests"].insert(
                {
                    "start": start,
                    "method": scope["method"],
                    "path": lookup(db, "paths", path),
                    "query_string": lookup(db, "query_strings", query_string),
                    "user_agent": lookup(db, "user_agents", user_agent),
                    "referer": lookup(db, "referers", referer),
                    "accept_language": lookup(db, "accept_languages", accept_language),
                    "http_status": http_status,
                    "content_type": lookup(db, "content_types", content_type),
                    "client_ip": scope.get("client", (None, None))[0],
                    "duration": end - start,
                    "body_size": body_size,
                },
                alter=True,
                foreign_keys=self.lookup_columns,
            )

        await self.db.execute_write_fn(_log_to_database)


def header(d, name):
    return d.get(name.encode("utf8"), b"").decode("utf8") or None


def lookup(db, table, value):
    return db[table].lookup({"name": value}) if value else None


@hookimpl
def asgi_wrapper(datasette):
    def wrap_with_class(app):
        return AsgiLogToSqliteViaWriteQueue(
            app, next(iter(datasette.databases.values()))
        )

    return wrap_with_class
@simonw

This comment has been minimized.

Copy link
Owner Author

simonw commented Feb 24, 2020

I'm going to muck around with a couple more demo plugins - in particular one derived from datasette-upload-csvs - to make sure I'm comfortable with this API - then add a couple of tests and merge it with documentation that warns "this is still an experimental feature and may change".

@simonw

This comment has been minimized.

Copy link
Owner Author

simonw commented Feb 24, 2020

I'm not convinced by the return value of the .execute_write_fn() method:

self._write_queue.put(WriteTask(fn, task_id, reply_queue))
if block:
return WriteResponse(uuid, await reply_queue.async_q.get())
else:
return WriteResponse(uuid, in_progress=True)

Do I really need that WriteResponse class or can I do something nicer?

@simonw

This comment has been minimized.

Copy link
Owner Author

simonw commented Feb 24, 2020

I think if block it makes sense to return the return value of the function that was executed. Without it all I really need to do is return the uuid so something could theoretically poll for completion later on.

But is it weird having a function that returns different types depending on if you passed block=True or not? Should they be differently named functions?

I'm OK with the block=True pattern changing the return value I think.

@simonw

This comment has been minimized.

Copy link
Owner Author

simonw commented Feb 24, 2020

Also: are UUIDs really necessary here or could I use a simpler form of task identifier? Like an in-memory counter variable that starts at 0 and increments every time this instance of Datasette issues a new task ID?

The neat thing about UUIDs is that I don't have to worry if there are multiple Datasette instances accepting writes behind a load balancer. That seems pretty unlikely (especially considering SQLite databases encourage only one process to be writing at a time)... but I am experimenting with PostgreSQL support in #670 so it's probably worth ensuring these task IDs really are globally unique.

I'm going to stick with UUIDs. They're short-lived enough that their size doesn't really matter.

@simonw

This comment has been minimized.

Copy link
Owner Author

simonw commented Feb 24, 2020

Another demo plugin: delete_table.py

from datasette import hookimpl
from datasette.utils import escape_sqlite
from starlette.responses import HTMLResponse
from starlette.endpoints import HTTPEndpoint


class DeleteTableApp(HTTPEndpoint):
    def __init__(self, scope, receive, send, datasette):
        self.datasette = datasette
        super().__init__(scope, receive, send)

    async def post(self, request):
        formdata = await request.form()
        database = formdata["database"]
        db = self.datasette.databases[database]
        await db.execute_write("drop table {}".format(escape_sqlite(formdata["table"])))
        return HTMLResponse("Table has been deleted.")


@hookimpl
def asgi_wrapper(datasette):
    def wrap_with_asgi_auth(app):
        async def wrapped_app(scope, recieve, send):
            if scope["path"] == "/-/delete-table":
                await DeleteTableApp(scope, recieve, send, datasette)
            else:
                await app(scope, recieve, send)

        return wrapped_app

    return wrap_with_asgi_auth

Then I saved this as table.html in the write-templates/ directory:

{% extends "default:table.html" %}

{% block content %}
<form action="/-/delete-table" method="POST">
    <p>
        <input type="hidden" name="database" value="{{ database }}">
        <input type="hidden" name="table" value="{{ table }}">
        <input type="submit" value="Delete this table">
    </p>
</form>
{{ super() }}
{% endblock %}

(Needs CSRF protection added)

I ran Datasette like this:

$ datasette --plugins-dir=write-plugins/ data.db --template-dir=write-templates/

Result: I can delete tables!

data__everything__30_132_rows_-_Mozilla_Firefox

@simonw

This comment has been minimized.

Copy link
Owner Author

simonw commented Feb 24, 2020

Here's the upload_csv.py plugin file I've been playing with:

from datasette import hookimpl
from starlette.responses import PlainTextResponse, HTMLResponse
from starlette.endpoints import HTTPEndpoint
import csv as csv_std
import codecs
import sqlite_utils


class UploadApp(HTTPEndpoint):
    def __init__(self, scope, receive, send, datasette):
        self.datasette = datasette
        super().__init__(scope, receive, send)

    def get_database(self):
        # For the moment just use the first one that's not immutable
        mutable = [db for db in self.datasette.databases.values() if db.is_mutable]
        return mutable[0]

    async def get(self, request):
        return HTMLResponse(
            await self.datasette.render_template(
                "upload_csv.html", {"database_name": self.get_database().name}
            )
        )

    async def post(self, request):
        formdata = await request.form()
        csv = formdata["csv"]
        # csv.file is a SpooledTemporaryFile, I can read it directly
        filename = csv.filename
        # TODO: Support other encodings:
        reader = csv_std.reader(codecs.iterdecode(csv.file, "utf-8"))
        headers = next(reader)
        docs = (dict(zip(headers, row)) for row in reader)
        if filename.endswith(".csv"):
            filename = filename[:-4]
        # Import data into a table of that name using sqlite-utils
        db = self.get_database()

        def fn(conn):
            writable_conn = sqlite_utils.Database(db.path)
            writable_conn[filename].insert_all(docs, alter=True)
            return writable_conn[filename].count

        # Without block=True we may attempt 'select count(*) from ...'
        # before the table has been created by the write thread
        count = await db.execute_write_fn(fn, block=True)

        return HTMLResponse(
            await self.datasette.render_template(
                "upload_csv_done.html",
                {
                    "database": self.get_database().name,
                    "table": filename,
                    "num_docs": count,
                },
            )
        )


@hookimpl
def asgi_wrapper(datasette):
    def wrap_with_asgi_auth(app):
        async def wrapped_app(scope, recieve, send):
            if scope["path"] == "/-/upload-csv":
                await UploadApp(scope, recieve, send, datasette)
            else:
                await app(scope, recieve, send)

        return wrapped_app

    return wrap_with_asgi_auth

I also dropped copies of the two template files from https://github.com/simonw/datasette-upload-csvs/tree/699e6ca591f36264bfc8e590d877e6852f274beb/datasette_upload_csvs/templates into my write-templates/ directory.

@simonw

This comment has been minimized.

Copy link
Owner Author

simonw commented Feb 24, 2020

I'm going to punt on the ability to introspect the write queue and poll for completion using a UUID for the moment. Can add those later.

@simonw

This comment has been minimized.

Copy link
Owner Author

simonw commented Feb 25, 2020

Basic stuff to cover in unit tests:

  • Exercise .execute_write(sql) - both with block=True and block=False
  • Exercise .execute_write_fn(fn) in the same way
  • Throw 10 updates in the queue, block on just the last one, check it worked correctly

I'm going to write these tests directly against a Database() object rather than booting up an entire Datasette instance.

@simonw

This comment has been minimized.

Copy link
Owner Author

simonw commented Feb 25, 2020

The other problem with the poll-for-UUID-completion idea: how long does this mean Datasette needs to keep holding onto the WriteTask objects?

Maybe we say you only get to ask "is this UUID still in the queue" and if the answer is "no" then you assume the task has been completed.

@simonw

This comment has been minimized.

Copy link
Owner Author

simonw commented Feb 25, 2020

This failing test is a nasty one - the whole thing just hangs (so I imagine Travis will run for a while before hopefully giving up). Here's what happens if I add --full-trace and then hit Ctrl+C to cancel a test run:

$ pytest -k test_execute_write_fn_block_true --full-trace
=================================================================== test session starts ===================================================================
platform darwin -- Python 3.7.5, pytest-5.2.4, py-1.8.1, pluggy-0.13.1
rootdir: /Users/simonw/Dropbox/Development/datasette, inifile: pytest.ini
plugins: asyncio-0.10.0
collected 410 items / 409 deselected / 1 selected                                                                                                         

tests/test_database.py ^C^C

================================================================= 409 deselected in 4.45s =================================================================
Traceback (most recent call last):
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/main.py", line 193, in wrap_session
    session.exitstatus = doit(config, session) or 0
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/main.py", line 237, in _main
    config.hook.pytest_runtestloop(session=session)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/main.py", line 258, in pytest_runtestloop
    item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py", line 80, in pytest_runtest_protocol
    runtestprotocol(item, nextitem=nextitem)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py", line 95, in runtestprotocol
    reports.append(call_and_report(item, "call", log))
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py", line 176, in call_and_report
    call = call_runtest_hook(item, when, **kwds)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py", line 201, in call_runtest_hook
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py", line 229, in from_call
    result = func()
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py", line 201, in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/runner.py", line 125, in pytest_runtest_call
    item.runtest()
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/python.py", line 1429, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pytest_asyncio/plugin.py", line 158, in pytest_pyfunc_call
    pyfuncitem.obj(**testargs), loop=event_loop))
  File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 566, in run_until_complete
    self.run_forever()
  File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 534, in run_forever
    self._run_once()
  File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 1735, in _run_once
    event_list = self._selector.select(timeout)
  File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/selectors.py", line 558, in select
    kev_list = self._selector.control(None, max_ev, timeout)
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/bin/pytest", line 8, in <module>
    sys.exit(main())
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/config/__init__.py", line 90, in main
    return config.hook.pytest_cmdline_main(config=config)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/main.py", line 230, in pytest_cmdline_main
    return wrap_session(config, _main)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/main.py", line 209, in wrap_session
    config.hook.pytest_keyboard_interrupt(excinfo=excinfo)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/terminal.py", line 680, in pytest_keyboard_interrupt
    self._keyboardinterrupt_memo = excinfo.getrepr(funcargs=True)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_code/code.py", line 598, in getrepr
    return fmt.repr_excinfo(self)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_code/code.py", line 830, in repr_excinfo
    reprtraceback = self.repr_traceback(excinfo)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_code/code.py", line 778, in repr_traceback
    reprentry = self.repr_traceback_entry(entry, einfo)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_code/code.py", line 737, in repr_traceback_entry
    reprargs = self.repr_args(entry) if not short else None
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_code/code.py", line 656, in repr_args
    args.append((argname, saferepr(argvalue)))
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_io/saferepr.py", line 67, in saferepr
    return SafeRepr(maxsize).repr(obj)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/_io/saferepr.py", line 36, in repr
    s = super().repr(x)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/reprlib.py", line 52, in repr
    return self.repr1(x, self.maxlevel)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/reprlib.py", line 60, in repr1
    return getattr(self, 'repr_' + typename)(x, level)
  File "/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/reprlib.py", line 112, in repr_dict
    for key in islice(_possibly_sorted(x), self.maxdict):
KeyboardInterrupt
Task was destroyed but it is pending!
task: <Task pending coro=<test_execute_write_fn_block_true() running at /Users/simonw/Dropbox/Development/datasette/tests/test_database.py:139> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x105580510>()]>>
@simonw

This comment has been minimized.

Copy link
Owner Author

simonw commented Feb 25, 2020

I'm happy with this now. I'm going to merge to master.

@simonw simonw merged commit a093c5f into master Feb 25, 2020
2 checks passed
2 checks passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
@simonw simonw deleted the write-queue branch Feb 25, 2020
@simonw

This comment has been minimized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

1 participant
You can’t perform that action at this time.