Deploy a comprehensive set of plugins

[simonw]

Original idea was to automatically deploy all github.com/simonw plugins with the datasette-plugin tag - so I don't have to keep updating the deploy script here.

[simonw]

I'm not sure this is a good idea. There's a good chance one or more of the plugins won't make sense in this context - datasette-glitch for example.

I'm going to create a list in this repo instead.

This query will help: https://github-to-sqlite.dogsheep.net/github?sql=select%0D%0A++name%0D%0Afrom%0D%0A++repos%0D%0Awhere%0D%0A++%22owner%22+%3D+9599%0D%0A++and+%22private%22+%3D+0%0D%0A++and+rowid+in+%28%0D%0A++++select%0D%0A++++++repos.rowid%0D%0A++++from%0D%0A++++++repos%2C%0D%0A++++++json_each%28repos.topics%29+j%0D%0A++++where%0D%0A++++++j.value+%3D+%27datasette-plugin%27%0D%0A++%29%0D%0Aorder+by%0D%0A++id

[simonw]

I could have a scheduled task that looks for newly published plugins and opens issues (or even pull requests) to add them to the list though. That would be pretty cool.

[simonw]

pip install $(cat plugins.txt) works locally.

[simonw]

To skip commented out plugins: pip install $(cat plugins.txt | grep -v '#')

[simonw]

Got some warnings:

ERROR: datasette-upload-csvs 0.4 has requirement datasette~=0.37.1, but you'll have datasette 0.45 which is incompatible.
ERROR: datasette-configure-fts 0.4.2 has requirement sqlite-utils~=2.4.1, but you'll have sqlite-utils 2.10.1 which is incompatible.
ERROR: datasette-edit-tables 0.1a0 has requirement sqlite-utils~=2.4.2, but you'll have sqlite-utils 2.10.1 which is incompatible.

[simonw]

I've been shipping new versions of those to fix those errors.

• https://github.com/simonw/datasette-configure-fts/releases/tag/1.0
• https://github.com/simonw/datasette-edit-tables/releases/tag/0.2a

Still working on datasette-upload-csvs - see simonw/datasette-upload-csvs#11

[simonw]

The deploy failed: https://github.com/simonw/latest-datasette-with-all-plugins/runs/988901202?check_suite_focus=true

Deployment failed
ERROR: (gcloud.run.deploy) Cloud Run error: Container failed to start. Failed to start and then listen on the port defined by the PORT environment variable. Logs for this revision might contain more information.
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.8.5/x64/bin/datasette", line 11, in <module>
    load_entry_point('datasette', 'console_scripts', 'datasette')()
  File "/opt/hostedtoolcache/Python/3.8.5/x64/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.8.5/x64/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/opt/hostedtoolcache/Python/3.8.5/x64/lib/python3.8/site-packages/click/core.py", line 1259, in invoke

[simonw]

Here's the code I'm using to install every plugin from plugins.txt:

latest-datasette-with-all-plugins/.github/workflows/deploy.yml

Lines 66 to 71 in 402b131

	datasette publish cloudrun fixtures.db \
	-m latest-datasette-with-all-plugins/metadata.yml \
	--branch=main \
	--extra-options="--config template_debug:1" \
	--service=datasette-with-all-plugins \
	$(cat latest-datasette-with-all-plugins/plugins.txt | grep -v '#' | awk '{ print "--install " $1 }')

[simonw]

Found this in the Cloud Run logs:

insertId: "5f383e9a0009bafec3af96a9"  
 
labels: {…}  
 logName: "projects/datasette-222320/logs/run.googleapis.com%2Fstderr"  
 receiveTimestamp: "2020-08-15T19:59:22.642820594Z"  
 
resource: {…}  
 textPayload: "Traceback (most recent call last):
  File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.8/site-packages/datasette/database.py", line 92, in _execute_writes
    conn = self.connect(write=True)
  File "/usr/local/lib/python3.8/site-packages/datasette/database.py", line 55, in connect
    assert not (write and not self.is_mutable)"  
 timestamp: "2020-08-15T19:59:22.637694Z"  
}

[simonw]

It looks like a plugin may be attempting a write on startup.

In my local environment this works (with all those plugins installed):

datasette -i fixtures.db --get /

BUT... if I actually try to start the server I get an error:

datasette -i fixtures.db        
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/simon/.local/share/virtualenvs/latest-datasette-with-all-plugins-PJL_Xy9e/lib/python3.8/site-packages/datasette/database.py", line 92, in _execute_writes
    conn = self.connect(write=True)
  File "/Users/simon/.local/share/virtualenvs/latest-datasette-with-all-plugins-PJL_Xy9e/lib/python3.8/site-packages/datasette/database.py", line 55, in connect
    assert not (write and not self.is_mutable)
AssertionError

Hunch: I think the --get mechanism may not be running the init startup hook.

[simonw]

Yes, that's what's going on here: --get doesn't fully invoke the startup routine:

https://github.com/simonw/datasette/blob/7702ea602188899ee9b0446a874a6a9b546b564d/datasette/cli.py#L417-L433

    ds = Datasette(files, **kwargs)

    if get:
        client = TestClient(ds.app())
        response = client.get(get)
        click.echo(response.text)
        return

    if return_instance:
        # Private utility mechanism for writing unit tests
        return ds

    # Run the "startup" plugin hooks
    asyncio.get_event_loop().run_until_complete(ds.invoke_startup())

    # Run async sanity checks - but only if we're not under pytest
    asyncio.get_event_loop().run_until_complete(check_databases(ds))

I'm going to treat that as a bug in Datasette itself.

[simonw]

I shipped datasette 0.47.3 with that improvement.... and found a new problem!

% datasette -i fixtures.db  --get /            
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/simon/.local/share/virtualenvs/latest-datasette-with-all-plugins-PJL_Xy9e/lib/python3.8/site-packages/datasette/database.py", line 92, in _execute_writes
    conn = self.connect(write=True)
  File "/Users/simon/.local/share/virtualenvs/latest-datasette-with-all-plugins-PJL_Xy9e/lib/python3.8/site-packages/datasette/database.py", line 55, in connect
    assert not (write and not self.is_mutable)
AssertionError

This hangs! Ideally it should exit with an error instead.

[simonw]

After some debugging I tracked down the culprit: https://github.com/simonw/datasette-saved-queries/blob/48abb4181c4f06615156c97e7fe29aa0183d2fcf/datasette_saved_queries/__init__.py is attempting to write to the database in the startup() hook without first checking if it's immutable:

@hookimpl
def startup(datasette):
    async def inner():
        db = datasette.get_database()
        await db.execute_write_fn(create_tables, block=True)

    return inner

[simonw]

So maybe there's a Datasette bug with await db.execute_write_fn(create_tables, block=True) hanging if the function raises an exception?

[simonw]

Since latest-datasette-with-all-plugins directly checks out the Datasette repo this should mean that my datasette -i fixtures.db --get / test here will now fail if I manually execute a build.

[simonw]

Oops that's not quite right:

Usage: datasette serve [OPTIONS] [FILES]...
Try 'datasette serve --help' for help.

Error: Invalid value for '-i' / '--immutable': Path 'fixtures.db' does not exist.

[simonw]

It failed against the new test! Great - I'm going to comment out datasette-saved-queries and see if I can get a successful deploy.

[simonw]

It worked! https://latest-with-plugins.datasette.io/-/plugins