Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prototype - start with ASGI hello world #1

Closed
simonw opened this issue Apr 30, 2022 · 32 comments
Closed

Prototype - start with ASGI hello world #1

simonw opened this issue Apr 30, 2022 · 32 comments

Comments

@simonw
Copy link
Owner

simonw commented Apr 30, 2022

Eventual goal is to run Datasette. For the moment I just want a "Hello world" served from a Python ASGI app that runs in a service worker - and ideally can respond "Hello NAME" if you hit it at /name.

@simonw
Copy link
Owner Author

simonw commented Apr 30, 2022

Here's hello world in ASGI:

async def hello_world(scope, receive, send):
    await send(
        {
            "type": "http.response.start",
            "status": 200,
            "headers": [
                [b"content-type", b"text/plain"],
            ],
        }
    )
    await send(
        {
            "type": "http.response.body",
            "body": b"Hello, world!",
        }
    )

@simonw
Copy link
Owner Author

simonw commented Apr 30, 2022

I'm going to try for a hello world service worker just written in JavaScript first.

@simonw
Copy link
Owner Author

simonw commented Apr 30, 2022

From examining this demo: https://mdn.github.io/sw-test/ - service worker code here: https://mdn.github.io/sw-test/sw.js

It looks like the key to this will be the event.respondWith() mechanism for the fetch event, see this fragment:

self.addEventListener('fetch', (event) => {
  event.respondWith(
    cacheFirst({
      request: event.request,
      preloadResponsePromise: event.preloadResponse,
      fallbackUrl: '/sw-test/gallery/myLittleVader.jpg',
    })
  );
});

https://developer.mozilla.org/en-US/docs/Web/API/FetchEvent/respondWith says:

The respondWith() method of FetchEvent prevents the browser's default fetch handling, and allows you to provide a promise for a Response yourself.

You can read event.request to get at the request: https://developer.mozilla.org/en-US/docs/Web/API/Request

@simonw
Copy link
Owner Author

simonw commented Apr 30, 2022

https://developer.mozilla.org/en-US/docs/Web/API/Service_Worker_API/Using_Service_Workers is a great starting tutorial.

Service workers are restricted to running across HTTPS for security reasons. GitHub is therefore a good place to host experiments, as it supports HTTPS. In order to facilitate local development, localhost is considered a secure origin by browsers as well.

So I can test with a localhost web server.

@simonw
Copy link
Owner Author

simonw commented Apr 30, 2022

I got that working - first TIL is: https://til.simonwillison.net/service-workers/intercept-fetch

@simonw
Copy link
Owner Author

simonw commented Apr 30, 2022

Starting to mess around with Pyodide now: https://pyodide.org/en/stable/usage/quickstart.html says:

Python code is run using the pyodide.runPython function. It takes as input a string of Python code. If the code ends in an expression, it returns the result of the expression, translated to JavaScript objects (see Type translations).

That sounds like an easy place to bridge between JavaScript and an ASGI app.

@simonw
Copy link
Owner Author

simonw commented Apr 30, 2022

There's documentation on using it with web workers here: https://pyodide.org/en/stable/usage/webworker.html

I imagine that will work fine with service workers too.

@simonw
Copy link
Owner Author

simonw commented May 1, 2022

@simonw
Copy link
Owner Author

simonw commented May 1, 2022

I'm seeing if I can install Datasette using micropip. Trying that here:

https://jupyterlite.readthedocs.io/en/latest/_static/lab/index.html

I ran:

import micropip
await micropip.install("datasette", keep_going=True)

And got this:

ValueError: Couldn't find a pure Python 3 wheel for: 'python-baseconv==1.2.2', 'click-default-group~=1.2.2'

@simonw
Copy link
Owner Author

simonw commented May 1, 2022

I got close. I made my own wheels for python-baseconv==1.2.2 and click-default-group~=1.2.2 by cloning those repos and running python3 -m build in them:

git clone https://github.com/semente/python-baseconv
cd python-baseconv
python3 -m build

Then I uploaded those wheels to my own S3 bucket with CORS enabled, see:

They are at:

It turned out I still needed to remove them from the setup.py list in Datasette - I applied this diff:

diff --git a/setup.py b/setup.py
index 7f0562f..46bb6ad 100644
--- a/setup.py
+++ b/setup.py
@@ -44,7 +44,7 @@ setup(
     install_requires=[
         "asgiref>=3.2.10,<3.6.0",
         "click>=7.1.1,<8.2.0",
-        "click-default-group~=1.2.2",
+        # "click-default-group~=1.2.2",
         "Jinja2>=2.10.3,<3.1.0",
         "hupper~=1.9",
         "httpx>=0.20",
@@ -57,7 +57,7 @@ setup(
         "PyYAML>=5.3,<7.0",
         "mergedeep>=1.1.1,<1.4.0",
         "itsdangerous>=1.1,<3.0",
-        "python-baseconv==1.2.2",
+        # "python-baseconv==1.2.2",
     ],
     entry_points="""
         [console_scripts]

Then I ran python3 -m build and uploaded that wheel file to my bucket too:

Then in https://pyodide.org/en/stable/console.html (to ensure the latest Pyodide) I ran this:

Welcome to the Pyodide terminal emulator 🐍
Python 3.10.2 (main, Apr  9 2022 20:52:01) on WebAssembly VM
Type "help", "copyright", "credits" or "license" for more information.
>>> import micropip
>>> await micropip.install(
    "https://s3.amazonaws.com/simonwillison-cors-allowed-public/python_baseconv-1.2.2-py3-none-any.whl",
    keep_going=True
)
>>> await micropip.install(
    "https://s3.amazonaws.com/simonwillison-cors-allowed-public/click_default_group-1.2.2-py3-none-any.whl",
    keep_going=True
)
>>> await micropip.install(
    "[https://files.pythonhosted.org/packages/4a/a8/0b2ced25639fb20cc1c9784de90a8c25f9504a7f18cd8b5397bd61696d7d/click-8.0.4-py3-none-any.wh](https://files.pythonhosted.org/packages/4a/a8/0b2ced25639fb20cc1c9784de90a8c25f9504a7f18cd8b5397bd61696d7d/click-8.0.4-py3-none-any.whl)
[l](https://files.pythonhosted.org/packages/4a/a8/0b2ced25639fb20cc1c9784de90a8c25f9504a7f18cd8b5397bd61696d7d/click-8.0.4-py3-none-any.whl)"
)
>>> await micropip.install(
    "https://s3.amazonaws.com/simonwillison-cors-allowed-public/datasette-0.61.1-py3-none-any.whl",
    keep_going=True
)
>>> micropip.list()
Name                | Version   | Source
------------------- | --------- | ---------------------------------------------------------------------------------------------------------
------------------------------
python_baseconv     | 1.2.2     | https://s3.amazonaws.com/simonwillison-cors-allowed-public/python_baseconv-1.2.2-py3-none-any.whl
click               | 8.0.4     | [https://files.pythonhosted.org/packages/4a/a8/0b2ced25639fb20cc1c9784de90a8c25f9504a7f18cd8b5397bd61696d7](https://files.pythonhosted.org/packages/4a/a8/0b2ced25639fb20cc1c9784de90a8c25f9504a7f18cd8b5397bd61696d7d/click-8.0.4-py3-)
[d/click-8.0.4-py3-](https://files.pythonhosted.org/packages/4a/a8/0b2ced25639fb20cc1c9784de90a8c25f9504a7f18cd8b5397bd61696d7d/click-8.0.4-py3-)
<long output truncated>
sette           | 0.61.1    | https://s3.amazonaws.com/simonwillison-cors-allowed-public/datasette-0.61.1-py3-none-any.whl
pyyaml              | 6.0       | pyodide
packaging           | 21.3      | pyodide
pluggy              | 1.0.0     | pyodide
typing-extensions   | 4.1.1     | pyodide
six                 | 1.16.0    | pyodide
markupsafe          | 2.1.1     | pyodide
distutils           | 1.0       | pyodide
micropip            | 0.1       | pyodide
pyparsing           | 3.0.7     | pyodide
>>> from datasette.app import Datasette
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/lib/python3.10/site-packages/datasette/app.py", line 9, in <module>
    import httpx
  File "/lib/python3.10/site-packages/httpx/__init__.py", line 2, in <module>
    from ._api import delete, get, head, options, patch, post, put, request, stream
  File "/lib/python3.10/site-packages/httpx/_api.py", line 4, in <module>
    from ._client import Client
  File "/lib/python3.10/site-packages/httpx/_client.py", line 9, in <module>
    from ._auth import Auth, BasicAuth, FunctionAuth
  File "/lib/python3.10/site-packages/httpx/_auth.py", line 10, in <module>
    from ._models import Request, Response
  File "/lib/python3.10/site-packages/httpx/_models.py", line 16, in <module>
    from ._content import ByteStream, UnattachedStream, encode_request, encode_response
  File "/lib/python3.10/site-packages/httpx/_content.py", line 17, in <module>
    from ._multipart import MultipartStream
  File "/lib/python3.10/site-packages/httpx/_multipart.py", line 7, in <module>
    from ._types import (
  File "/lib/python3.10/site-packages/httpx/_types.py", line 5, in <module>
    import ssl
  File "/lib/python3.10/ssl.py", line 98, in <module>
    import _ssl             # if we can't import it, let the error propagate
ModuleNotFoundError: No module named '_ssl'
>>> import pyodide_js
>>> pyodide_js.version
'0.20.0'
>>> import ssl
>>> import _ssl
>>> from datasette.app import Datasette
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/lib/python3.10/site-packages/datasette/app.py", line 14, in <module>
    import pkg_resources
ModuleNotFoundError: No module named 'pkg_resources'
>>> import pkg_resources
>>> from datasette.app import Datasette
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/lib/python3.10/site-packages/datasette/app.py", line 29, in <module>
    import uvicorn
  File "/lib/python3.10/site-packages/uvicorn/__init__.py", line 2, in <module>
    from uvicorn.main import Server, main, run
  File "/lib/python3.10/site-packages/uvicorn/main.py", line 24, in <module>
    from uvicorn.supervisors import ChangeReload, Multiprocess
  File "/lib/python3.10/site-packages/uvicorn/supervisors/__init__.py", line 3, in <module>
    from uvicorn.supervisors.basereload import BaseReload
  File "/lib/python3.10/site-packages/uvicorn/supervisors/basereload.py", line 12, in <module>
    from uvicorn.subprocess import get_subprocess
  File "/lib/python3.10/site-packages/uvicorn/subprocess.py", line 14, in <module>
    multiprocessing.allow_connection_pickling()
  File "/lib/python3.10/multiprocessing/context.py", line 170, in allow_connection_pickling
    from . import connection
  File "/lib/python3.10/multiprocessing/connection.py", line 21, in <module>
    import _multiprocessing
ModuleNotFoundError: No module named '_multiprocessing'
>>> import _multiprocessing
Traceback (most recent call last):
  File "<console>", line 1, in <module>
ModuleNotFoundError: No module named '_multiprocessing'
>>> 

@simonw
Copy link
Owner Author

simonw commented May 1, 2022

So in that case it broke because Uvicorn is a dependency.

Interesting how running import ssl before a line causes the error from a missing _ssl to not show up any more - same for pkg_resources.

@simonw
Copy link
Owner Author

simonw commented May 1, 2022

I tried the same sequence in Safari and got a different error:

Welcome to the Pyodide terminal emulator 🐍
Python 3.10.2 (main, Apr  9 2022 20:52:01) on WebAssembly VM
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyodide_js
>>> pyodide_js.version
'0.20.0'
>>> import micropip
>>> await micropip.install(
    "https://s3.amazonaws.com/simonwillison-cors-allowed-public/python_baseconv-1.2.2-py3-none-any.whl",
    keep_going=True
)
>>> await micropip.install(
    "https://s3.amazonaws.com/simonwillison-cors-allowed-public/click_default_group-1.2.2-py3-none-any.whl",
    keep_going=True
)
>>> await micropip.install(
    "[https://files.pythonhosted.org/packages/4a/a8/0b2ced25639fb20cc1c9784de90a8c25f9504a7f18cd8b5397bd61696d7d/click-8.0.](https://files.pythonhosted.org/packages/4a/a8/0b2ced25639fb20cc1c9784de90a8c25f9504a7f18cd8b5397bd61696d7d/click-8.0.4-py3-none-any.wh)
[4-py3-none-any.wh](https://files.pythonhosted.org/packages/4a/a8/0b2ced25639fb20cc1c9784de90a8c25f9504a7f18cd8b5397bd61696d7d/click-8.0.4-py3-none-any.wh)
l"
)
  File "<console>", line 2
    "[https://files.pythonhosted.org/packages/4a/a8/0b2ced25639fb20cc1c9784de90a8c25f9504a7f18cd8b5397bd61696d7d/click-8.0.](https://files.pythonhosted.org/packages/4a/a8/0b2ced25639fb20cc1c9784de90a8c25f9504a7f18cd8b5397bd61696d7d/click-8.0.4-py3-none-any.wh)
[4-py3-none-any.wh](https://files.pythonhosted.org/packages/4a/a8/0b2ced25639fb20cc1c9784de90a8c25f9504a7f18cd8b5397bd61696d7d/click-8.0.4-py3-none-any.wh)
    ^
SyntaxError: unterminated string literal (detected at line 2)
  File "<console>", line 1
    l"
     ^
SyntaxError: unterminated string literal (detected at line 1)
  File "<console>", line 1
    )
    ^
SyntaxError: unmatched ')'
>>> await micropip.install(
    "[https://files.pythonhosted.org/packages/4a/a8/0b2ced25639fb20cc1c9784de90a8c25f9504a7f18cd8b5397bd61696d7d/click-8.0.](https://files.pythonhosted.org/packages/4a/a8/0b2ced25639fb20cc1c9784de90a8c25f9504a7f18cd8b5397bd61696d7d/click-8.0.4-py3-none-any.whl)
[4-py3-none-any.whl](https://files.pythonhosted.org/packages/4a/a8/0b2ced25639fb20cc1c9784de90a8c25f9504a7f18cd8b5397bd61696d7d/click-8.0.4-py3-none-any.whl)")
>>> await micropip.install(
    "https://s3.amazonaws.com/simonwillison-cors-allowed-public/datasette-0.61.1-py3-none-any.whl",
    keep_going=True
)
ConsoleFuture exception was never retrieved
future: <ConsoleFuture finished exception=SyntaxError('unterminated string literal (detected at line 2)', ('<console>', 2,
 5, '    "[https://files.pythonhosted.org/packages/4a/a8/0b2ced25639fb20cc1c9784de90a8c25f9504a7f18cd8b5397bd61696d7d/click](https://files.pythonhosted.org/packages/4a/a8/0b2ced25639fb20cc1c9784de90a8c25f9504a7f18cd8b5397bd61696d7d/click-8.0.4-py3-none-any.wh)
[-8.0.4-py3-none-any.wh](https://files.pythonhosted.org/packages/4a/a8/0b2ced25639fb20cc1c9784de90a8c25f9504a7f18cd8b5397bd61696d7d/click-8.0.4-py3-none-any.wh)', 2, 5))>
ConsoleFuture exception was never retrieved
future: <ConsoleFuture finished exception=SyntaxError('unterminated string literal (detected at line 1)', ('<console>', 1,
 2, 'l"', 1, 2))>
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/lib/python3.10/asyncio/futures.py", line 284, in __await__
    yield self  # This tells Task to wait for completion.
  File "/lib/python3.10/asyncio/tasks.py", line 304, in __wakeup
    future.result()
  File "/lib/python3.10/asyncio/futures.py", line 201, in result
    raise self._exception
  File "/lib/python3.10/asyncio/tasks.py", line 234, in __step
    result = coro.throw(exc)
  File "/lib/python3.10/site-packages/micropip/_micropip.py", line 183, in install
    transaction = await self.gather_requirements(requirements, ctx, keep_going)
  File "/lib/python3.10/site-packages/micropip/_micropip.py", line 173, in gather_requirements
    await gather(*requirement_promises)
  File "/lib/python3.10/asyncio/futures.py", line 284, in __await__
    yield self  # This tells Task to wait for completion.
  File "/lib/python3.10/asyncio/tasks.py", line 304, in __wakeup
    future.result()
  File "/lib/python3.10/asyncio/futures.py", line 201, in result
    raise self._exception
  File "/lib/python3.10/asyncio/tasks.py", line 232, in __step
    result = coro.send(None)
  File "/lib/python3.10/site-packages/micropip/_micropip.py", line 245, in add_requirement
    await self.add_wheel(name, wheel, version, (), ctx, transaction)
  File "/lib/python3.10/site-packages/micropip/_micropip.py", line 316, in add_wheel
    await self.add_requirement(recurs_req, ctx, transaction)
  File "/lib/python3.10/site-packages/micropip/_micropip.py", line 291, in add_requirement
    await self.add_wheel(
  File "/lib/python3.10/site-packages/micropip/_micropip.py", line 316, in add_wheel
    await self.add_requirement(recurs_req, ctx, transaction)
  File "/lib/python3.10/site-packages/micropip/_micropip.py", line 291, in add_requirement
    await self.add_wheel(
  File "/lib/python3.10/site-packages/micropip/_micropip.py", line 316, in add_wheel
    await self.add_requirement(recurs_req, ctx, transaction)
  File "/lib/python3.10/site-packages/micropip/_micropip.py", line 276, in add_requirement
    raise ValueError(
ValueError: Requested 'h11<0.13,>=0.11', but h11==0.13.0 is already installed
>>> 

@simonw
Copy link
Owner Author

simonw commented May 1, 2022

My hunch here is that I can probably do a custom build of datasette (or datasette-core) that excludes uvicorn and get it working - if I build some extra wheels as seen above.

@simonw
Copy link
Owner Author

simonw commented May 1, 2022

Idea: I could set up CI on a branch in the simonw/datasette repo that builds a wheel and uploads it to an S3 bucket with CORS enabled created using s3-credentials.

@simonw
Copy link
Owner Author

simonw commented May 2, 2022

I still haven't managed to get Pyodide to load in a Service Worker.

This may have a clue:

I managed to get a debug shell up and running against the service worker via about:serviceworkers in Firefox and then searching for localhost - I'm seeing this error when I inspect it:

image

@simonw
Copy link
Owner Author

simonw commented May 2, 2022

Maybe this example (found via GitHub codesearch) could help: https://github.com/kahowell/nutrition/blob/2b74d5a6fc79d8ba526017a0ee93bb0b22903c99/pyodideapp/sw.js#L4

No, that's caching Pyodide but not actually executing it.

@simonw
Copy link
Owner Author

simonw commented May 2, 2022

My latest failing attempt looks like this:

importScripts("https://cdn.jsdelivr.net/pyodide/v0.20.0/full/pyodide.js");

let PYODIDE = null;

async function load() {
  if (!PYODIDE) {
    PYODIDE = await loadPyodide();
    console.log(await pyodide.runPythonAsync(`
    import sys
    sys.version
  `)); 
  }
  return PYODIDE;
}

self.addEventListener('fetch', async (event) => {
  const request = event.request;
  const url = (new URL(request.url));
  if (url.pathname == "/" || /\.js$/.exec(url.pathname) || /\.tar$/.exec(url.pathname) || /packages\.json$/.exec(url.pathname) || /pyodide.*$/.exec(url.pathname)) {
    // Don't intercept hits to the homepage or .js files
    return;
  }

  let pyodide = await load();

  // Pyodide is now ready to use...
  let pythonVersion = await pyodide.runPythonAsync(`
    import sys
    sys.version
  `);

  const params = new URLSearchParams(url.search);
  const info = {
    url: request.url,
    method: request.method,
    path: url.pathname,
    params: Array.from(params.entries()),
    pythonVersion: pythonVersion,
    a: 2,
    l: 'thang: ' + loadPyodide
  };
  event.respondWith(new Response(
    `<!DOCTYPE html><p>Hello world! Request was: <pre>${JSON.stringify(info, null, 4)}</p>`, {
    headers: { 'Content-Type': 'text/html' }
  }));
});

@simonw
Copy link
Owner Author

simonw commented May 2, 2022

@simonw
Copy link
Owner Author

simonw commented May 2, 2022

Managed to spot this error in Chrome DevTools:

CleanShot 2022-05-01 at 17 34 18@2x

Could it be that XMLHttpRequest is not available in Service Workers but IS available in Web Workers?

@simonw
Copy link
Owner Author

simonw commented May 2, 2022

https://stackoverflow.com/a/37114241/6083 confirms that XMLHttpRequest is available in service workers but not in web workers.

@simonw
Copy link
Owner Author

simonw commented May 2, 2022

https://web.dev/workers-overview/ explains the difference between service workers and web workers a bit, but doesn't mention XMLHttpRequest.

@simonw
Copy link
Owner Author

simonw commented May 2, 2022

@simonw
Copy link
Owner Author

simonw commented May 2, 2022

Maybe I can do this all in a Web Worker instead? I could intercept browser history events and run code when they happen instead.

@simonw
Copy link
Owner Author

simonw commented May 2, 2022

I built a Datasette wheel with the import uvicorn bit removed from app.py as well and it ALMOST worked...

Welcome to the Pyodide terminal emulator 🐍
Python 3.10.2 (main, Apr  9 2022 20:52:01) on WebAssembly VM
Type "help", "copyright", "credits" or "license" for more information.
>>> import micropip
>>> micropip.list()
Name      | Version | Source
--------- | ------- | -------
distutils | 1.0     | pyodide
micropip  | 0.1     | pyodide
packaging | 21.3    | pyodide
pyparsing | 3.0.7   | pyodide
>>> await micropip.install(
    "https://s3.amazonaws.com/simonwillison-cors-allowed-public/python_baseconv-1.2.2-py3-none-any.whl",
    keep_going=True
)
>>> await micropip.install(
    "https://s3.amazonaws.com/simonwillison-cors-allowed-public/click_default_group-1.2.2-py3-none-any.whl",
    keep_going=True
)
>>> await micropip.install(
    "https://s3.amazonaws.com/simonwillison-cors-allowed-public/datasette-0.61.1-py3-none-any.whl?",
    keep_going=True
)
Traceback (most recent call last):
  File "/lib/python3.10/site-packages/packaging/requirements.py", line 102, in __init__
    req = REQUIREMENT.parseString(requirement_string)
  File "/lib/python3.10/site-packages/pyparsing/core.py", line 1134, in parse_string
    raise exc.with_traceback(None)
pyparsing.exceptions.ParseException: Expected string_end, found ':'  (at char 5), (line:1, col:6)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/lib/python3.10/asyncio/futures.py", line 284, in __await__
    yield self  # This tells Task to wait for completion.
  File "/lib/python3.10/asyncio/tasks.py", line 304, in __wakeup
    future.result()
  File "/lib/python3.10/asyncio/futures.py", line 201, in result
    raise self._exception
  File "/lib/python3.10/asyncio/tasks.py", line 234, in __step
    result = coro.throw(exc)
  File "/lib/python3.10/site-packages/micropip/_micropip.py", line 183, in install
    transaction = await self.gather_requirements(requirements, ctx, keep_going)
  File "/lib/python3.10/site-packages/micropip/_micropip.py", line 173, in gather_requirements
    await gather(*requirement_promises)
  File "/lib/python3.10/asyncio/futures.py", line 284, in __await__
    yield self  # This tells Task to wait for completion.
  File "/lib/python3.10/asyncio/tasks.py", line 304, in __wakeup
    future.result()
  File "/lib/python3.10/asyncio/futures.py", line 201, in result
    raise self._exception
  File "/lib/python3.10/asyncio/tasks.py", line 232, in __step
    result = coro.send(None)
  File "/lib/python3.10/site-packages/micropip/_micropip.py", line 248, in add_requirement
    req = Requirement(requirement)
  File "/lib/python3.10/site-packages/packaging/requirements.py", line 104, in __init__
    raise InvalidRequirement(
packaging.requirements.InvalidRequirement: Parse error at "'://s3.am'": Expected string_end
>>> await micropip.install(
    "https://s3.amazonaws.com/simonwillison-cors-allowed-public/datasette-0.61.1-py3-none-any.whl",
    keep_going=True
)
>>> micropip.list()
Name                | Version   | Source
------------------- | --------- | -----------------------------------------------------------------------------------------
------------
python_baseconv     | 1.2.2     | [https://s3.amazonaws.com/simonwillison-cors-allowed-public/python_baseconv-1.2.2-py3-none](https://s3.amazonaws.com/simonwillison-cors-allowed-public/python_baseconv-1.2.2-py3-none-any.whl)
[-any.whl](https://s3.amazonaws.com/simonwillison-cors-allowed-public/python_baseconv-1.2.2-py3-none-any.whl)
click               | 8.1.3     | pypi
click_default_group | 1.2.2     | [https://s3.amazonaws.com/simonwillison-cors-allowed-public/click_default_group-1.2.2-py3-](https://s3.amazonaws.com/simonwillison-cors-allowed-public/click_default_group-1.2.2-py3-none-any.whl)
[none-any.whl](https://s3.amazonaws.com/simonwillison-cors-allowed-public/click_default_group-1.2.2-py3-none-any.whl)
jinja2          
<long output truncated>
sette           | 0.61.1    | https://s3.amazonaws.com/simonwillison-cors-allowed-public/datasette-0.61.1-py3-none-any.whl
markupsafe          | 2.1.1     | pyodide
typing-extensions   | 4.1.1     | pyodide
packaging           | 21.3      | pyodide
pluggy              | 1.0.0     | pyodide
pyyaml              | 6.0       | pyodide
six                 | 1.16.0    | pyodide
distutils           | 1.0       | pyodide
micropip            | 0.1       | pyodide
pyparsing           | 3.0.7     | pyodide
>>> from datasette.app import Datasette
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/lib/python3.10/site-packages/datasette/app.py", line 9, in <module>
    import httpx
  File "/lib/python3.10/site-packages/httpx/__init__.py", line 2, in <module>
    from ._api import delete, get, head, options, patch, post, put, request, stream
  File "/lib/python3.10/site-packages/httpx/_api.py", line 4, in <module>
    from ._client import Client
  File "/lib/python3.10/site-packages/httpx/_client.py", line 9, in <module>
    from ._auth import Auth, BasicAuth, FunctionAuth
  File "/lib/python3.10/site-packages/httpx/_auth.py", line 10, in <module>
    from ._models import Request, Response
  File "/lib/python3.10/site-packages/httpx/_models.py", line 16, in <module>
    from ._content import ByteStream, UnattachedStream, encode_request, encode_response
  File "/lib/python3.10/site-packages/httpx/_content.py", line 17, in <module>
    from ._multipart import MultipartStream
  File "/lib/python3.10/site-packages/httpx/_multipart.py", line 7, in <module>
    from ._types import (
  File "/lib/python3.10/site-packages/httpx/_types.py", line 5, in <module>
    import ssl
  File "/lib/python3.10/ssl.py", line 98, in <module>
    import _ssl             # if we can't import it, let the error propagate
ModuleNotFoundError: No module named '_ssl'
>>> 
>>> import ssl/
  File "<console>", line 1
    import ssl/
              ^
SyntaxError: invalid syntax
>>> import ssl
>>> import ssl/
  File "<console>", line 1
    import ssl/
              ^
SyntaxError: invalid syntax
>>> from datasette.app import Datasette
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/lib/python3.10/site-packages/datasette/app.py", line 14, in <module>
    import pkg_resources
ModuleNotFoundError: No module named 'pkg_resources'
>>> import pkg_resources
>>> from datasette.app import Datasette
>>> ds = Datasette(memory=True)
>>> await ds.client.get("/")
Traceback (most recent call last):
  File "/lib/python3.10/site-packages/datasette/app.py", line 1253, in route_path
    response = await view(request, send)
  File "/lib/python3.10/site-packages/datasette/views/base.py", line 134, in view
    return await self.dispatch_request(request)
  File "/lib/python3.10/site-packages/datasette/views/base.py", line 89, in dispatch_request
    await self.ds.refresh_schemas()
  File "/lib/python3.10/site-packages/datasette/app.py", line 350, in refresh_schemas
    await self._refresh_schemas()
  File "/lib/python3.10/site-packages/datasette/app.py", line 355, in _refresh_schemas
    await init_internal_db(internal_db)
  File "/lib/python3.10/site-packages/datasette/utils/internal_db.py", line 65, in init_internal_db
    await db.execute_write_script(create_tables_sql)
  File "/lib/python3.10/site-packages/datasette/database.py", line 113, in execute_write_script
    results = await self.execute_write_fn(_inner, block=block)
  File "/lib/python3.10/site-packages/datasette/database.py", line 144, in execute_write_fn
    self._write_thread.start()
  File "/lib/python3.10/threading.py", line 928, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread
<Response [500 Internal Server Error]>
>>> 

So the problem now is threads: Datasette tries to start them, and you can't do that in Pyodide.

@simonw
Copy link
Owner Author

simonw commented May 2, 2022

The test routine is:

import micropip
await micropip.install(
    "https://s3.amazonaws.com/simonwillison-cors-allowed-public/python_baseconv-1.2.2-py3-none-any.whl",
    keep_going=True
)
await micropip.install(
    "https://s3.amazonaws.com/simonwillison-cors-allowed-public/click_default_group-1.2.2-py3-none-any.whl",
    keep_going=True
)
await micropip.install(
    "https://s3.amazonaws.com/simonwillison-cors-allowed-public/datasette-0.61.1-py3-none-any.whl",
    keep_going=True
)
import ssl
import pkg_resources
from datasette.app import Datasette
ds = Datasette(memory=True)
await ds.client.get("/")

@simonw
Copy link
Owner Author

simonw commented May 2, 2022

OK well I got that to work!

Welcome to the Pyodide terminal emulator 🐍
Python 3.10.2 (main, Apr  9 2022 20:52:01) on WebAssembly VM
Type "help", "copyright", "credits" or "license" for more information.
>>> import micropip
await micropip.install(
    "https://s3.amazonaws.com/simonwillison-cors-allowed-public/python_baseconv-1.2.2-py3-none-any.whl",
    keep_going=True
)
await micropip.install(
    "https://s3.amazonaws.com/simonwillison-cors-allowed-public/click_default_group-1.2.2-py3-none-any.whl",
    keep_going=True
)
await micropip.install(
    "https://s3.amazonaws.com/simonwillison-cors-allowed-public/datasette-0.61.1-py3-none-any.whl",
    keep_going=True
)
import ssl
import pkg_resources
from datasette.app import Datasette
ds = Datasette(memory=True)
await ds.client.get("/")
<Response [200 OK]>
>>> r = _
>>> r.text
'<!DOCTYPE html>\n<html>\n<head>\n    <title>Datasette: _memory</title>\n    <link rel="stylesheet" href="/-/static/app.css
?cead5a">\n    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">\n\n<link rel="alterna
te" type="application/json+datasette" href="http://localhost/.json"></head>\n<body class="index">\n<div class="not-footer">
\n<header><nav>\n    \n    \n</nav></header>\n\n\n\n    \n\n\n\n<section class="content">\n\n<h1>Datasette</h1>\n\n\n\n\n\n
    <h2 
<long output truncated>
r detailsClickedWithin = null;\n    while (target && target.tagName != \'DETAILS\') {\n        target = target.parentNode;\
n    }\n    if (target && target.tagName == \'DETAILS\') {\n        detailsClickedWithin = target;\n    }\n    Array.from(d
ocument.getElementsByTagName(\'details\')).filter(\n        (details) => details.open && details != detailsClickedWithin\n 
   ).forEach(details => details.open = false);\n});\n</script>\n\n\n\n<!-- Templates considered: *index.html -->\n</body>\n
</html>'
>>> 

@simonw
Copy link
Owner Author

simonw commented May 2, 2022

Changes are in this branch: https://github.com/simonw/datasette/commits/8af32bc5b03c30b1f7a4a8cc4bd80eb7e2ee7b81

But it's just this diff so far:

diff --git a/datasette/app.py b/datasette/app.py
index d269372..6c0c5fc 100644
--- a/datasette/app.py
+++ b/datasette/app.py
@@ -15,7 +15,6 @@ import pkg_resources
 import re
 import secrets
 import sys
-import threading
 import traceback
 import urllib.parse
 from concurrent import futures
@@ -26,7 +25,6 @@ from itsdangerous import URLSafeSerializer
 from jinja2 import ChoiceLoader, Environment, FileSystemLoader, PrefixLoader
 from jinja2.environment import Template
 from jinja2.exceptions import TemplateNotFound
-import uvicorn
 
 from .views.base import DatasetteError, ureg
 from .views.database import DatabaseDownload, DatabaseView
@@ -813,7 +811,6 @@ class Datasette:
             },
             "datasette": datasette_version,
             "asgi": "3.0",
-            "uvicorn": uvicorn.__version__,
             "sqlite": {
                 "version": sqlite_version,
                 "fts_versions": fts_versions,
@@ -854,23 +851,7 @@ class Datasette:
         ]
 
     def _threads(self):
-        threads = list(threading.enumerate())
-        d = {
-            "num_threads": len(threads),
-            "threads": [
-                {"name": t.name, "ident": t.ident, "daemon": t.daemon} for t in threads
-            ],
-        }
-        # Only available in Python 3.7+
-        if hasattr(asyncio, "all_tasks"):
-            tasks = asyncio.all_tasks()
-            d.update(
-                {
-                    "num_tasks": len(tasks),
-                    "tasks": [_cleaner_task_str(t) for t in tasks],
-                }
-            )
-        return d
+        return {"num_threads": 0, "threads": []}
 
     def _actor(self, request):
         return {"actor": request.actor}
diff --git a/datasette/database.py b/datasette/database.py
index ba594a8..b50142d 100644
--- a/datasette/database.py
+++ b/datasette/database.py
@@ -4,7 +4,6 @@ from pathlib import Path
 import janus
 import queue
 import sys
-import threading
 import uuid
 
 from .tracer import trace
@@ -21,8 +20,6 @@ from .utils import (
 )
 from .inspect import inspect_hash
 
-connections = threading.local()
-
 AttachedDatabase = namedtuple("AttachedDatabase", ("seq", "name", "file"))
 
 
@@ -43,12 +40,12 @@ class Database:
         self.hash = None
         self.cached_size = None
         self._cached_table_counts = None
-        self._write_thread = None
-        self._write_queue = None
         if not self.is_mutable and not self.is_memory:
             p = Path(path)
             self.hash = inspect_hash(p)
             self.cached_size = p.stat().st_size
+        self._read_connection = None
+        self._write_connection = None
 
     @property
     def cached_table_counts(self):
@@ -134,60 +131,17 @@ class Database:
         return results
 
     async def execute_write_fn(self, fn, block=True):
-        task_id = uuid.uuid5(uuid.NAMESPACE_DNS, "datasette.io")
-        if self._write_queue is None:
-            self._write_queue = queue.Queue()
-        if self._write_thread is None:
-            self._write_thread = threading.Thread(
-                target=self._execute_writes, daemon=True
-            )
-            self._write_thread.start()
-        reply_queue = janus.Queue()
-        self._write_queue.put(WriteTask(fn, task_id, reply_queue))
-        if block:
-            result = await reply_queue.async_q.get()
-            if isinstance(result, Exception):
-                raise result
-            else:
-                return result
-        else:
-            return task_id
-
-    def _execute_writes(self):
-        # Infinite looping thread that protects the single write connection
-        # to this database
-        conn_exception = None
-        conn = None
-        try:
-            conn = self.connect(write=True)
-            self.ds._prepare_connection(conn, self.name)
-        except Exception as e:
-            conn_exception = e
-        while True:
-            task = self._write_queue.get()
-            if conn_exception is not None:
-                result = conn_exception
-            else:
-                try:
-                    result = task.fn(conn)
-                except Exception as e:
-                    sys.stderr.write("{}\n".format(e))
-                    sys.stderr.flush()
-                    result = e
-            task.reply_queue.sync_q.put(result)
+        # We always treat it as if block=True now
+        if self._write_connection is None:
+            self._write_connection = self.connect(write=True)
+            self.ds._prepare_connection(self._write_connection, self.name)
+        return fn(self._write_connection)
 
     async def execute_fn(self, fn):
-        def in_thread():
-            conn = getattr(connections, self.name, None)
-            if not conn:
-                conn = self.connect()
-                self.ds._prepare_connection(conn, self.name)
-                setattr(connections, self.name, conn)
-            return fn(conn)
-
-        return await asyncio.get_event_loop().run_in_executor(
-            self.ds.executor, in_thread
-        )
+        if self._read_connection is None:
+            self._read_connection = self.connect()
+            self.ds._prepare_connection(self._read_connection, self.name)
+        return fn(self._read_connection)
 
     async def execute(
         self,
diff --git a/setup.py b/setup.py
index 7f0562f..c41669c 100644
--- a/setup.py
+++ b/setup.py
@@ -44,20 +44,20 @@ setup(
     install_requires=[
         "asgiref>=3.2.10,<3.6.0",
         "click>=7.1.1,<8.2.0",
-        "click-default-group~=1.2.2",
+        # "click-default-group~=1.2.2",
         "Jinja2>=2.10.3,<3.1.0",
         "hupper~=1.9",
         "httpx>=0.20",
         "pint~=0.9",
         "pluggy>=1.0,<1.1",
-        "uvicorn~=0.11",
+        # "uvicorn~=0.11",
         "aiofiles>=0.4,<0.9",
         "janus>=0.6.2,<1.1",
         "asgi-csrf>=0.9",
         "PyYAML>=5.3,<7.0",
         "mergedeep>=1.1.1,<1.4.0",
         "itsdangerous>=1.1,<3.0",
-        "python-baseconv==1.2.2",
+        # "python-baseconv==1.2.2",
     ],
     entry_points="""
         [console_scripts]

@simonw
Copy link
Owner Author

simonw commented May 2, 2022

Here's the proof of concept with a web worker: webworker.js contains:

importScripts("https://cdn.jsdelivr.net/pyodide/dev/full/pyodide.js");

async function startDatasette() {
  self.pyodide = await loadPyodide({indexURL : "https://cdn.jsdelivr.net/pyodide/dev/full/"});
  await pyodide.loadPackage('micropip');
  await pyodide.loadPackage('ssl');
  await pyodide.loadPackage('setuptools'); // For pkg_resources
  await self.pyodide.runPythonAsync(`
  import micropip
  await micropip.install(
      "https://s3.amazonaws.com/simonwillison-cors-allowed-public/python_baseconv-1.2.2-py3-none-any.whl",
      keep_going=True
  )
  await micropip.install(
      "https://s3.amazonaws.com/simonwillison-cors-allowed-public/click_default_group-1.2.2-py3-none-any.whl",
      keep_going=True
  )
  await micropip.install(
      "https://s3.amazonaws.com/simonwillison-cors-allowed-public/datasette-0.61.1-py3-none-any.whl",
      keep_going=True
  )
  from datasette.app import Datasette
  ds = Datasette(memory=True)
  `);
}

let readyPromise = startDatasette();

self.onmessage = async (event) => {
  // make sure loading is done
  await readyPromise;
  // event.data has the incoming data - ignore for the moment
  try {
    let results = await self.pyodide.runPythonAsync(
      `
      import json
      json.dumps((await ds.client.get("/-/versions.json")).json())
      `
    );
    self.postMessage({ results });
  } catch (error) {
    self.postMessage({ error: error.message });
  }
};

And index.html contains:

<!DOCTYPE html>
<h1>Web worker demo</h1>

<script>
const datasetteWorker = new Worker("/webworker.js");

datasetteWorker.onmessage = (event) => {
  console.log(event, event.data);
};
datasetteWorker.postMessage({"path": "/"});
</script>

@simonw simonw changed the title ASGI hello world in a browser with a service worker and Pyodide Run Datasette using Pyodide - start with ASGI hello world May 2, 2022
@simonw
Copy link
Owner Author

simonw commented May 2, 2022

Latest version:

<!DOCTYPE html>
<h1>Web worker demo</h1>

<script>
const datasetteWorker = new Worker("/webworker.js");

datasetteWorker.onmessage = (event) => {
  console.log(event.data);
  document.getElementById("output").innerHTML = event.data.text;
  document.getElementById("status").innerHTML = event.data.status;
};
</script>
<form>
<p><input id="path" type="text" style="width: 80%" value="/.json"><input type="submit" value="Go"></p>
</form>
<p id="status"></p>
<div id="output"></div>

<script>
document.forms[0].onsubmit = function(ev) {
  ev.preventDefault();
  var path = document.getElementById("path").value;
  datasetteWorker.postMessage({path});
}
</script>

And webworker.js:

importScripts("https://cdn.jsdelivr.net/pyodide/dev/full/pyodide.js");

async function startDatasette() {
  self.pyodide = await loadPyodide({indexURL : "https://cdn.jsdelivr.net/pyodide/dev/full/"});
  await pyodide.loadPackage('micropip');
  await pyodide.loadPackage('ssl');
  await pyodide.loadPackage('setuptools'); // For pkg_resources
  await self.pyodide.runPythonAsync(`
  import micropip
  await micropip.install(
      "https://s3.amazonaws.com/simonwillison-cors-allowed-public/python_baseconv-1.2.2-py3-none-any.whl",
      keep_going=True
  )
  await micropip.install(
      "https://s3.amazonaws.com/simonwillison-cors-allowed-public/click_default_group-1.2.2-py3-none-any.whl",
      keep_going=True
  )
  await micropip.install(
      "https://s3.amazonaws.com/simonwillison-cors-allowed-public/datasette-0.61.1-py3-none-any.whl",
      keep_going=True
  )
  from datasette.app import Datasette
  ds = Datasette(memory=True)
  `);
}

let readyPromise = startDatasette();

self.onmessage = async (event) => {
  // make sure loading is done
  await readyPromise;
  console.log(event, event.data);
  try {
    let [status, text] = await self.pyodide.runPythonAsync(
      `
      import json
      response = await ds.client.get(${JSON.stringify(event.data.path)})
      [response.status_code, response.text]
      `
    );
    self.postMessage({status, text});
  } catch (error) {
    self.postMessage({ error: error.message });
  }
};

Result:

CleanShot 2022-05-01 at 21 34 27@2x

Need to solve CSS. I might just hard code that into index.html!

@simonw
Copy link
Owner Author

simonw commented May 2, 2022

<!DOCTYPE html>
<link rel="stylesheet" href="https://latest.datasette.io/-/static/app.css?cead5a">
<div style="padding: 1em">
<h1>Web worker demo</h1>

<script>
const datasetteWorker = new Worker("/webworker.js");

datasetteWorker.onmessage = (event) => {
  console.log(event.data);
  document.getElementById("output").innerHTML = event.data.text;
  document.getElementById("status").innerHTML = event.data.status;
};
</script>
<form>
<p><input id="path" type="text" style="width: 80%" value="/.json"><input type="submit" value="Go"></p>
</form>
<p id="status"></p>
<hr></div>

<div id="output"></div>

<script>
document.forms[0].onsubmit = function(ev) {
  ev.preventDefault();
  var path = document.getElementById("path").value;
  datasetteWorker.postMessage({path});
}
</script>

CleanShot 2022-05-01 at 21 39 38@2x

@simonw
Copy link
Owner Author

simonw commented May 2, 2022

I need to hook up event handlers so clicks on links and forms within that area are turned into web worker postMessage calls. Then I need to hook up the HTML5 history API - or probably switch to ugly # URLs because I don't want to have people get a 404 when they first visit a /foo/bar page.

@simonw simonw transferred this issue from another repository May 2, 2022
@simonw simonw changed the title Run Datasette using Pyodide - start with ASGI hello world Prototype - start with ASGI hello world May 2, 2022
@simonw
Copy link
Owner Author

simonw commented May 2, 2022

This prototype has served its purpose - I know that this is possible now. Moving this to a new repository and continuing the work in additional issues.

@simonw simonw closed this as completed May 2, 2022
simonw added a commit that referenced this issue May 2, 2022
@simonw simonw added this to the First public release milestone May 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant