New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Prototype - start with ASGI hello world #1
Comments
Here's hello world in ASGI: async def hello_world(scope, receive, send):
await send(
{
"type": "http.response.start",
"status": 200,
"headers": [
[b"content-type", b"text/plain"],
],
}
)
await send(
{
"type": "http.response.body",
"body": b"Hello, world!",
}
) |
I'm going to try for a hello world service worker just written in JavaScript first. |
From examining this demo: https://mdn.github.io/sw-test/ - service worker code here: https://mdn.github.io/sw-test/sw.js It looks like the key to this will be the self.addEventListener('fetch', (event) => {
event.respondWith(
cacheFirst({
request: event.request,
preloadResponsePromise: event.preloadResponse,
fallbackUrl: '/sw-test/gallery/myLittleVader.jpg',
})
);
}); https://developer.mozilla.org/en-US/docs/Web/API/FetchEvent/respondWith says:
You can read |
https://developer.mozilla.org/en-US/docs/Web/API/Service_Worker_API/Using_Service_Workers is a great starting tutorial.
So I can test with a |
I got that working - first TIL is: https://til.simonwillison.net/service-workers/intercept-fetch |
Starting to mess around with Pyodide now: https://pyodide.org/en/stable/usage/quickstart.html says:
That sounds like an easy place to bridge between JavaScript and an ASGI app. |
There's documentation on using it with web workers here: https://pyodide.org/en/stable/usage/webworker.html I imagine that will work fine with service workers too. |
I'm seeing if I can install Datasette using https://jupyterlite.readthedocs.io/en/latest/_static/lab/index.html I ran: import micropip
await micropip.install("datasette", keep_going=True) And got this:
|
I got close. I made my own wheels for
Then I uploaded those wheels to my own S3 bucket with CORS enabled, see: They are at:
It turned out I still needed to remove them from the diff --git a/setup.py b/setup.py
index 7f0562f..46bb6ad 100644
--- a/setup.py
+++ b/setup.py
@@ -44,7 +44,7 @@ setup(
install_requires=[
"asgiref>=3.2.10,<3.6.0",
"click>=7.1.1,<8.2.0",
- "click-default-group~=1.2.2",
+ # "click-default-group~=1.2.2",
"Jinja2>=2.10.3,<3.1.0",
"hupper~=1.9",
"httpx>=0.20",
@@ -57,7 +57,7 @@ setup(
"PyYAML>=5.3,<7.0",
"mergedeep>=1.1.1,<1.4.0",
"itsdangerous>=1.1,<3.0",
- "python-baseconv==1.2.2",
+ # "python-baseconv==1.2.2",
],
entry_points="""
[console_scripts] Then I ran Then in https://pyodide.org/en/stable/console.html (to ensure the latest Pyodide) I ran this:
|
So in that case it broke because Uvicorn is a dependency. Interesting how running |
I tried the same sequence in Safari and got a different error:
|
My hunch here is that I can probably do a custom build of |
Idea: I could set up CI on a branch in the |
I still haven't managed to get Pyodide to load in a Service Worker. This may have a clue: I managed to get a debug shell up and running against the service worker via |
Maybe this example (found via GitHub codesearch) could help: https://github.com/kahowell/nutrition/blob/2b74d5a6fc79d8ba526017a0ee93bb0b22903c99/pyodideapp/sw.js#L4 No, that's caching Pyodide but not actually executing it. |
My latest failing attempt looks like this: importScripts("https://cdn.jsdelivr.net/pyodide/v0.20.0/full/pyodide.js");
let PYODIDE = null;
async function load() {
if (!PYODIDE) {
PYODIDE = await loadPyodide();
console.log(await pyodide.runPythonAsync(`
import sys
sys.version
`));
}
return PYODIDE;
}
self.addEventListener('fetch', async (event) => {
const request = event.request;
const url = (new URL(request.url));
if (url.pathname == "/" || /\.js$/.exec(url.pathname) || /\.tar$/.exec(url.pathname) || /packages\.json$/.exec(url.pathname) || /pyodide.*$/.exec(url.pathname)) {
// Don't intercept hits to the homepage or .js files
return;
}
let pyodide = await load();
// Pyodide is now ready to use...
let pythonVersion = await pyodide.runPythonAsync(`
import sys
sys.version
`);
const params = new URLSearchParams(url.search);
const info = {
url: request.url,
method: request.method,
path: url.pathname,
params: Array.from(params.entries()),
pythonVersion: pythonVersion,
a: 2,
l: 'thang: ' + loadPyodide
};
event.respondWith(new Response(
`<!DOCTYPE html><p>Hello world! Request was: <pre>${JSON.stringify(info, null, 4)}</p>`, {
headers: { 'Content-Type': 'text/html' }
}));
}); |
I'm going to try the code from this example: https://github.com/pyodide/pyodide/blob/6900afdc5bc6bcb791484a69ba0c10240d2730ed/docs/usage/webworker.md#web-worker |
https://stackoverflow.com/a/37114241/6083 confirms that |
https://web.dev/workers-overview/ explains the difference between service workers and web workers a bit, but doesn't mention XMLHttpRequest. |
Filed an issue here: |
Maybe I can do this all in a Web Worker instead? I could intercept browser history events and run code when they happen instead. |
I built a Datasette wheel with the
So the problem now is threads: Datasette tries to start them, and you can't do that in Pyodide. |
The test routine is: import micropip
await micropip.install(
"https://s3.amazonaws.com/simonwillison-cors-allowed-public/python_baseconv-1.2.2-py3-none-any.whl",
keep_going=True
)
await micropip.install(
"https://s3.amazonaws.com/simonwillison-cors-allowed-public/click_default_group-1.2.2-py3-none-any.whl",
keep_going=True
)
await micropip.install(
"https://s3.amazonaws.com/simonwillison-cors-allowed-public/datasette-0.61.1-py3-none-any.whl",
keep_going=True
)
import ssl
import pkg_resources
from datasette.app import Datasette
ds = Datasette(memory=True)
await ds.client.get("/") |
OK well I got that to work!
|
Changes are in this branch: https://github.com/simonw/datasette/commits/8af32bc5b03c30b1f7a4a8cc4bd80eb7e2ee7b81 But it's just this diff so far: diff --git a/datasette/app.py b/datasette/app.py
index d269372..6c0c5fc 100644
--- a/datasette/app.py
+++ b/datasette/app.py
@@ -15,7 +15,6 @@ import pkg_resources
import re
import secrets
import sys
-import threading
import traceback
import urllib.parse
from concurrent import futures
@@ -26,7 +25,6 @@ from itsdangerous import URLSafeSerializer
from jinja2 import ChoiceLoader, Environment, FileSystemLoader, PrefixLoader
from jinja2.environment import Template
from jinja2.exceptions import TemplateNotFound
-import uvicorn
from .views.base import DatasetteError, ureg
from .views.database import DatabaseDownload, DatabaseView
@@ -813,7 +811,6 @@ class Datasette:
},
"datasette": datasette_version,
"asgi": "3.0",
- "uvicorn": uvicorn.__version__,
"sqlite": {
"version": sqlite_version,
"fts_versions": fts_versions,
@@ -854,23 +851,7 @@ class Datasette:
]
def _threads(self):
- threads = list(threading.enumerate())
- d = {
- "num_threads": len(threads),
- "threads": [
- {"name": t.name, "ident": t.ident, "daemon": t.daemon} for t in threads
- ],
- }
- # Only available in Python 3.7+
- if hasattr(asyncio, "all_tasks"):
- tasks = asyncio.all_tasks()
- d.update(
- {
- "num_tasks": len(tasks),
- "tasks": [_cleaner_task_str(t) for t in tasks],
- }
- )
- return d
+ return {"num_threads": 0, "threads": []}
def _actor(self, request):
return {"actor": request.actor}
diff --git a/datasette/database.py b/datasette/database.py
index ba594a8..b50142d 100644
--- a/datasette/database.py
+++ b/datasette/database.py
@@ -4,7 +4,6 @@ from pathlib import Path
import janus
import queue
import sys
-import threading
import uuid
from .tracer import trace
@@ -21,8 +20,6 @@ from .utils import (
)
from .inspect import inspect_hash
-connections = threading.local()
-
AttachedDatabase = namedtuple("AttachedDatabase", ("seq", "name", "file"))
@@ -43,12 +40,12 @@ class Database:
self.hash = None
self.cached_size = None
self._cached_table_counts = None
- self._write_thread = None
- self._write_queue = None
if not self.is_mutable and not self.is_memory:
p = Path(path)
self.hash = inspect_hash(p)
self.cached_size = p.stat().st_size
+ self._read_connection = None
+ self._write_connection = None
@property
def cached_table_counts(self):
@@ -134,60 +131,17 @@ class Database:
return results
async def execute_write_fn(self, fn, block=True):
- task_id = uuid.uuid5(uuid.NAMESPACE_DNS, "datasette.io")
- if self._write_queue is None:
- self._write_queue = queue.Queue()
- if self._write_thread is None:
- self._write_thread = threading.Thread(
- target=self._execute_writes, daemon=True
- )
- self._write_thread.start()
- reply_queue = janus.Queue()
- self._write_queue.put(WriteTask(fn, task_id, reply_queue))
- if block:
- result = await reply_queue.async_q.get()
- if isinstance(result, Exception):
- raise result
- else:
- return result
- else:
- return task_id
-
- def _execute_writes(self):
- # Infinite looping thread that protects the single write connection
- # to this database
- conn_exception = None
- conn = None
- try:
- conn = self.connect(write=True)
- self.ds._prepare_connection(conn, self.name)
- except Exception as e:
- conn_exception = e
- while True:
- task = self._write_queue.get()
- if conn_exception is not None:
- result = conn_exception
- else:
- try:
- result = task.fn(conn)
- except Exception as e:
- sys.stderr.write("{}\n".format(e))
- sys.stderr.flush()
- result = e
- task.reply_queue.sync_q.put(result)
+ # We always treat it as if block=True now
+ if self._write_connection is None:
+ self._write_connection = self.connect(write=True)
+ self.ds._prepare_connection(self._write_connection, self.name)
+ return fn(self._write_connection)
async def execute_fn(self, fn):
- def in_thread():
- conn = getattr(connections, self.name, None)
- if not conn:
- conn = self.connect()
- self.ds._prepare_connection(conn, self.name)
- setattr(connections, self.name, conn)
- return fn(conn)
-
- return await asyncio.get_event_loop().run_in_executor(
- self.ds.executor, in_thread
- )
+ if self._read_connection is None:
+ self._read_connection = self.connect()
+ self.ds._prepare_connection(self._read_connection, self.name)
+ return fn(self._read_connection)
async def execute(
self,
diff --git a/setup.py b/setup.py
index 7f0562f..c41669c 100644
--- a/setup.py
+++ b/setup.py
@@ -44,20 +44,20 @@ setup(
install_requires=[
"asgiref>=3.2.10,<3.6.0",
"click>=7.1.1,<8.2.0",
- "click-default-group~=1.2.2",
+ # "click-default-group~=1.2.2",
"Jinja2>=2.10.3,<3.1.0",
"hupper~=1.9",
"httpx>=0.20",
"pint~=0.9",
"pluggy>=1.0,<1.1",
- "uvicorn~=0.11",
+ # "uvicorn~=0.11",
"aiofiles>=0.4,<0.9",
"janus>=0.6.2,<1.1",
"asgi-csrf>=0.9",
"PyYAML>=5.3,<7.0",
"mergedeep>=1.1.1,<1.4.0",
"itsdangerous>=1.1,<3.0",
- "python-baseconv==1.2.2",
+ # "python-baseconv==1.2.2",
],
entry_points="""
[console_scripts] |
Here's the proof of concept with a web worker: importScripts("https://cdn.jsdelivr.net/pyodide/dev/full/pyodide.js");
async function startDatasette() {
self.pyodide = await loadPyodide({indexURL : "https://cdn.jsdelivr.net/pyodide/dev/full/"});
await pyodide.loadPackage('micropip');
await pyodide.loadPackage('ssl');
await pyodide.loadPackage('setuptools'); // For pkg_resources
await self.pyodide.runPythonAsync(`
import micropip
await micropip.install(
"https://s3.amazonaws.com/simonwillison-cors-allowed-public/python_baseconv-1.2.2-py3-none-any.whl",
keep_going=True
)
await micropip.install(
"https://s3.amazonaws.com/simonwillison-cors-allowed-public/click_default_group-1.2.2-py3-none-any.whl",
keep_going=True
)
await micropip.install(
"https://s3.amazonaws.com/simonwillison-cors-allowed-public/datasette-0.61.1-py3-none-any.whl",
keep_going=True
)
from datasette.app import Datasette
ds = Datasette(memory=True)
`);
}
let readyPromise = startDatasette();
self.onmessage = async (event) => {
// make sure loading is done
await readyPromise;
// event.data has the incoming data - ignore for the moment
try {
let results = await self.pyodide.runPythonAsync(
`
import json
json.dumps((await ds.client.get("/-/versions.json")).json())
`
);
self.postMessage({ results });
} catch (error) {
self.postMessage({ error: error.message });
}
}; And <!DOCTYPE html>
<h1>Web worker demo</h1>
<script>
const datasetteWorker = new Worker("/webworker.js");
datasetteWorker.onmessage = (event) => {
console.log(event, event.data);
};
datasetteWorker.postMessage({"path": "/"});
</script> |
Latest version: <!DOCTYPE html>
<h1>Web worker demo</h1>
<script>
const datasetteWorker = new Worker("/webworker.js");
datasetteWorker.onmessage = (event) => {
console.log(event.data);
document.getElementById("output").innerHTML = event.data.text;
document.getElementById("status").innerHTML = event.data.status;
};
</script>
<form>
<p><input id="path" type="text" style="width: 80%" value="/.json"><input type="submit" value="Go"></p>
</form>
<p id="status"></p>
<div id="output"></div>
<script>
document.forms[0].onsubmit = function(ev) {
ev.preventDefault();
var path = document.getElementById("path").value;
datasetteWorker.postMessage({path});
}
</script> And importScripts("https://cdn.jsdelivr.net/pyodide/dev/full/pyodide.js");
async function startDatasette() {
self.pyodide = await loadPyodide({indexURL : "https://cdn.jsdelivr.net/pyodide/dev/full/"});
await pyodide.loadPackage('micropip');
await pyodide.loadPackage('ssl');
await pyodide.loadPackage('setuptools'); // For pkg_resources
await self.pyodide.runPythonAsync(`
import micropip
await micropip.install(
"https://s3.amazonaws.com/simonwillison-cors-allowed-public/python_baseconv-1.2.2-py3-none-any.whl",
keep_going=True
)
await micropip.install(
"https://s3.amazonaws.com/simonwillison-cors-allowed-public/click_default_group-1.2.2-py3-none-any.whl",
keep_going=True
)
await micropip.install(
"https://s3.amazonaws.com/simonwillison-cors-allowed-public/datasette-0.61.1-py3-none-any.whl",
keep_going=True
)
from datasette.app import Datasette
ds = Datasette(memory=True)
`);
}
let readyPromise = startDatasette();
self.onmessage = async (event) => {
// make sure loading is done
await readyPromise;
console.log(event, event.data);
try {
let [status, text] = await self.pyodide.runPythonAsync(
`
import json
response = await ds.client.get(${JSON.stringify(event.data.path)})
[response.status_code, response.text]
`
);
self.postMessage({status, text});
} catch (error) {
self.postMessage({ error: error.message });
}
}; Result: Need to solve CSS. I might just hard code that into |
<!DOCTYPE html>
<link rel="stylesheet" href="https://latest.datasette.io/-/static/app.css?cead5a">
<div style="padding: 1em">
<h1>Web worker demo</h1>
<script>
const datasetteWorker = new Worker("/webworker.js");
datasetteWorker.onmessage = (event) => {
console.log(event.data);
document.getElementById("output").innerHTML = event.data.text;
document.getElementById("status").innerHTML = event.data.status;
};
</script>
<form>
<p><input id="path" type="text" style="width: 80%" value="/.json"><input type="submit" value="Go"></p>
</form>
<p id="status"></p>
<hr></div>
<div id="output"></div>
<script>
document.forms[0].onsubmit = function(ev) {
ev.preventDefault();
var path = document.getElementById("path").value;
datasetteWorker.postMessage({path});
}
</script> |
I need to hook up event handlers so clicks on links and forms within that area are turned into web worker |
This prototype has served its purpose - I know that this is possible now. Moving this to a new repository and continuing the work in additional issues. |
Eventual goal is to run Datasette. For the moment I just want a "Hello world" served from a Python ASGI app that runs in a service worker - and ideally can respond "Hello NAME" if you hit it at
/name
.The text was updated successfully, but these errors were encountered: