await datasette.client.get(path) mechanism for executing internal requests

[simonw]

datasette-graphql works by making internal requests to the TableView class (in order to take advantage of existing pagination logic, plus options like ?_search= and ?_where=) - see #915

I want to support a mod_rewrite style mechanism for putting nicer URLs on top of Datasette pages - I botched that together for a project here using an internal ASGI proxying trick: natbat/rockybeaches@ec102c6

If the datasette object provided a documented method for executing internal requests (in a way that makes sense with logging etc - i.e. doesn't get logged as a separate request) both of these use-cases would be much neater.

[simonw]

Could be as simple as response = await datasette.get("/path/blah") - which could also be re-used by the implementation of the datasette --get / CLI option introduced in #927.

Bit weird calling it .get() since that clashes with Python's dictionary .get() method.

[simonw]

Should it default to treating things as if they had the .json extension? There are use-cases for the non-JSON method, such as natbat/rockybeaches@ec102c6

I think I'm OK with people having to add .json to their internal calls. Maybe they could use format="json") as an optional parameter which would automatically handle the very weird edge-cases where you need to use ?_format=json instead of .json (due to table names existing with a .json suffix).

[simonw]

Alternative name possibilities:

• datasette.http_get(...) - slightly misleading since it's not going over the HTTP protocol
• datasette.internal_get(...) - the internal_ might suggest its not an API for external use, which isn't true - it's for plugins
• datasette.get(...) - clashes with dict.get() but I'm not at all sure that's a good reason not to use it

[simonw]

Actually no - requests.get() and httpx.get() prove that having a .get() method for an HTTP-related API isn't confusing to people at all.

datasette.get() it is.

(I'll probably add datasette.post() in the future too).

[simonw]

Should internal requests executed in this way be handled by plugins that used the asgi_wrapper() hook?

Hard to be sure one way or the other. I'm worried about logging middleware triggering twice - but actually anyone doing serious logging of their Datasette instance is probably doing it in a different layer (uvicorn logs or nginx proxy or whatever) so they wouldn't be affected. There aren't any ASGI logging middlewares out there that I've seen.

Also: if you run into a situation where your stuff is breaking because datasette.get() is calling ASGI middleware twice you can fix it by running your ASGI middleware outside of the asgi_wrapper plugin hook mechanism.

So I think it DOES execute asgi_wrapper() middleware.

[simonw]

What about authentication checks etc? Won't they run twice?

I think that's OK too, in fact it's desirable: think of the case of datasette-graphql where a bunch of different TableView calls are being made as part of the same GraphQL queries. Having those calls take advantage of finely grained per-table authentication and permission checks seems like a good feature.

[simonw]

Right now calling datasette.app() instantiates an ASGI application - complete with a bunch of routes and wrappers - and returns that application object. Calling it twice instantiates another ASGI application.

I think a single Datasette instance should only ever create a single ASGI app - so the .app() method should cache the ASGI app that it returns the first time and return the same application again on future calls.

[simonw]

One thing to consider here: Datasette's table and database name escaping rules can be a little bit convoluted.

If a plugin wants to get back the first five rows of a table, it will need to construct a URL /dbname/tablename?_size=5 - but it will need to know how to turn the database and table names into the correctly escaped dbname and tablename values.

Here's how the row.html table handles that right now:

datasette/datasette/templates/row.html

Lines 19 to 23 in b21ed23

	<p class="crumbs">
	<a href="{{ base_url }}">home</a> /
	<a href="{{ database_url(database) }}">{{ database }}</a> /
	<a href="{{ database_url(database) }}/{{ table|quote_plus }}">{{ table }}</a>
	</p>

It would be an improvement to have this logic abstracted out somewhere and documented so plugins can use it.

[simonw]

Maybe allow this:

response = await datasette.get("/{database}/{table}.json", database=database, table=table)

This could cause problems if users ever need to pass literal { in their paths. Maybe allow this too:

response = await datasette.get("/{database}/{table}.json", interpolate=False)

Not convinced this is useful - it's a bit unintuitive.

[simonw]

I just realised that this mechanism is kind of like being able to use microservices - make API calls within your application - except that everything runs in the same process against SQLite databases so calls will be lightning fast.

It also means that a plugin can add a new internal API to Datasette that's accessible to other plugins by registering a new route with register_routes!

[simonw]

Also fun: the inevitable plugin that exposes this to the template language - so Datasette templates can stitch together data from multiple other internal API calls. Fun way to take advantage of async support in Jinja.