Datasette 1.0 rc tracking issue

[simonw]

The milestone is here: https://github.com/simonw/datasette/milestone/33 - this issue will provide some added structure and commentary.

The three goals for Datasette 1.0 are plugin stability, API stability, metadata stability and template context stability. Plus ensuring there aren't any vestigial problems which will block us from making fully backwards-compatible changes in 1.1 and higher.

Plugin stability and permissions

This is mostly there.

I had thought get_metadata() might be a sticking point, but actually I think that only needs a small change:

• get_metadata() plugin hook should support async await_me_maybe pattern #2331

We should also think about if the canned queries hook is the correct design.

My bigger question is around permissions. The existing permission_allowed() hook has a design flaw: it can answer the question "can user X do action Y on resource Z" but it doesn't offer an efficient path for questions like "list all tables this user can view" or "list all users that can perform this specific action".

The internal database is designed to help here - by giving every table an internal representation that can then be joined against to answer these kinds of questions.

The efficient way to answer these questions is to have a permissions system that works on the basis of SQL fragments - but this could require significant rethinking of how the hooks and permissions system works.

This is the biggest open question for 1.0 right now.

One option: keep the existing permission_allowed() hook (since dozens of plugins use it already), but offer a new plugin hook which uses the SQL-level permissions instead. If your plugin implements that hook then it will be shown in lists of users-who-can-do-x and things-user-x-can-do AND you'll get the permission allowed behavior for free.

That way plugins that don't upgrade will remain secure, while plugins that DO upgrade will gain extra functionality.

Relevant issues:

• Efficiently calculate list things a user can do / list of users who can do a thing #1152
• Permissions defined in datasette.yml do not correctly obey the veto rule #2292
• /-/permissions should list available permissions #1943
• check_visibility gives confusing (wrong?) results if permission is None #2018
• Reconsider ensure_permissions() logic, can it be less confusing? #1676

API stability

This is mostly there - the most important API endpoints are tables and queries, and those are both now on the new format - where they return "rows": [{"id": 1, "name": "name"}] style responses.

One sticking point is the ?_extra= parameter, which works on tables but not queries and is not yet documented or stabilized

• Add ?_extra= mechanism for requesting extra properties in JSON #262

Other Datasette endpoints also need some work - the /.json and /data.json endpoints need a redesign.

Homepage redesign - tables, not databases

The Datasette / homepage isn't useful enough yet. I think it needs to list all tables in the Datasette instance, no matter which database they are in - rather than just listing databases and a few tables for each. This list should be paginated and should respect permissions.

The table is the most important unit within Datasette and should be treated as such.

Metadata stability

• Mechanism for storing metadata in _metadata tables #1168

But also: cascading metadata is confusing, and probably only makes sense for licenses. Do we want people to add their own custom metadata keys? If yes, we need to document how that should work.

Context stability

The idea behind context stability is to ensure that users who create custom templates for their Datasette instance will not see their templates break if they upgrade from 1.0 to 1.1. This requires a review of the existing context design, ideally completely merging it with the JSON output (using extras). Then it requires comprehensive documentation, and tests that protect against the documented context breaking by accident in the future.