?_col= and ?_nocol= support for toggling columns on table view

[simonw]

Split off from #292 (I guess this is a re-opening of #312).

[simonw]

First step: stop using select * and switch to selecting columns instead. This will also make the View and edit SQL button more useful.

[simonw]

Demo: visit https://52fa79c.datasette.io/fixtures/roadside_attractions and click "View and edit SQL" - you'll be sent to https://52fa79c.datasette.io/fixtures?sql=select+pk%2C+name%2C+address%2C+latitude%2C+longitude+from+roadside_attractions+order+by+pk+limit+101

[simonw]

I'm going to always include the primary key (or rowid) - otherwise the table view will sometimes not include links to the row pages, which seems bad.

[simonw]

I started looking at this again, inspired by #1326. I have a new diff that works against the latest main branch.

diff --git a/datasette/views/table.py b/datasette/views/table.py
index 4879228..f4b2ee2 100644
--- a/datasette/views/table.py
+++ b/datasette/views/table.py
@@ -64,6 +64,36 @@ class Row:
 
 
 class RowTableShared(DataView):
+    async def columns_to_select(self, db, table, request):
+        table_columns = await db.table_columns(table)
+        if "_col" in request.args and "_nocol" in request.args:
+            raise DatasetteError("Cannot use _col and _nocol at the same time")
+        if "_col" in request.args:
+            new_columns = []
+            for column in request.args.getlist("_col"):
+                if column not in table_columns:
+                    raise DatasetteError("_col={} is an invalid column".format(column))
+                new_columns.append(column)
+            return new_columns
+        elif "_nocol" in request.args:
+            # Return all columns EXCEPT these
+            bad_columns = [
+                column
+                for column in request.args.getlist("_nocol")
+                if column not in table_columns
+            ]
+            if bad_columns:
+                raise DatasetteError(
+                    "_nocol={} - invalid columns".format(", ".join(bad_columns))
+                )
+            return [
+                column
+                for column in table_columns
+                if column not in request.args.getlist("_nocol")
+            ]
+        else:
+            return table_columns
+
     async def sortable_columns_for_table(self, database, table, use_rowid):
         db = self.ds.databases[database]
         table_metadata = self.ds.table_metadata(database, table)
@@ -321,18 +351,16 @@ class TableView(RowTableShared):
         )
 
         pks = await db.primary_keys(table)
-        table_column_details = await db.table_column_details(table)
-        table_columns = [column.name for column in table_column_details]
-
-        select_columns = ", ".join(escape_sqlite(t) for t in table_columns)
+        table_columns = await self.columns_to_select(db, table, request)
+        select_clause = ", ".join(escape_sqlite(t) for t in table_columns)
 
         use_rowid = not pks and not is_view
         if use_rowid:
-            select = f"rowid, {select_columns}"
+            select = f"rowid, {select_clause}"
             order_by = "rowid"
             order_by_pks = "rowid"
         else:
-            select = select_columns
+            select = select_clause
             order_by_pks = ", ".join([escape_sqlite(pk) for pk in pks])
             order_by = order_by_pks
 
@@ -715,6 +743,8 @@ class TableView(RowTableShared):
                 column = fk["column"]
                 if column not in columns_to_expand:
                     continue
+                if column not in columns:
+                    continue
                 expanded_columns.append(column)
                 # Gather the values
                 column_index = columns.index(column)

[simonw]

Still needed:

• Unit tests
• Documentation

And maybe a UI mechanism for this? I could at least add a "remove this column" item to the cog menu on columns.

[simonw]

Here's that prototype of a "Hide this column" cog menu button:

Need a way to undo that once you've hidden a column - maybe a list of currently hidden columns that lets you un-hide them.

[simonw]

Natalie suggests a quick way to implement "undo" would be to add a "Show all columns" item to that menu which only appears when at least one column is hidden.

[simonw]

I've changed my mind about forbidding ?_col= and ?_nocol= from being applied at the same time - I'm going to come up with a simple resolution rule instead.

[simonw]

Here's a bug: removing the rowid column returns an error.

[simonw]

A better interface for this would be a full list of columns each with a checkbox for making it visible on invisible - this could then be used to apply a bulk change (rather than refreshing the interface after every removed column) and it could also be easily designed to work on narrow mobile screens where the cog icon is not visible.

[simonw]

Interesting side-effect of this implementation is that you can both control column order and request the same column multiple times:

/fivethirtyeight/airline-safety%2Fairline-safety?_col=fatal_accidents_00_14&_col=fatalities_00_14&_col=airline&_col=airline

[simonw]

Would it be useful to allow this mechanism to alias columns, for example supporting one of the following:

• ?_col=airline as name_of_airline
• ?_col=airline:name_of_airline

This could be handy for renaming columns to match a specific expected JSON output.

[simonw]

I can't think of any good reasons to support passing the same column twice ?col=airline&_col=airline so I'm going to de-duplicate and silently ignore the second one.

[simonw]

Here's a bug: removing the rowid column returns an error.

Removing the rowid column should work. We can continue to show the Link column, ensuring users can still navigate to the row page for each row.

[simonw]

I tried allowing the removal of rowid and got this exception:

  File "/Users/simon/Dropbox/Development/datasette/datasette/views/table.py", line 831, in extra_template
    display_columns, display_rows = await self.display_columns_and_rows(
  File "/Users/simon/Dropbox/Development/datasette/datasette/views/table.py", line 163, in display_columns_and_rows
    pk_path = path_from_row_pks(row, pks, not pks, False)
  File "/Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py", line 82, in path_from_row_pks
    bits = [row["rowid"]]
IndexError: No item with that key

I'm going to disable the removal of rowid - or indeed any of the primary keys, since they are needed to construct the row permalink.

[simonw]

Demos:

• https://latest.datasette.io/fixtures/sortable?_nocol=sortable
• https://latest.datasette.io/fixtures/sortable?_col=sortable&_col=text

Also try the column cog menu on that page.

Documentation: https://docs.datasette.io/en/latest/json_api.html#special-table-arguments