New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Facets #255

Closed
simonw opened this Issue May 12, 2018 · 16 comments

Comments

Projects
None yet
1 participant
@simonw
Owner

simonw commented May 12, 2018

Ability to display facets and facet counts on the table view.

Facets can be specified in the URL with ?_facet=column&_facet=othercolumn or the default facets for a table can be set using a new "facets": [...] property in metadata.json

  • Implement ?_facet=
  • Implement metadata.json facets key
  • Design for how facets should be presented
  • Facets should be able to toggle off as well as on
  • Expand labels for facets that are foreign keys
  • Suggest potential facets (if we can do so within a tight time limit)
  • Documentation
@simonw

This comment has been minimized.

Owner

simonw commented May 12, 2018

Facet counts will be generated by extra SQL queries with their own aggressive time limit.

@simonw simonw changed the title from Faceted search to Facets May 12, 2018

simonw added a commit that referenced this issue May 12, 2018

@simonw

This comment has been minimized.

Owner

simonw commented May 12, 2018

Adding some TODOs to the original description (so they show up as a todo progress bar)

@simonw

This comment has been minimized.

Owner

simonw commented May 12, 2018

@simonw

This comment has been minimized.

@simonw

This comment has been minimized.

Owner

simonw commented May 14, 2018

It would be neat if there was a mechanism for calculating aggregates per facet - e.g. calculating the sum() of specific columns against each facet result on https://datasette-facets-demo.now.sh/fivethirtyeight-2628db9/nba-elo%2Fnbaallelo?_facet=lg_id&_facet=fran_id&lg_id=ABA&_facet=team_id

@simonw

This comment has been minimized.

Owner

simonw commented May 14, 2018

Can I get facets working across many2many relationships?

This would be fiendishly useful, but the querystring and metadata.json syntax is non-obvious.

@simonw

This comment has been minimized.

Owner

simonw commented May 14, 2018

To decide which facets to suggest: for each column, is the unique value count less than the number of rows matching the current query or is it less than 20 (if we are showing more than 20 rows)?

Maybe only do this if there are less than ten non-float columns. Or always try for foreign keys and booleans, then if there are none of those try indexed text and integer fields, then finally try non-indexed text and integer fields but only if there are less than ten.

simonw added a commit that referenced this issue May 15, 2018

simonw added a commit that referenced this issue May 15, 2018

simonw added a commit that referenced this issue May 15, 2018

Facet results now have "truncated" field
 To indicate if there was more than 20 distinct values. Refs #255

simonw added a commit that referenced this issue May 15, 2018

simonw added a commit that referenced this issue May 15, 2018

simonw added a commit that referenced this issue May 15, 2018

@simonw

This comment has been minimized.

Owner

simonw commented May 15, 2018

Activity has now moved to this branch: https://github.com/simonw/datasette/commits/suggested-facets

@simonw

This comment has been minimized.

Owner

simonw commented May 15, 2018

New demo (published with datasette publish now --branch=suggested-facets fivethirtyeight.db sf-trees.db --name=datastte-suggested-facets-demo): https://datasette-suggested-facets-demo.now.sh/fivethirtyeight-2628db9/comic-characters%2Fmarvel-wikia-data

After turning on a couple of suggested facets... https://datasette-suggested-facets-demo.now.sh/fivethirtyeight-2628db9/comic-characters%2Fmarvel-wikia-data?_facet=SEX&_facet=ID

2018-05-15 at 7 24 am

simonw added a commit that referenced this issue May 16, 2018

@simonw

This comment has been minimized.

Owner

simonw commented May 16, 2018

I updated that demo to demonstrate the new foreign key label expansions: https://datasette-suggested-facets-demo.now.sh/sf-trees-02c8ef1/Street_Tree_List?_facet=qLegalStatus

2018-05-15 at 8 58 pm

simonw added a commit that referenced this issue May 16, 2018

@simonw

This comment has been minimized.

Owner

simonw commented May 16, 2018

Maybe suggested_facets should only be calculated for the HTML view.

simonw added a commit that referenced this issue May 16, 2018

simonw added a commit that referenced this issue May 16, 2018

@simonw

This comment has been minimized.

simonw added a commit that referenced this issue May 16, 2018

simonw added a commit that referenced this issue May 16, 2018

Facet results now have "truncated" field
 To indicate if there was more than 20 distinct values. Refs #255

simonw added a commit that referenced this issue May 16, 2018

@simonw

This comment has been minimized.

Owner

simonw commented May 16, 2018

This is now landed in master, ready for the next release.

@simonw simonw closed this May 16, 2018

simonw added a commit that referenced this issue May 17, 2018

simonw added a commit that referenced this issue May 17, 2018

simonw added a commit that referenced this issue May 18, 2018

Show facets that timed out using new InterruptedError
If the user requests some _facet= options that do not successfully execute in
the configured facet_time_limit_ms, we now show a warning message like this:

    These facets timed out: rowid, Title

To build this I had to clean up our SQLite interrupted logic. We now raise a
custom InterruptedError exception when SQLite terminates due to exceeding a
time limit.

In implementing this I found and fixed a logic error where invalid SQL was
being generated in some cases for our faceting calculations but the resulting
sqlite3.OperationalError had been incorrectly captured and treated as a
timeout.

Refs #255
Closes #269
@simonw

This comment has been minimized.

Owner

simonw commented May 22, 2018

This shipped in Datasette 0.22. Here's my blog post about it: https://simonwillison.net/2018/May/20/datasette-facets/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment