Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Datasette Library #417

Open
simonw opened this issue Mar 15, 2019 · 3 comments

Comments

@simonw
Copy link
Owner

commented Mar 15, 2019

The ability to run Datasette in a mode where it automatically picks up new (or modified) files in a directory tree without needing to restart the server.

Suggested command:

datasette library /path/to/mydbs/
@simonw

This comment has been minimized.

Copy link
Owner Author

commented Mar 15, 2019

This would allow Datasette to be easily used as a "data library" (like a data warehouse but less expectation of big data querying technology such as Presto).

One of the things I learned at the NICAR CAR 2019 conference in Newport Beach is that there is a very real need for some kind of easily accessible data library at most newsrooms.

@simonw

This comment has been minimized.

Copy link
Owner Author

commented Mar 15, 2019

A neat ability of Datasette Library would be if it can work against other files that have been dropped into the folder. In particular: if a user drops a CSV file into the folder, how about automatically converting that CSV file to SQLite using sqlite-utils?

@psychemedia

This comment has been minimized.

Copy link
Contributor

commented Mar 19, 2019

This would be really interesting but several possibilities in use arise, I think?

For example:

  • I put a new CSV file into the import dir and a new table is created therefrom
  • I put a CSV file into the import dir that replaces a previous file / table of the same name as a pre-existing table (eg files that contain monthly data in year to date). The data may also patch previous months, so a full replace / DROP on the original table may well be in order.
  • I put a CSV file into the import dir that updates a table of the same name as a pre-existing table (eg files that contain last month's data)

CSV files may also have messy names compared to the table you want. Or for an update CSV, may have the form MYTABLENAME-February2019.csv etc

simonw added a commit that referenced this issue Jul 26, 2019
Refs #417

First proof-of-concept for Datasette Library. Run like this:

    datasette -d ~/Library

Uses a new plugin hook - available_databases()

BUT... I don't think this is quite the way I want to go.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.