Skip to content

GENERATED column support #1116

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
nattaylor opened this issue Nov 30, 2020 · 9 comments
Closed

GENERATED column support #1116

nattaylor opened this issue Nov 30, 2020 · 9 comments
Labels

Comments

@nattaylor
Copy link

I think this is a feature request... perhaps I should just try to contribute it myself, but thought I'd check in case support is planned already.

For a table with the following schema, datasette 0.51.1 doesn't pick up the GENERATED columns and the column list only contains (rowid, body) If I edit the SQL and select the generated columns, it will happily show them.

At first glance it appears that def table_column_details(conn, table): would have to be refactored to use a different methodology to get the columns, since PRAGMA table_info(deeds); returns just 0|body|TEXT|0||0 so maybe it wouldn't be worth it.

CREATE TABLE deeds (
    body TEXT,
    id INT GENERATED ALWAYS AS (json_extract(body, '$.id')) STORED,
    consideration INT GENERATED ALWAYS AS (json_extract(body, '$.consideration')) STORED
);
@simonw simonw added the bug label Nov 30, 2020
@simonw
Copy link
Owner

simonw commented Nov 30, 2020

I'm treating this as a bug - these columns should definitely be visible in Datasette.

I created my own test database using SQLite from Homebrew like this:

/usr/local/Cellar/sqlite/3.33.0/bin/sqlite3 deeds.db << EOF
CREATE TABLE deeds (
    body TEXT,
    id INT GENERATED ALWAYS AS (json_extract(body, '$.id')) STORED,
    consideration INT GENERATED ALWAYS AS (json_extract(body, '$.consideration')) STORED
);
INSERT INTO deeds (body) VALUES ('{
    "id": 1,
    "consideration": "This is the consideration"
}');
EOF

@simonw
Copy link
Owner

simonw commented Nov 30, 2020

@simonw
Copy link
Owner

simonw commented Nov 30, 2020

Generated column support was added in SQLite 3.31.0, so any unit tests I write for this should use skipIf to only run on that version or later.

@simonw
Copy link
Owner

simonw commented Nov 30, 2020

This change will also have an impact on how hidden virtual FTS tables are displayed, since apparently those have some hidden columns: https://latest.datasette.io/fixtures?sql=select+*+from+pragma_table_xinfo%28%27searchable_fts%27%29

cid name type notnull dflt_value pk hidden
0 text1 0 0 0
1 text2 0 0 0
2 name with . and spaces 0 0 0
3 searchable_fts 0 0 1
4 docid 0 0 1
5 __langid 0 0 1

@nattaylor
Copy link
Author

@simonw thanks for investigating so quickly. If it is undesirable to change that hidden behavior, maybe something like this is a suitable workaround:

SELECT * FROM pragma_table_xinfo('deeds') where hidden in (0,2);
0|body|TEXT|0||0|0
1|id|INT GENERATED ALWAYS|0||0|2
2|consideration|INT GENERATED ALWAYS|0||0|2

@simonw simonw closed this as completed in 37f87b5 Nov 30, 2020
@simonw
Copy link
Owner

simonw commented Nov 30, 2020

I'm OK exposing hidden columns, unless someone comes up with a pressing reason not to.

@simonw
Copy link
Owner

simonw commented Nov 30, 2020

Ouch, the tests pass on my laptop but failed in CI: https://github.com/simonw/datasette/actions/runs/392367997

Lots of failures look like this:

ERROR: conn=<sqlite3.Connection object at 0x7f44f0494030>, sql = 'select rowid,  from facetable order by rowid limit 51', params = {}: near "from": syntax error

Note the select rowid, from... - so it looks like invalid SQL queries are being constructed maybe due to mis-detecting columns somehow.

I wonder why it didn't fail on my laptop?

@simonw
Copy link
Owner

simonw commented Nov 30, 2020

Working on this in a pull request: #1117

@simonw simonw reopened this Nov 30, 2020
@simonw
Copy link
Owner

simonw commented Nov 30, 2020

Here's the problem: https://www.sqlite.org/changes.html#version_3_26_0

2018-12-01 (3.26.0)

CI is running 3.22.0.

@simonw simonw closed this as completed in 461670a Nov 30, 2020
simonw added a commit that referenced this issue Dec 3, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants