You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Admin search over taxonomy terms, bylines, users, and media currently runs unindexed full-table scans. These should be made indexable so search latency doesn't degrade linearly with row count.
Problem
Each of these entities is searched with a case-insensitive substring filter:
Media — lower(filename) LIKE '%term%' (database/repositories/media.ts)
Bylines — display_name/slug LIKE '%term%' (database/repositories/byline.ts)
Users — email/name LIKE '%term%' (auth/adapters/kysely.ts)
Taxonomy terms — no search at all today
Plain B-tree indexes already exist on several of these columns (idx_users_email, idx_media_filename, idx_taxonomies_name, byline slug), but none of them can serve these queries:
A leading % wildcard (%term%) is unindexable by a B-tree in both SQLite and Postgres.
lower(col) doesn't match a plain index on col, so even prefix queries would scan.
Net result: every search is a sequential scan. Fine on small sites, but user / term / byline / media tables can reach 10^5–10^6 rows on real deployments, where this is a real latency hit (and on D1, every scanned row is a cost).
Goal
Search over these entities should be always indexed — no full-table scans:
Appropriate indexes exist.
The query is always able to use them.
Proposed approach
Use SQLite FTS5 (consistent with the existing content search). The FTS5 inverted index makes token / token-prefix MATCH queries indexed, which is what these typeahead-style fields need. Tokenizer: unicode61 (stemming layer TBD — keeping porter is on the table).
Notes / open questions:
Filename tradeoff (accepted): FTS5 tokenizes on word boundaries, so it won't do mid-string / extension substring matching on media filenames the way LIKE '%...%' does. Considered acceptable.
Dialect: FTS5 is SQLite-only. The existing content search no-ops on Postgres; this should follow the same precedent (SQLite/D1 = real, Postgres = fallback) unless we decide to add a Postgres path (tsvector/pg_trgm).
Machinery: the current FTSManager is hardwired to the ec_* / _emdash_collections collection model. These four are system tables, so it needs generalizing, or dedicated per-entity FTS tables + sync triggers (respecting each table's locale / deleted_at where present).
Out of scope (deferred)
Trigram indexing (the indexed answer for true substring search).
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Summary
Admin search over taxonomy terms, bylines, users, and media currently runs unindexed full-table scans. These should be made indexable so search latency doesn't degrade linearly with row count.
Problem
Each of these entities is searched with a case-insensitive substring filter:
lower(filename) LIKE '%term%'(database/repositories/media.ts)display_name/slug LIKE '%term%'(database/repositories/byline.ts)email/name LIKE '%term%'(auth/adapters/kysely.ts)Plain B-tree indexes already exist on several of these columns (
idx_users_email,idx_media_filename,idx_taxonomies_name, bylineslug), but none of them can serve these queries:%wildcard (%term%) is unindexable by a B-tree in both SQLite and Postgres.lower(col)doesn't match a plain index oncol, so even prefix queries would scan.Net result: every search is a sequential scan. Fine on small sites, but user / term / byline / media tables can reach 10^5–10^6 rows on real deployments, where this is a real latency hit (and on D1, every scanned row is a cost).
Goal
Search over these entities should be always indexed — no full-table scans:
Proposed approach
Use SQLite FTS5 (consistent with the existing content search). The FTS5 inverted index makes token / token-prefix
MATCHqueries indexed, which is what these typeahead-style fields need. Tokenizer:unicode61(stemming layer TBD — keepingporteris on the table).Notes / open questions:
LIKE '%...%'does. Considered acceptable.tsvector/pg_trgm).FTSManageris hardwired to theec_*/_emdash_collectionscollection model. These four are system tables, so it needs generalizing, or dedicated per-entity FTS tables + sync triggers (respecting each table'slocale/deleted_atwhere present).Out of scope (deferred)
All reactions