feat: alternate search index for nimbus #38662
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Updates to Preview Branch (charis/docs-415-spike-of-adapting-current-database-to-hide-pages-in-the) ↗︎
Tasks are run on every commit but only new migration files are pushed.
View logs for this Workflow Run ↗︎. |
| @@ -0,0 +1,325 @@ | |||
| -- Create nimbus tables for feature-flag-filtered search | |||
There was a problem hiding this comment.
I didn't change any logic on these tables and functions, they're just copied over from the existing versions except for appending _nimbus to the names.
Probably some of these are redundant or obsolete, but let's fix that later, just getting the alt search index working now
Create an alternate search index for Nimbus that filters out
feature-flagged pages (equivalent to setting all feature flags to
false).
Notes:
- Creates two new DB tables, `page_nimbus` and `page_section_nimbus`,
which are filtered versions of `page` and `page_section`
- Makes `nimbus` versions of all the DB search functions
- Refactored the embedding upload script. Changes to make it faster (got
annoyed by how slow it was when testing...), incorporate retries, and
produce better summary logs.
- Upload script, when run with the environment variable
ENABLED_FEATURES_OVERRIDE_DISABLE_ALL, produces and uploads the
alternate search index
- Changed all the search calls in frontend/API to check for
`isFeatureEnabled('search:fullIndex')` to determine whether to search
the full or alternate index
charislam
force-pushed
the
charis/docs-415-spike-of-adapting-current-database-to-hide-pages-in-the
branch
from
c640b8a to
f46b193
Compare
|
@charislam Is there a step missing between 6 and 7? I tried and don't seem to see a difference, even rerunning the embeddings and no difference. |
|
@ChrisChinchilla hmmm I don't think so, here is the comparison on my local: with 
with 
(the dart references at results 4 & 5 should be gone) |
|
@charislam Either I missunderstood something or a simple "doing it again" fixed the issue. All good now. |
…database-to-hide-pages-in-the
github-actions
bot
added
the
documentation
charislam
deleted the
charis/docs-415-spike-of-adapting-current-database-to-hide-pages-in-the
branch
2 participants
Create an alternate search index for Nimbus that filters out feature-flagged pages (equivalent to setting all feature flags to
false).
Notes:
page_nimbusandpage_section_nimbus, which are filtered versions ofpageandpage_sectionnimbusversions of all the DB search functionsENABLED_FEATURES_OVERRIDE_DISABLE_ALL, produces and uploads the alternate search indexisFeatureEnabled('search:fullIndex')to determine whether to search the full or alternate indexTo test
It'll be easiest to test this on local.
pnpm supabase start. If this errors you may need to add dummyGITHUB_CLIENT_IDandGITHUB_SECRETto your.env.pnpm supabase migration up --localapps/docs/.env.localhasNEXT_PUBLIC_SUPABASE_URL,NEXT_PUBLIC_SUPABASE_ANON_KEY, andSUPABASE_SECRET_KEYpointing to local Supabasepnpm embeddings:refreshandpnpm embeddings:nimbusenabled-features.json, toggle off thesearch:fullIndexfeature flag