You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Updating and querying this table is very slow and bogs down both the loader and ranking calculations by a substantial amount. The actual data itself is pretty small, 75,000,000 x 16 bytes = 1.2 GB in prod, but all the indices and uniqueness constraints blows this up to ~10 GB in MariaDB. If we split the data across nodes this would be a few hundred megabytes each, and we'd shave off hours of processing.
Sketch of a solution:
Keep it in memory in the index service and have some API for querying it (maybe via QS)
Loader writes to file to a file which gets switched over atomically as a whole batch by the index svc
The text was updated successfully, but these errors were encountered:
We also probably want to rename the LinkDB component to something else, to avoid confusion, since it's not really links but documents. To do this we need a migration step to rename the file...
Updating and querying this table is very slow and bogs down both the loader and ranking calculations by a substantial amount. The actual data itself is pretty small, 75,000,000 x 16 bytes = 1.2 GB in prod, but all the indices and uniqueness constraints blows this up to ~10 GB in MariaDB. If we split the data across nodes this would be a few hundred megabytes each, and we'd shave off hours of processing.
Sketch of a solution:
The text was updated successfully, but these errors were encountered: