Skip to content

0.1a3

Pre-release
Pre-release
Compare
Choose a tag to compare
@simonw simonw released this 08 Apr 00:06
8dc282b

This alpha changes the way embedding vectors are stored. Previously they were stored on the table itself, as columns with names like emb_text_embedding_3_large_256. #3

They are now instead stored in a separate table, with the name _embeddings_TABLE where TABLE is the name of the original table.

If you have previously calculated embedding vectors stored in columns on an existing table, you can migrate them to the new table structure by following these steps: #11

  1. Create a new shadow table to store your embeddings. This table should have the name _embeddings_TABLE where TABLE is the name of your source table, and the same primary key (or compound primary keys). It should then have a emb_xxx column for each stored embedding vector.

    One way to create this table is to select a single row and embed just that - this will create the table for you while only costing you one embedding run.

  2. Run the following SQL query to copy across your vectors:

    insert or replace into _embeddings_TABLE (id, emb_xxx)
    select id, emb_xxx from TABLE where emb_xxx is not null
  3. Drop the old column: alter table TABLE drop column emb_xxx