Initial CLI support and plugin hook for embeddings

[simonw]

Refs:

• Initial abstraction for running embeddings #185

[simonw]

Here's an example plugin that works with https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2

import llm
from sentence_transformers import SentenceTransformer


@llm.hookimpl
def register_embedding_models(register):
    model_id = "sentence-transformers/all-MiniLM-L6-v2"
    register(SentenceTransformerModel(model_id, model_id, 384))


class SentenceTransformerModel(llm.EmbeddingModel):
    def __init__(self, model_id, model_name, embedding_size):
        self.model_id = model_id
        self.model_name = model_name
        self.embedding_size = embedding_size
        self._model = None

    def embed(self, text):
        if self._model is None:
            self._model = SentenceTransformer(self.model_name)
        return list(self._model.encode([text])[0])

[simonw]

I introduced https://github.com/simonw/sqlite-migrate here, because I need a mechanism for ensuring any database passed to llm embed -d/--database has the necessary tables - independent of the migrations that create the responses etc tables in the main logs.db database.

I should absolutely port the other LLM migrations to this system too, having two migrations systems is gnarly.

[simonw]

In order to write tests for this I'm going to build a plugin that only activates during tests, which has a VERY simple embedding mechanism: it returns 16 floats, each float is the length of a word in the input text, truncates to 16 words and fills in with 0 for missing words.

I'll use this pattern: https://til.simonwillison.net/pytest/registering-plugins-in-tests

[simonw]

I'm going to document this, then land it and then work on llm similar separately.