Phi Silica - Embeddings

[shaybenel]

Can you please further clarify the usage patterns for LanguageModel.GenerateEmbeddingVector* ?

[yeelam-gordon]

Will the code sample help?
WindowsAppSDK Sample – LanguageModelModel.cs (Line 72)

And the concept of embedding vectors in this case:
https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/14-Embeddings/README.md

[shaybenel]

@yeelam-gordon some follow-up clarifications:

Are these embeddings intended to be used for system prompt caching? I tried to experiment with leveraging these embeddings along with GenerateResponseFromEmbeddingsWithProgressAsync() to evaluate the latency savings impact, but when calling GenerateResponseFromEmbeddingsWithProgressAsync() I get a System.UnauthorizedAccessException: {"Access is denied.\r\n\r\nAccess is denied.\r\n"} (was running as administrator).
Am I trying to use this correctly? Is this error expected for the experimental version, i.e. not available yet?
Is there a full e2e example of how to leverage these embeddings?
Thanks

[connorMicrosoft]

Thanks for your feedback! We are discussing this Issue internally.

[connorMicrosoft]

Can you clarify how you're using the GenerateResponseFromEmbeddingsWithProgressAsync function? What are you trying to achieve here? Thanks!

[shaybenel]

I am evaluating if there are ways to increase generation throughput.
Specifically, if leveraging cached embeddings for recurring calls would substantiate improve my system’s latency.
In any case, my point is that it’s not clear from documentation what this API is to be used for.

[jorvanMS]

GenerateEmbeddingVector will return the lookup embeddings for each token in the vocab.

The general flow when a string prompt is passed through to the language model is that the string is first tokenized and then each token is converted into the corresponding embeddingvector.

In your case using cached embeddings will skip the tokenize->embedding lookup step when compared to passing in a string prompt. In terms of using it for the system prompt, right now you can only pass a string into the CreateContext for the system prompt, but that is a one-time call for the lifetime of the context object.

You should see slight improvements using cached embedding prompts versus string prompts. There are improvements to both the api shape and the documentation around embedding vectors coming in the next version of the api.