Skip to content

🚀 FastFlowLM v0.9.15: New Embedding Capabilities & API Integration

Choose a tag to compare

View all tags
@FastFlowLM FastFlowLM released this 24 Oct 22:17
· 1090 commits to main since this release
v0.9.15
d5f340a
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

🔎 1. New Model: EmbeddingGemma-300m

The first Embedding model on FLM:

  • Runs fully offline on AMD Ryzen™ AI NPU
  • Supports chunk sizes up to 2048 tokens

Try it out:

  1. Start flm in server mode with embedding model enabled:
flm serve gemma3:4b --embed 1 # Load embedding model in background, with concurrent LLM loading (gemma3:4b).

⚠️ Note: Embedding model is not allowed in CLI mode.

🌐 2. OpenAI-Compatible Embedding API: v1/embeddings

FastFlowLM now supports the OpenAI v1/embeddings endpoint making it easy to integrate embedding model into any OpenAI-compatible client or UI.

How to use:

  1. Start FLM server with embedding model enabled:
# serve
flm serve gemma3:4b --embed 1 # Load embedding model in background, with concurrent LLM loading (gemma3:4b).
  1. Send file(s) to:
    POST /v1/embeddings
    via any OpenAI client or Open WebUI.

Examples: Test in OpenAI client

from openai import OpenAI

client = OpenAI(
   base_url="http://localhost:52625/v1", # FastFlowLM's local API endpoint
   api_key="flm", # Dummy key (FastFlowLM doesn't require authentication)
)

resp = client.embeddings.create(
   model="embed-gemma",
   input="Hi, everyone!"
)

print(resp.data[0].embedding)

Example: Open WebUI

  1. Follow Open WebUI setup guide.
  2. In the bottom-left corner, click User icon, then select Settings.
  3. In the bottom panel, open Admin Settings.
  4. In the left sidebar, navigate to Documents.
  5. Set Embedding Model Engine to OpenAI.
  6. Enter:
    -- API Base URL: http://host.docker.internal:52625/v1
    -- API KEY: flm (any value works)
    -- Embedding Model: embed-gemma:300m
  7. Save the setting.
  8. Follow the RAG + FastFlowLM example to launch your Local Private Database with RAG all powered by FLM.

🙏 Acknowledgement

Special thanks to julienM77 for contributing the message normalizer that improves handling of corrupted user messages.

🌟 Summary

FastFlowLM v0.9.15 introduces offline embedding with embedding gemma and support for OpenAI's v1/embeddings API. Just start the server with --embed 1 and you're ready to build local, private, and intelligent applications.

Assets 3
Loading