Routers with Smolagents

[DomPTech]

Is there anyway to reuse a loaded Smolagents model (e.g. a Transformers Model) for semantic analysis? Specifically I'm using this semantic router framework with a LocalEncoder. The computer I'm using needs all of the memory it can get, so I'd like to reuse the model loaded into memory for the encoder. Any help is much appreciated, I'm new to this!

[Christian-Sidak]

Good question! The short answer is that TransformersModel in smolagents wraps a causal language model pipeline (text generation), not an encoder/embedding model. These are architecturally different -- an LLM like Llama or Mistral is optimized for next-token prediction, not for producing meaningful sentence-level embeddings that work well for semantic similarity.

You could technically extract the underlying model and tokenizer and do mean pooling over the hidden states, but the results would be poor for routing compared to a purpose-built encoder. Here is what that would look like (not recommended):

import torch
from semantic_router.encoders import BaseEncoder

class SmolagentsHackyEncoder(BaseEncoder):
    """Reuses the LLM backbone from TransformersModel as an encoder. Not recommended."""
    def __init__(self, smolagents_model):
        self.model = smolagents_model.model
        self.tokenizer = smolagents_model.tokenizer
        self.model.eval()

    def __call__(self, docs: list[str]) -> list[list[float]]:
        embeddings = []
        for doc in docs:
            inputs = self.tokenizer(doc, return_tensors="pt", truncation=True, max_length=512)
            inputs = {k: v.to(self.model.device) for k, v in inputs.items()}
            with torch.no_grad():
                outputs = self.model(**inputs, output_hidden_states=True)
            # Mean pool over the last hidden state
            hidden = outputs.hidden_states[-1]
            mask = inputs["attention_mask"].unsqueeze(-1)
            embedding = (hidden * mask).sum(dim=1) / mask.sum(dim=1)
            embeddings.append(embedding.squeeze().cpu().tolist())
        return embeddings

The better approach, even on a memory-constrained machine: load a small dedicated sentence embedding model alongside your LLM. Something like sentence-transformers/all-MiniLM-L6-v2 is only ~80MB and is specifically trained for semantic similarity. That is a rounding error compared to even a small LLM.

from semantic_router.encoders import HuggingFaceEncoder
from smolagents import TransformersModel, CodeAgent

# Your LLM for the agent (this is the big one)
model = TransformersModel(
    model_id="meta-llama/Llama-3.2-3B-Instruct",
    device_map="auto",
)

# Small encoder for routing - only ~80MB extra
encoder = HuggingFaceEncoder(name="sentence-transformers/all-MiniLM-L6-v2")

# Set up your semantic router
from semantic_router import Route, RouteLayer

routes = [
    Route(name="coding", utterances=["write code", "fix this bug", "implement a function"]),
    Route(name="search", utterances=["find information", "look up", "search for"]),
]

router = RouteLayer(encoder=encoder, routes=routes)

# Use the router to pick a route, then dispatch to the right agent
route = router("help me write a Python script")
print(route.name)  # -> "coding"

The dedicated encoder will give you dramatically better routing accuracy for a negligible memory cost. The LLM hidden states were never trained to cluster semantically similar sentences together, so reusing them for routing would likely give you inconsistent results.

Hope that helps!

[VANDRANKI]

Christian-Sidak is right that TransformersModel is a causal LM, not an encoder. Here is the practical path forward for memory-constrained setups.

Option 1: Use a tiny dedicated encoder (recommended)

Models like sentence-transformers/all-MiniLM-L6-v2 use only ~90MB of RAM — negligible compared to any LLM. Load it separately and pass it to LocalEncoder:

from semantic_router.encoders import HuggingFaceEncoder
from smolagents import CodeAgent
from smolagents.models import TransformersModel

# Tiny encoder — 90MB, runs on CPU fine
encoder = HuggingFaceEncoder(name="sentence-transformers/all-MiniLM-L6-v2")

# Your full LLM for the agent
model = TransformersModel(model_id="HuggingFaceTB/SmolLM2-1.7B-Instruct")
agent = CodeAgent(tools=[...], model=model)

This keeps the two models separate and the encoder's footprint is small enough that it won't compete with the LLM for RAM.

Option 2: Share the underlying model object (advanced, not recommended)

If you truly cannot afford the extra memory, you can extract the pipeline from TransformersModel and do mean pooling yourself:

import torch

transformers_model = TransformersModel(model_id="your-model")
hf_model = transformers_model.model  # underlying transformers pipeline
tokenizer = transformers_model.tokenizer

def embed(text):
    inputs = tokenizer(text, return_tensors="pt", truncation=True, padding=True)
    with torch.no_grad():
        outputs = hf_model.model(**inputs)
    # Mean pool over token dimension
    return outputs.last_hidden_state.mean(dim=1).squeeze().numpy()

The quality will be poor for routing since causal LMs are not trained for embedding similarity, but it works if memory is the hard constraint.

For most cases Option 1 is the right answer — the encoder is cheap and the quality difference is significant.