Generator.AppendTokens / 'embedding' chain routing for non-VLM Llama models with --extra_options exclude_embeds=true

[PMeeske]

Use case

I'm building runtime-mutable embeddings for a custom assistant — the goal is to let downstream subsystems (memory, persona, prompt compressor) drive per-token embedding drift without retraining the model. The natural mechanism is to use --extra_options exclude_embeds=true so the embedding tensor lives outside the graph, then either (a) feed inputs_embeds per call from C# or (b) override the embedding sub-graph's initializer at session creation.

What I observed

When I build a Llama-typed model (NousResearch/Hermes-3-Llama-3.1-8B, model.type: "llama" in genai_config.json) with --extra_options exclude_embeds=true, the resulting genai_config.json correctly declares inputs_embeds as the decoder input:

"decoder": {
  "inputs": {
    "inputs_embeds": "inputs_embeds",
    "attention_mask": "attention_mask",
    "position_ids": "position_ids",
    ...
  }
}

I then hand-built a small embedding.onnx that does Gather(embed_tokens.weight, input_ids) -> inputs_embeds (mirroring the structure used in test/test_models/qwen-vision-preprocessing/genai_config.json) and added the corresponding embedding block:

"embedding": {
  "filename": "embedding.onnx",
  "inputs": {"input_ids": "input_ids"},
  "outputs": {"inputs_embeds": "inputs_embeds"},
  "session_options": {"log_id": "...", "provider_options": []}
}

Generator.AppendTokens(prompt_ids) then fails with:

RuntimeError: Model input was not found: input_ids

Tested on 0.13.2 Python bindings, CPU EP. The Generator looks up input_ids on the main decoder graph, not on the embedding sub-graph.

Question

Is the embedding block's input_ids -> inputs_embeds chain intended to route only for VLM model.type values (qwen2_5_vl, gemma3, etc.), or is the lack of routing for Llama-typed models an unintended limitation?

The README documents exclude_embeds=true as a generally-available builder option, and the embedding block schema is the natural way to wire a separate embedding source for any decoder. The Qwen-VL test fixture (test/test_models/qwen-vision-preprocessing/genai_config.json) is the only example I've found with a non-trivial embedding block — its model.type is VLM.

What would unblock the use case

Either of:

1. Documentation clarifying that the embedding block is VLM-only, with a recommended pattern for non-VLM "supply your own embedding source" workflows (e.g., a sketch of the direct-ORT inference loop).
2. Generator routing fix that consults the embedding block irrespective of model.type — i.e., if model.embedding.filename is present and the decoder declares inputs_embeds rather than input_ids, route AppendTokens through the embedding sub-graph first.

I'd be happy to put up a PR for option 2 if a maintainer can confirm direction. Same patch shape as #2116 (the Seed-OSS PR I opened earlier today): small surgical change, high leverage, no kernel touches.

Why I'm not just patching it locally

I can — but if this is a known design choice and the recommended path for non-VLM custom-embedding workflows is "use ORT directly, not ORT-GenAI Generator," that's worth knowing before I take on writing a custom inference loop on the C# side. Saving a week of integration work either way.

Thanks for the time. Excited about the project — Seed-OSS PR #2116 is a related contribution from the same workstream.

[Qiming Bao (14H034160212)]

Short answer: with the Generator API, the embedding block only does anything for the multimodal model types. A text-only type like llama never runs it, so exclude_embeds=true on a plain decoder gives you a graph the Generator can't feed. It's a design limitation, not a config mistake. Recent versions now raise an explicit error for it.

How routing works (src/models/model.cpp, CreateModel): model.type decides the model class. Types in ModelType::IsLLM (llama, qwen2, phi3, ...) become DecoderOnly_Model, which feeds input_ids straight to the decoder (DefaultInputIDs input_ids_ in decoder_only.h). The input_ids -> embedding.onnx -> inputs_embeds chain is only wired up in MultiModalLanguageModel (the IsVLM/IsMMM types). That is why 0.13.2 reported Model input was not found: input_ids.

On current main / 0.19.0, DecoderOnly_Model does read model.embedding, but only for the new CPU-embedding path, which is Engine-only (see docs/cpu-embedding-engine.md: "The Generator API, static batching, and multimodal embedding pipelines are separate workflows"). I reproduced this on 0.19.0 (CPU, SmolLM2-135M, also model.type: llama, built with --extra_options exclude_embeds=true):

Config	og.Generator(...) / append_tokens
no embedding block	RuntimeError: Model input was not found: input_ids
+ hand-built embedding.onnx (Gather) and your embedding block	RuntimeError: Decoder-only CPU embedding requires the dynamic-batching Engine.
same, model.type set to qwen2_5_vl_text / gemma3_vl_text	same error

The Engine path won't help you either. It exists to keep the embedding table on the CPU under CUDA graphs, it still looks up a fixed table, and the documented export for it uses use_paged_attention=true exclude_embeds=true. The 0.19.0 builder from PyPI rejects that combination (ValueError: use_paged_attention cannot be combined with exclude_embeds), so for now it's a main-branch feature.

What works today for per-token, runtime-modified embeddings: keep the model builder's exclude_embeds=true export and drive model.onnx with plain ONNX Runtime. The graph takes inputs_embeds, attention_mask and past_key_values.N.{key,value}, and returns logits + present.N.*, so the loop is short. This Python version matches the GenAI Generator's greedy output token for token on the same model (' Paris. Paris is the largest city in', 8/8 tokens identical). It maps 1:1 to Microsoft.ML.OnnxRuntime InferenceSession in C#.

import numpy as np, onnxruntime as ort
sess = ort.InferenceSession("model/model.onnx", providers=["CPUExecutionProvider"])
kv = [i.name for i in sess.get_inputs() if i.name.startswith("past_key_values")]
out_names = [o.name for o in sess.get_outputs()]
# num_key_value_heads / head_size come from genai_config.json
past = {n: np.zeros((1, num_kv_heads, 0, head_size), np.float32) for n in kv}

step_ids, generated = prompt_ids, []
for _ in range(max_new_tokens):
    embeds = E[np.array(step_ids)][None]        # E = your embedding table; apply drift/persona edits here
    total_len = len(prompt_ids) + len(generated)
    res = sess.run(None, {"inputs_embeds": embeds,
                          "attention_mask": np.ones((1, total_len), np.int64), **past})
    nxt = int(res[0][0, -1].argmax())           # or your own sampler
    generated.append(nxt); step_ids = [nxt]
    past = {n.replace("present", "past_key_values"): r
            for n, r in zip(out_names, res) if n.startswith("present")}

You can still use og.Tokenizer for encode/decode. What you give up is GenAI's search/sampling, KV-cache management (including the static/shared buffer options) and guidance.

If you'd rather have this inside the Generator, the change you describe in option 2 is the right shape: let DecoderOnly_State run an embedding session when model.embedding.filename is set and the decoder declares inputs_embeds. Since main currently throws on purpose in DecoderOnly_Model::CreateState, it's worth getting a maintainer's go-ahead first.

I used Claude (Anthropic) to help investigate and draft this; I reviewed it and ran the checks above myself.