How to get token_type_ids and attention_mask and get the result text?

[ljzj2]

I am using android application 10
I have get InferenceSession and tokenizer but I searched that I have to get token_type_ids and attention_mask but there is EncodingToIds
there is no token_type_ids.

what I want
1.ask questions about my txt file.

 var vocabPath = Path.Combine(CacheDir.AbsolutePath, "vocab.txt");
 using (var s = Assets.Open("Models/vocab.txt"));
     using (var fs = File.Create(vocabPath));
         s.CopyTo(fs);

         fs.Dispose();

         tokenizer = BertTokenizer.Create(vocabPath, new BertOptions { RemoveNonSpacingMarks = true });

         var encodings = tokenizer.EncodeToIds("This is a sample");

         var path = Path.Combine(CacheDir.AbsolutePath, "allmini.onnx");
         using (var s1 = Assets.Open("Models/model.onnx"))
         {
             using (var fs1 = File.Create(path))
             {
                 s1.CopyTo(fs1);
             }
         }
         _session = new InferenceSession(path);
var inputs = new[]
{
    NamedOnnxValue.CreateFromTensor("input_ids",idTensor),
};

var results = _session.Run(inputs);

there is the repohttps://github.com/ljzj2/XMedicalAndroid

[Wael El Ghazzawi (identity-wael)]

There are two separate issues here: constructing the model inputs and choosing a model that can produce the answer you want.

In the linked repository's tensor-building loop, idTensor[0, i] = 1 puts 1 in every occupied input-ID position. Those positions need the actual tokenizer IDs. The ones belong in the attention mask instead:

// idTensor and maskTensor are separate tensors with matching [1, length] shapes.
for (int i = 0; i < encodings.Count; i++)
{
    idTensor[0, i] = encodings[i];
    maskTensor[0, i] = 1;
}

Ensure encodings.Count fits the allocated length and the model's limit. Fill unused input positions with the tokenizer's padding ID and leave their mask entries at zero. Keep required special tokens in the encoded sequence. For a single BERT sequence, segment IDs (token_type_ids) are normally all zero; paired sequences use different segments.

Before adding inputs, inspect _session.InputMetadata for the actual required names, element types, and dimensions. Only supply token_type_ids if this ONNX export expects it; use long tensors only if it expects int64. Also inspect _session.OutputMetadata instead of assuming the output is text.

If allmini.onnx is an export of all-MiniLM-L6-v2, it is an embedding model: its model card describes 384-dimensional sentence vectors for tasks such as semantic search. These floats cannot be decoded into an answer using the tokenizer. Some exports include pooling; others return token embeddings that need attention-mask-aware pooling.

For questions about a text file, you can use MiniLM to embed chunks of the file and retrieve chunks similar to the question. A separate generative model can then answer using those chunks, or your app can simply display the retrieved passages. Could you share the model's original download/export link and the input/output metadata? That would establish the exact inputs and whether pooling is already included.

[ljzj2]

thank you.

fill unused input positions with the tokenizer's padding ID and leave their mask entries at zero

"this is a sample" to input_ids only have 4 I have to add other 0 to the input_ids?

yes allmini.onnx is an export of all-MiniLM-L6-V2 you said these floats cannot be decoded into an answer using tokenizer so I can not get an answer?

you can use MiniLM to embed chunks of the file and retrieve chunks similar to the question how to do it?

I am sorry the model's original download link is in other repos and I believe it is download in huggingface. and input/output metadata where can I copy it? I have a vocab.txt.

thank you again .

I wonder if there is any chance give me an example? thank you.

[jinwoo (dev-jinwoohong)]

Hi ljzj2,

No padding needed when you run one sentence at a time. Just make all three inputs the same length as the id list. EncodeToIds already adds [CLS] and [SEP] for you, so use its count as is. token_type_ids is all 0, attention_mask is all 1. You only pad when you batch several sentences together.

For the model itself: I opened the all-MiniLM-L6-v2 onnx file, inputs are input_ids, attention_mask, token_type_ids (all int64, shape [batch, seq]) and the output is last_hidden_state (float, [batch, seq, 384]). You can also see this from _session.InputMetadata / _session.OutputMetadata in C#, or by opening the file in Netron.

MiniLM is an embedding model, it won't write an answer. The usual way to use it on a txt file: split the file into chunks (paragraphs), embed each chunk once, embed the question, and show the chunks with the highest cosine similarity. Something like this:

float[] Embed(string text)
{
    var ids = tokenizer.EncodeToIds(text);   // [CLS] ... [SEP] included
    int n = ids.Count;
    var inputIds = new DenseTensor<long>(new[] { 1, n });
    var mask     = new DenseTensor<long>(new[] { 1, n });
    var typeIds  = new DenseTensor<long>(new[] { 1, n });
    for (int i = 0; i < n; i++) { inputIds[0, i] = ids[i]; mask[0, i] = 1; typeIds[0, i] = 0; }

    using var results = _session.Run(new[] {
        NamedOnnxValue.CreateFromTensor("input_ids", inputIds),
        NamedOnnxValue.CreateFromTensor("attention_mask", mask),
        NamedOnnxValue.CreateFromTensor("token_type_ids", typeIds),
    });
    var hidden = results.First().AsTensor<float>();   // [1, n, 384]

    // mean pooling over tokens, then L2 normalize
    var emb = new float[384];
    for (int i = 0; i < n; i++)
        for (int d = 0; d < 384; d++) emb[d] += hidden[0, i, d];
    float norm = 0;
    for (int d = 0; d < 384; d++) { emb[d] /= n; norm += emb[d] * emb[d]; }
    norm = MathF.Sqrt(norm);
    for (int d = 0; d < 384; d++) emb[d] /= norm;
    return emb;
}

Since the vectors are normalized, cosine similarity is just the dot product of the two arrays. Pick the top 2-3 chunks for the question and display them. If you want a generated answer on top of that you need a second, generative model, MiniLM can't do it.