[Performance] Why does genai run 2x as fast as vanilla managed onnxruntime?

[elephantpanda]

Describe the issue

I am running phi3-mini-int4 using the usual onnxruntime c# API and it is 2x as slow as when I use the genai code. I am using DirectML c# managed API and am testing it with sequence_length=1 each iteration and using bound inputs and outputs. Basically I am just calling this in a loop, and not changing the input each time for testing but it is still not as fast as genai:
session.RunWithBinding(runOptions, binding);

So in that sense I can say well done for making genai so fast. 🙂

On the other hand, I wonder if you can share the settings or source code for things like sessionOptions and so on. GenAI is good but I really need to use the full capability of onnxruntime API. Since I believe GenAI is built on top of onnxruntime, it would be nice to be able to see the source code for this so I can make my app using onnxruntime API as fast as the GenAI code.

I am using the managed onnxruntime library from nuget 1.19.1 and it is using the DirectML.dll which was installed with genai.

Thanks for any help you can give.

To reproduce

running a phi-3 model using genai code and then trying to run the same model using onnxruntime c# api

Urgency

No response

Platform

Windows

OS Version

10

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.19.1

ONNX Runtime API

C#

Architecture

X64

Execution Provider

DirectML

Execution Provider Library Version

No response

Model File

No response

Is this a quantized model?

Yes

[tianleiwu]

Source code of genai: https://github.com/microsoft/onnxruntime-genai.

For example, use i/o binding to bind past and present to a fixed buffer. Otherwise, copying kv cache will slow down generation significantly.

[RyanUnderhill]

In case you look at the GenAI code, the GenAI library doesn't use I/O binding but it passes preallocated output OrtValues to the Session::Run() function. This has the same performance benefit, as it avoids copies and allocations. I'm not sure if this is convenient in the C# APIs.

[elephantpanda]

Thanks I will try it. BTW I am using the net standard 2.0 API for onnxruntime. I don't know if it would make a difference using a different version like net 6.0? (I assumed it wouldn't be since it's just calling functions in the dll mostly?)

If I can give you some more information about why I want to use onnxruntime API rather than the genai API, it's because mainly I would like to have more control about manipulating the inputs and outputs. e.g. the input tokens and the output probability vectors. Which unfortunately is not accessibly currently with the genai API (even though it's good to get up and running fast which is appreciated.) In an ideal world it would be nice if these two libraries had more compatibility - such as using the same tensor format. Thanks.

These are my session options so far which I tried to copy from the genai code. Apart from the execution provider the others don't seem to have much effect:

        var options = new SessionOptions();
        options.AppendExecutionProvider_DML();
        options.GraphOptimizationLevel = GraphOptimizationLevel.ORT_ENABLE_ALL;
        options.AddSessionConfigEntry("ep.dml.enable_graph_capture", "1");
        options.AddSessionConfigEntry("ep.dml.disable_memory_arena", "1");
        options.IntraOpNumThreads = 4;
        session = new InferenceSession(modelPath + @"\model.onnx",options);

[elephantpanda]

I changed the bound output from:
logits = OrtValue.CreateTensorValueFromMemory(new Float16[VSIZE * inputLength], new long[] { 1, inputLength,VSIZE });
to
logits = OrtValue.CreateAllocatedTensorValue(allocator, TensorElementType.Float16, new long[] { 1, inputLength, VSIZE });

using

var meminfo = new OrtMemoryInfo("DML", OrtAllocatorType.DeviceAllocator, 0, OrtMemType.Default);
allocator = new OrtAllocator(session, meminfo);

binding.BindOutput("logits", logits);

But now when I call:

binding.SynchronizeBoundOutputs()

it slows down again. So back to the drawing board... ☹️ Also When I try to get the read the output it crashes...

[yuslepukhin]

You do not need IOBinding. With the new OrtValue based API you can achieve the same performance and avoid much of the garbage colleciton.

https://onnxruntime.ai/docs/tutorials/csharp/basic_csharp.html

[elephantpanda]

You do not need IOBinding. With the new OrtValue based API you can achieve the same performance and avoid much of the garbage colleciton.

https://onnxruntime.ai/docs/tutorials/csharp/basic_csharp.html

OK thanks. Well that makes things a easier. 😊

I'm still not sure why my onnxruntime code is slower than the genai code. I'll see if I can share my project. Or if there is already a pure c# onnxruntime API project that someone has made for an LLM it would be nice to look at it. I think it's actually the model itself that is running faster using the genai code. There's probably some trick I missed somewhere. 🤔 Or perhaps it's just the managed dot net runtime that is missing some trick (like does it support int4?) Or perhaps there's some setting I'm missing when passing back in the cached key values. I'll keep trying at it.

[RyanUnderhill]

If I can give you some more information about why I want to use onnxruntime API rather than the genai API, it's because mainly I would like to have more control about manipulating the inputs and outputs. e.g. the input tokens and the output probability vectors. Which unfortunately is not accessibly currently with the genai API (even though it's good to get up and running fast which is appreciated.) In an ideal world it would be nice if these two libraries had more compatibility - such as using the same tensor format. Thanks.

We (the GenAI team) have been trying to figure out what types of custom scoring people will be doing to keep the API simple, can you share more about what custom scoring you're doing? We have some proposed APIs to return the logits and let you append tokens during the generation loop, but with all of the different providers (cuda/directml/etc) it's tricky optimizing the data flow to avoid copies.

A simple pseudocode of what you're doing, perhaps with an imaginary GenAI API would be great, so we can see if we can make it possible

[elephantpanda]

If I can give you some more information about why I want to use onnxruntime API rather than the genai API, it's because mainly I would like to have more control about manipulating the inputs and outputs. e.g. the input tokens and the output probability vectors. Which unfortunately is not accessibly currently with the genai API (even though it's good to get up and running fast which is appreciated.) In an ideal world it would be nice if these two libraries had more compatibility - such as using the same tensor format. Thanks.

We (the GenAI team) have been trying to figure out what types of custom scoring people will be doing to keep the API simple, can you share more about what custom scoring you're doing? We have some proposed APIs to return the logits and let you append tokens during the generation loop, but with all of the different providers (cuda/directml/etc) it's tricky optimizing the data flow to avoid copies.

A simple pseudocode of what you're doing, perhaps with an imaginary GenAI API would be great, so we can see if we can make it possible

Hi thanks for your reply. Here is an example. Well one problem I'm having is that sometimes GenAI generates a premature END token. And I want to tell it, to pick a different one. In other words I want to change the probability of certain tokens at various steps, or just to have my own custom function to select the token myself given the probabilities.

Also, just for experimentation purposes to try out different algorithms such as doing my own implementations of beam search or trying out speculative decoding (using a smaller model to predict a few tokens in advance) . It is nice to have hard-coded solutions but I'd also like the flexibility to experiment. For making an app especially in a game it is important to be able to experiment and optimise and find different "tricks".

I would be quite happy if there was a function like GetLogitsAtPosition(n) which would return the 36072 probabilities.

Here is some pseudo code for a chat-like model:

Tokens[] A
LOOP
    inputString = GetInputFromUser()
    A += TokenizeInput(inputString)
    model.SetInput(A)
    LOOP
       logits =GenerateLogits()
       UseCustomFunctionToSelectToken(logits)
    A+=output  //we want to add the output of the LLM also to the next input

So GenAi works great except for a few issues:

1. unexplained crash for certain specific set of input tokens with the error message not really helping.
2. not enough control over token selection
3. not sure the best way to add more tokens to the input for chat-like scenario
4. no way to inspect or change the NamedTensor object returned by ProcessImages to see what is inside for debugging and experimentation. (e.g. this creates an input of size about 2500 and it would be nice to experiment on smaller inputs)
5. Phi-3-vision not yet working for DML but the documentation page doesn't say it should - assuming that is up to date.
6. no way to go back a few steps and generate the tokens again.
7. can crash and error messages not very specific just saying something went wrong with a message sent to the GPU. (Perhaps it should just fail more gracefully and have more inciteful error messages?)

So these are my main roadblocks. For balance here are my points about why I would like to use GenAi over pure onnxruntime code:

1. It appears to be about 2x faster (than any code I can write so far)
2. It is much simpler to use without having to worry about caching, tokenization and other things.

Hope this helps 🙂

[yufenglee]

A big improvement from GenAI that is not mentioned above is that the past and present KV cache share the sample buffer, i.e., there only needs to append kv for new generated tokens to existing one. It avoids copying of past kv cache.

For the issues of genai, we can discuss in details in GenAI repro.
1 and 7 looks like same issues. we can track them with the microsoft/onnxruntime-genai#833.
For 2, could you please add more on it?
For 3 and 6, we are working on it, i.e., adding support of interactive decoding.
For 5, DML does work for phi-3-vision model, what issues did you hit?
For 4, NamedTensor is an opaque object. If you want to debug, you have to debug with C++.

[elephantpanda]

A big improvement from GenAI that is not mentioned above is that the past and present KV cache share the sample buffer, i.e., there only needs to append kv for new generated tokens to existing one. It avoids copying of past kv cache.

Interesting perhaps that is what is giving the big speed up? 🤔Well, who knows.

For the issues of genai, we can discuss in details in GenAI repro. 1 and 7 looks like same issues. we can track them with the microsoft/onnxruntime-genai#833. For 2, could you please add more on it? For 3 and 6, we are working on it, i.e., adding support of interactive decoding. For 5, DML does work for phi-3-vision model, what issues did you hit? For 4, NamedTensor is an opaque object. If you want to debug, you have to debug with C++.

2. As above, I would like to be able to either use the logits for a position in my own custom function to select a token. e.g. sometimes the sampling might give me a token I don't want (for example an END token) and I want to choose another one. Or I just want to try out my own method of choosing a token that's not top-p, top-k or one of the pre-defined options. This also relates to this problem. The ability to be able to experiment with different functions is import I feel.
3. &6 That's good, more flexibility in going to different positions to re-generate tokens or add new tokens would be great.
4. I would expect a NamedTensor object to at least give be the ability to see if names and shapes of the tensors within even if readonly. Otherwise there doesn't seem to be any point in it existing as a separate entity.
5. I have put the error for phi-3-vision here.

Thanks.