Configuring SessionOptions programmatically

[f2bo]

I needed to enable verbose logging to troubleshoot an issue I was having. As far as I know, this is done by configuring the LogSeverityLevel property of the SessionOptions.

I'm currently using Microsoft.Extensions.AI and the OnnxRuntimeGenAIChatClient. The only mechanism to configure the execution environment that appears to be currently available is to create a Config object and use it to construct a Model, which is then used to initialize the client. However, even though the Config class has a SetProviderOption method, it seems to be missing a SetOption method.

For example:

...
using Config config = new(modelPath);

// THE FOLLOWING IS NOT CURRENTLY POSSIBLE
config.SetOption("log_severity_level", OrtLoggingLevel.ORT_LOGGING_LEVEL_VERBOSE);   

using Model model = new(config);
using var client = new OnnxRuntimeGenAIChatClient(model, true, options);
...

I'm aware that you can do this by setting the property in genai_config.json, but there seems to be missing a mechanism to do this programmatically.

"session_options": {
    "log_id": "onnxruntime-genai",
    "log_severity_level": 0,
    "provider_options": [
        {
            "dml": {}
        }
    ]
}

[baijumeswani]

We have an env flag that you can enable to get debug logs set ORTGENAI_ORT_VERBOSE_LOGGING=1. However, this isn't well documented and is mostly for internal debugging.

At the moment, we don't have any mechanism to programmatically control the verbose logging through an API mostly because we see this as an ONNX Runtime specific control to be used only for debugging that can be controlled through the genai_config.json (does not need runtime dependence). But I see that this can be useful. Not sure when we can get to this though.

[RyanUnderhill]

Ah, we do have a way to overlay options onto the genai_config.h at runtime through the config.Overlay(json overlay) method, but it's not currently exposed in C#. The way you'd use it for your case would be like this (C++ example):

config.Overlay(R"({ "model": { "decoder": { "session_options": { "log_severity_level": 0 } } } })");

Let us know if that is what you need and we'll expose it for C#

[f2bo]

@baijumeswani

We have an env flag that you can enable to get debug logs set ORTGENAI_ORT_VERBOSE_LOGGING=1. However, this isn't well documented and is mostly for internal debugging.

I wasn't aware of this environment variable. Not a big deal, but I notice that it doesn't allow configuring the severity level. It's either verbose or nothing but I'll be sure to use it next time I need this.

we see this as an ONNX Runtime specific control to be used only for debugging that can be controlled through the genai_config.json

This is what I ended up doing though it's not very convenient, particularly when you need to choose from different models at runtime. It entails editing the genai_config.json file for each model that could be loaded and once you are done troubleshooting, changing the config files back to restore their original settings. Also, and it's not my case, but you might not have write access to the model's storage location. I'm not aware of a mechanism to load the config file from a different place so you would need to copy everything locally.

@RyanUnderhill

Ah, we do have a way to overlay options onto the genai_config.h at runtime through the config.Overlay(json overlay) method, but it's not currently exposed in C#. The way you'd use it for your case would be like this (C++ example):

config.Overlay(R"({ "model": { "decoder": { "session_options": { "log_severity_level": 0 } } } })");

Let us know if that is what you need and we'll expose it for C#

My immediate need could be resolved with the ORTGENAI_ORT_VERBOSE_LOGGING environment variable mentioned above, but I can see that this might be useful if I ever need to change other options, though I don't know how likely that is.

Having said that, if I had the choice, I would prefer setting the value for individual options rather than building a JSON string. For example:

config.SetOption("log_severity_level", OrtLoggingLevel.ORT_LOGGING_LEVEL_VERBOSE);

Or even better, having strongly typed properties exposed by the Config object.

config.LogSeverityLevel = OrtLoggingLevel.ORT_LOGGING_LEVEL_VERBOSE;

Thanks @baijumeswani and @RyanUnderhill!

[RyanUnderhill]

Yes, for things that we learn are popular we can always add specific APIs for them. But for models with multiple sessions, you can set the options per model (encoder model having different options than the decoder model). So some are hard to add just one single API for.

[f2bo]

Yes, I understand. I did find a need to be able to determine context_length for a model. Currently, you need to load and parse the config file to retrieve it, at least as far as I know.