Session options?

[johngebbie]

Hello and thank you for making this.
I was wondering if there is or will be a way to set the number of threads for a session?

This is what I'm porting from python:

opts = onnxruntime.SessionOptions()
opts.inter_op_num_threads = 1
opts.intra_op_num_threads = 1
landmarking = onnxruntime.InferenceSession("model.onnx", opts, providers)

Without the thread options, like currently in my Go version, my CPU goes all the way up to 389% with seemingly no benefit in my case.

Thanks again.

[yalue]

Hi, and thank you for the comment! I currently do not plan to add a feature to control this from Go, and would be happy to take a PR if you want to add one yourself. If not, I may get to it eventually, but do not know when.

However, in the meantime, there's a simple hack that ought to work. If you modify onnxruntime_wrapper.c, you can modify the CreateSession function as follows:

OrtStatus *CreateSession(void *model_data, size_t model_data_length,
  OrtEnv *env, OrtSession **out) {
  OrtStatus *status = NULL;
  OrtSessionOptions *options = NULL;
  status = ort_api->CreateSessionOptions(&options);
  if (status) return status;
  
  // Add these lines! ////////////////////////////////////////////////////
  status = ort_api->SetInterOpNumThreads(options, 1);
  if (status) {
    ort_api->ReleaseSessionOptions(options);
    return status;
  }
  status = ort_api->SetIntraOpNumThreads(options, 1);
  if (status) {
    ort_api->ReleaseSessionOptions(options);
    return status;
  }
  // End of new stuff! //////////////////////////////////////////////////////

  status = ort_api->CreateSessionFromArray(env, model_data, model_data_length,
    options, out);
  // It's OK to release the session options now, right? The docs don't say.
  ort_api->ReleaseSessionOptions(options);
  return status;
}

Note that I haven't tested it, but it should be very close to what you need. It won't be configurable, it will just hardcode it to use a single thread. However, if that's what you need, this quick hack ought to work for you.

I hope this helps!

[johngebbie]

Thank you so much! Your lines worked and then I realized gocv was using multiple threads too. I might try to make a PR if all goes to plan. I'm cheese'n!

[yalue]

Hi again, just commenting on this issue in case someone looks at it in the future, but as of commit 7bf5a6e (PR #14),the library now supports setting InterOpNumThreads and IntraOpNumThreads when creating a session:

// Note: Omitting error checking

// Create a SessionOptions object and configure the # of threads.
options, _ := ort.NewSessionOptions()
defer options.Destroy()
options.SetIntraOpNumThreads(1)
options.SetInterOpNumThreads(1)

// Provide the options pointer when creating a new session
session, _ := ort.NewAdvancedSession("network.onnx",
    []string{"InputTensor"}, []string{"OutputTensor"},
    []ArbitraryTensor{inputTensor}, []ArbitraryTensor{outputTensor},
    options)
defer session.Destroy()

// ...

[johngebbie]

Fab and thanks for letting me know!

I made a little head tracking util with onnxruntime_go and now I shouldn't need to patch it :)