Skip to content

Conversation

kpedro88
Copy link

@kpedro88 kpedro88 commented Oct 7, 2025

The function at::set_num_interop_threads() in PyTorch can only be called once. Subsequent calls will result in an exception.

(The function at::set_num_threads() can be called multiple times, but subsequent calls after the first will issue a warning message and will not have an effect.)

This can cause the server to crash if two models are loaded that both specify values for the corresponding configuration parameters:

terminate called after throwing an instance of 'c10::Error'
  what():  Error: cannot set number of interop threads after parallel work has started or set_num_interop_threads called
Exception raised from set_num_interop_threads at /opt/pytorch/pytorch/aten/src/ATen/ParallelThreadPoolNative.cpp:54 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x88 (0x7f1f656896c8 in /opt/tritonserver/backends/pytorch/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, char const*) + 0x59 (0x7f1f65625341 in /opt/tritonserver/backends/pytorch/libc10.so)

In this PR, std::call_once() is used to prevent this (along with a try/catch block just in case).

Testing the above case (two models that both specify values) with this PR shows that the exception no longer occurs and inference can proceed.

The documentation is updated accordingly.

@kpedro88
Copy link
Author

kpedro88 commented Oct 7, 2025

Updated to take #158 (review) into account

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

1 participant