Debugging OpenVINO Python GIL Error

[sspintel]

OpenVINO Source => GitHub
OpenVINO Version => master branch
Operating System / Platform =>Ubuntu 20
Compiler => Cmake 3.27.0 & GCC 9.4.0
Problem classification: Python Debugging
Device use: CPU
Python version: 3.9.5

I'm trying to debug issues when enabling new models through the torch.compile feature. However, it seems like debug mode builds of OpenVINO might be broken due to a GIL error.

My CMake build command:

cmake .. -DCMAKE_BUILD_TYPE=Debug -DENABLE_PYTHON=ON -DENABLE_WHEEL=ON -DENABLE_OV_PYTORCH_FRONTEND=ON -DCMAKE_INSTALL_PREFIX=/home/adl/SSP/openvino_debug_install

Then I'm setting PYTHONPATH to /home/adl/SSP/openvino_debug_install/python/ as suggested in #18786

Code to reproduce:

import os
import torch

import openvino.frontend.pytorch.torchdynamo.backend
import numpy as np
from time import perf_counter

os.environ["PYTORCH_TRACING_MODE"] = "TORCHFX"
os.environ["OPENVINO_DEVICE"] = "CPU"
os.environ["TORCH_LOGS"] = "+dynamo"
os.environ["TORCHDYNAMO_VERBOSE"] = "1"

torch._dynamo.config.suppress_errors = True

torch_models = [
    "resnet50",
]

for model_name in torch_models:
    model = torch.hub.load('pytorch/vision:v0.15.1', model_name)
    with torch.no_grad():
        random_input = torch.tensor(np.random.rand(1, 3, 224, 224).astype(np.float32))
        model.eval()
        model = torch.compile(model, backend="openvino")

        start = perf_counter()
        result = model(random_input)
        end = perf_counter()
        
        avg_inf_time = 0

        sum_inf_time = 0
        print(f'First Inference Time(ms):  {(end - start)*1000} ')
        for i in range(100):
            start = perf_counter()
            result = model(random_input)
            end = perf_counter()
            sum_inf_time = sum_inf_time + (end - start)
    avg_inf_time = sum_inf_time / 100

    print(f"Average Inference Time(ms): {avg_inf_time*1000}")
    print("Finished")

In Release build, this runs fine but in Debug build I get the following exception:

pybind11::handle::inc_ref() is being called while the GIL is either not held or invalid. Please see https://pybind11.readthedocs.io/en/stable/advanced/misc.html#common-sources-of-global-interpreter-lock-errors for debugging advice.The failing pybind11::handle::inc_ref() call was triggered on a NoneType object.

I was able to backtrace this error to this line in python bindings:

openvino/src/bindings/python/src/pyopenvino/core/compiled_model.cpp

Line 31 in 4f13f0c

return std::make_shared<InferRequestWrapper>(self.create_infer_request(), self.inputs(), self.outputs());

I was also able to find an earlier PR that inserted GIL releases in appropriate places earlier. #10810

[ilya-lavrenov]

@sspintel could you please try whether the provided solution helps?

[sspintel]

Hi @ilya-lavrenov, Thank you for the quick response. That did the trick! I don't see the exception anymore.