[Bug] Significant Memory Leak When Using OpenVino EP With ONNX Runtime For CRAFT

[narolski]

System information (version)

• OpenVINO => 2022.1.0
• Operating System / Platform => Debian Buster
• Compiler => gcc
• Problem classification: Model Inference
• Framework: ONNXRuntime (1.11 or 1.12), Nvidia Triton Inference Server (22.06)
• Model name: CRAFT (from https://github.com/clovaai/CRAFT-pytorch converted into ONNX)

Detailed description

When I use the OpenVino Execution Provider (EP) in ONNX Runtime for CPU-based inference, the memory is not de-allocated between inference sessions, causing a memory leak for inputs of various dimensions (model uses dynamic-sized inputs).

After reading #11667, I've tried to build OpenVino checking out the commit that introduced usstq@3857559, but this did not resolve the memory leak.

Steps to reproduce

• Build ONNXRuntime 1.11 or 1.12 with --use_openvino AUTO:CPU,GPU
• Load the CRAFT model using ONNXRuntime with OpenVino EP
• Perform various inference requests (via Nvidia Triton Inference Server)

Issue submission checklist

• I report the issue, it's not a question
• I checked the problem with documentation, FAQ, open issues, Stack Overflow, etc and have not found solution
• There is reproducer code and related data files: images, videos, models, etc.

[mohsinmx]

@narolski Can you please try inferencing with onnxruntime==1.12.0 by enabling "enable_dynamic_shapes" flag to true.