Add Intel GPU Hardware Acceleration for Image Classification using OpenVINO

[Nehereus]

Add Intel GPU Hardware Acceleration for Image Classification using OpenVINO

Description

This pull request introduces hardware acceleration support for Intel GPUs using Intel OpenVINO in the image classification tasks of our open-source image storage solution. This enhancement aims to improve performance by leveraging GPU capabilities for machine learning operations.

Changes

• Integrated Intel OpenVINO to enable GPU support for image classification.
• Dockerfile (Dockerfile_openvino) is added to build the application with OpenVINO support.
• Automatic detection of compatible Intel GPUs. If an Intel GPU is available and compatible, the module will utilize it for hardware acceleration.
• Fallback to CPU mode: In the absence of a compatible GPU, the application will automatically switch back to the original CPU-based implementation.

Usage

To build the Docker image with OpenVINO support, use the following command:

docker build -f Dockerfile_openvino .

Known Limitations

Due to the current implementation constraints of Intel OpenVINO, the maximum concurrency for image classification tasks is restricted to 1 when using GPU acceleration. Exceeding this limit will result in a runtime error:

RuntimeError: Infer Request is busy.

TODO

Add WebGUI configuration option to enable/disable hardware acceleration for machine learning tasks. This will provide users with more control over the use of hardware resources and adaptability to different system configurations.

[mertalev]

Thank you for your work! I think this is a good first step, but it makes many more changes than necessary to add OpenVINO support. Additionally, Optimum may or may not be removed from the ML service in the future. This is due to model exporting issues on some hardware, as well as it being the only reason we have to bundle PyTorch and some other large packages into the image.

I would suggest using the onnxruntime-openvino package and extending from an official OpenVINO image. You can make both onnxruntime-openvino and the normal onnxruntime package optional extras for Poetry. The default image would install onnxruntime, while the OpenVINO Dockerfile would install onnxruntime-openvino.

The only code change needed would be to check if the OpenVINO execution provider is available (using ONNX Runtime's get_available_providers) and add it as the first provider if so.

This would work for all tasks instead of just image classification and produce a smaller image.

Edit: Actually, onnxruntime-openvino bundles the required OpenVINO libraries, so you don't need to extend from an OpenVINO image.

[Nehereus]

Thank you for your helpful suggestion. I have reviewed the documentation for onnxruntime-openvino and found that the current release supports up to Python 3.10 for now. Is there a specific reason for the ML module to use Python 3.11 at this time? If not, could we consider using Python 3.10 for the module for now, with the possibility of updating it later?

[mertalev]

Python 3.11 is preferred since it's about 10~60%* faster than 3.10. But since this is a separate Dockerfile anyway, I think it's fine to use 3.10 for OpenVINO.

*Note: this is for Python code; it doesn't affect C/C++ bindings, so the performance difference for this codebase is on the lower end of this figure.

[mertalev]

Closing in favor of #5619