ultralytics 8.1.25 OpenVINO LATENCY and THROUGHPUT modes

[glenn-jocher]

@adrianboguszewski throughput mode PR here

• add throughput mode arg
• possibly default to throughput mode for large inference sources (i.e. directory of >10 images)

Example code

https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/109-performance-tricks/109-throughput-tricks.ipynb

from openvino.runtime import AsyncInferQueue

def benchmark_model(model: Any, frames: np.ndarray, async_queue: AsyncInferQueue = None, benchmark_name: str = "OpenVINO model", device_name: str = "CPU") -> float:
    """
    Helper function for benchmarking the model. It measures the time and prints results.
    """
    # measure the first inference separately - it may be slower as it contains also initialization
    start = time.perf_counter()
    model(frames[0])
    if async_queue:
        async_queue.wait_all()
    end = time.perf_counter()
    first_infer_time = end - start
    print(f"{benchmark_name} on {device_name}. First inference time: {first_infer_time :.4f} seconds")

    # benchmarking
    start = time.perf_counter()
    for batch in frames:
        model(batch)
    # wait for all threads if async processing
    if async_queue:
        async_queue.wait_all()
    end = time.perf_counter()

    # elapsed time
    infer_time = end - start

    # print second per image and FPS
    mean_infer_time = infer_time / FRAMES_NUMBER
    mean_fps = FRAMES_NUMBER / infer_time
    print(f"{benchmark_name} on {device_name}: {mean_infer_time :.4f} seconds per image ({mean_fps :.2f} FPS)")

    return mean_fps

🛠️ PR Summary

_{Made with ❤️ by Ultralytics Actions}

🌟 Summary

Enhanced model export capabilities and inference support across various formats.

📊 Key Changes

• Updated OpenVINO export to utilize new runtime.save_model method for better compatibility and efficiency.
• Refined the model loading process to make it clearer and added support for more model formats, including CoreML, TensorFlow GraphDef, NCNN, and more.
• Improved the handling of different inference modes in OpenVINO for optimized performance based on use case (latency vs throughput).
• Added detailed comments and organized code for better readability and maintenance.

🎯 Purpose & Impact

• Enhanced Export and Inference Support: Users can now export models to OpenVINO with the latest features and enjoy improved inference support for a wider range of formats, including ONNX, TorchScript, TensorRT, and more, catering to a variety of application needs across platforms.
• Optimized Performance: The introduction of different inference modes for OpenVINO allows users to optimize their models for latency or throughput based on their specific requirements, potentially improving the efficiency and performance of their applications.
• Better User Experience: Clearer code and additional comments make it easier for developers to understand and maintain the code, while also making it simpler to integrate with different backends and platforms.

[sentry-io]

🔍 Existing Issues For Review

Your pull request is modifying functions with the following pre-existing issues:

📄 File: ultralytics/nn/autobackend.py

Function	Unhandled Issue
forward	AssertionError: input size torch.Size([1, 3, 640, 640]) not equal to max model size (1, 3, 1280, 1280) ... Event Count: 2
forward	ValueError: Cannot set tensor: Dimension mismatch. Got 640 but expected 960 for dimension 1 of input 0. ... Event Count: 1
forward	AssertionError: input size torch.Size([1, 3, 640, 640]) not equal to max model size (1, 3, 640, 480) ... Event Count: 1

---

_{Did you find this useful? React with a 👍 or 👎}

[codecov]

Codecov Report

Attention: Patch coverage is 75.55556% with 11 lines in your changes are missing coverage. Please review.

Project coverage is 75.85%. Comparing base (609a0ce) to head (9c3f16c).
Report is 1 commits behind head on main.

❗ Current head 9c3f16c differs from pull request most recent head f1b4685. Consider uploading reports for the commit f1b4685 to get more accurate results

Files	Patch %	Lines
ultralytics/nn/autobackend.py	73.80%	11 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8058      +/-   ##
==========================================
- Coverage   75.90%   75.85%   -0.06%     
==========================================
  Files         117      117              
  Lines       14809    14822      +13     
==========================================
+ Hits        11241    11243       +2     
- Misses       3568     3579      +11     

Flag	Coverage Δ
Benchmarks	36.78% <75.55%> (-0.02%)	⬇️
GPU	38.86% <6.66%> (-0.04%)	⬇️
Tests	70.88% <57.77%> (-0.05%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[adrianboguszewski]

@glenn-jocher I'm listing below what we need to implement:

• a flag for throughput/batch mode for the predictor (similar to int8 argument for exporter)
• add config={"PERFORMANCE_HINT": "THROUGHPUT"} to the compile_model function (

  ultralytics/ultralytics/nn/autobackend.py

  Line 196 in 318a343

  ov_compiled_model = core.compile_model(ov_model, device_name="AUTO") # AUTO selects best available device

  )
• detect if many frames are available at once - then set the throughput/batch flag
• implement async preprocessing (requires thinking about how to do it) - this will give a huge benefit in the throughput/batch mode

def callback(infer_request, info):
        result = infer_request.get_output_tensor(0).data[0]
        show_result(result)
        pass

infer_queue = ov.AsyncInferQueue(ov_model)
infer_queue.set_callback(callback)  # set callback to post-process (show) results

infer_queue.start_async(video_frames[0])
infer_queue.wait_all()

[glenn-jocher]

@adrianboguszewski ok I've added test code for throughput mode and also the config argument you mentioned above. I'm going to debug and test this some more.

ultralytics/ultralytics/nn/autobackend.py

Lines 196 to 200 in 5477165

	ov_compiled_model = core.compile_model(
	ov_model,
	device_name="AUTO", # AUTO selects best available device
	config={"PERFORMANCE_HINT": "THROUGHPUT"},
	)

[glenn-jocher]

@adrianboguszewski ok I've got good and bad news here:

• The good news is we've implemented OpenVINO inference using either latency or throughput inference_mode now (with setting hard-coded locally for now for debugging). This is working correctly as intended I believe, at least in my limited testing on Detect models. Other tasks like Segment that require multiple outputs may require additional testing.

• The bad news is that throughput mode is not able to operate at its potential because the Predictor is passing images at batch-size 1, which is a known Predictor limitation currently for most dataloaders. Only parallel streaming using the StreamLoader supports batch-size>1, i.e. if 4 YouTube videos are passed for inference then the StreamLoader will present the data at batch-size 4 with 1 frame per streaming source.

The other dataloaders, like the ones that collect images from directories, need further structural modifications to accept higher batch sizes. This is not a trivial change and something we need to to implement throughout the Predictor<>Dataloader pipeline and debug and test with all model formats and inference source types, but I'll notify the team and discuss next steps on this as I believe it's a valuable improvement over the current fixed-batch-size 1 system we have in place.

@Laughing-q @AyushExel @fcakyon @Burhan-Q @lakshanthad @RizwanMunawar

[adrianboguszewski]

We can have 2 approaches here:

1. That one you already implemented, which require providing bigger batches
2. Implement async processing in predictor, which may be even more complicated but worth considering

[adrianboguszewski]

I had one important question though. Is the mode of the compiled model modifiable once it's compiled, or is this a read-only value? i.e. is it possible to do something like ov_compiled_model.config["PERFORMANCE_HINT"]="THROUGHPUT" after I've already compiled the model?

mode = "LATENCY"  # either 'LATENCY', 'THROUGHPUT' (not recommended), or 'CUMULATIVE_THROUGHPUT'
ov_compiled_model = core.compile_model(
        ov_model,
        device_name="AUTO",  # AUTO selects best available device, do not modify
        config={"PERFORMANCE_HINT": mode},
)

Nope. The model must be recompiled to change the mode.

[adrianboguszewski]

Btw. please also look at the comment related to the result order. I cannot unresolve the conversation so it's not visible.

[glenn-jocher]

@adrianboguszewski ok THROUGHPUT mode sort order should now be good to go in f19995e

I updated the logic to insert the results directly into the array at the correct index

[glenn-jocher]

@adrianboguszewski PR appears complete. Tests passing. I'll merge this now and then this should form part of ultralytics 8.1.25 which will publish tomorrow.

[glenn-jocher]

@Laughing-q this is the PR we need to build off of by adding batched dataloaders for the Predictor class.

[glenn-jocher]

@adrianboguszewski PR merged!! Will work on batched Predictor inference now with @Laughing-q in a new PR.

[adrianboguszewski]

Thanks. It looks good. I'll test it.

[Burhan-Q]

@glenn-jocher want me to open an issue for batched prediction and add it to the YOLO Project? I also recently tested batching via list of numpy array images, as suggested by @Laughing-q and found that the model will run batch inference (or what appears to be batched inference).

import cv2 as cv
from ultralytics import YOLO

model = YOLO("yolov8n.pt")
im1 = "ultralytics/assets/bus.jpg"
im2 = "ultralytics/assets/zidane.jpg"

img1 = cv.imread(im1)
img2 = cv.imread(im2)

# Multi-image
results = model.predict(source=[img1, img2, img1, img1, img2, img1, img2, img2],)

>>> 0: 640x640 4 persons, 1 bus, 
1: 640x640 2 persons, 1 tie, 
2: 640x640 4 persons, 1 bus, 
3: 640x640 4 persons, 1 bus, 
4: 640x640 2 persons, 1 tie, 
5: 640x640 4 persons, 1 bus, 
6: 640x640 2 persons, 1 tie, 
7: 640x640 2 persons, 1 tie, 
155.0ms

[r.speed for r in results]

>>> [
{'preprocess': 2.5001466274261475, 'inference': 19.374817609786987, 'postprocess': 10.336160659790039}, 
{'preprocess': 2.5001466274261475, 'inference': 19.374817609786987, 'postprocess': 10.336160659790039}, 
{'preprocess': 2.5001466274261475, 'inference': 19.374817609786987, 'postprocess': 10.336160659790039}, 
{'preprocess': 2.5001466274261475, 'inference': 19.374817609786987, 'postprocess': 10.336160659790039}, 
{'preprocess': 2.5001466274261475, 'inference': 19.374817609786987, 'postprocess': 10.336160659790039}, 
{'preprocess': 2.5001466274261475, 'inference': 19.374817609786987, 'postprocess': 10.336160659790039}, 
{'preprocess': 2.5001466274261475, 'inference': 19.374817609786987, 'postprocess': 10.336160659790039}, 
{'preprocess': 2.5001466274261475, 'inference': 19.374817609786987, 'postprocess': 10.336160659790039}
]

# Single image
results2 = model.predict(source=[img1,])

>>> 0: 640x480 4 persons, 1 bus, 1 stop sign, 192.1ms

results2[0].speed
>>> {'preprocess': 2.998828887939453, 'inference': 192.11935997009277, 'postprocess': 8.776426315307617}

[glenn-jocher]

@Burhan-Q yes some predictions sources will run in batches, but I think a main one that's missing is the glob or directory inference, though txt list of sources may also be missing. Yes please open an issue and tag us in it along with @adrianboguszewski from Intel. Thanks!

[Kayzwer]

__version__ in ultralytics/__init__.py is still "8.1.24"

[glenn-jocher]

@Kayzwer thanks for pointing this out! 🌟 We'll update the __version__ in ultralytics/__init__.py to reflect the latest release. Here's a quick fix you can use in the meantime:

import ultralytics
ultralytics.__version__ = "8.1.25"

Stay tuned for the update in the repo!