Ultralytics
Replies: 7 comments 11 replies
-
|
Hello! Thanks for reaching out with your issue. 🤔 From what you've described, it seems the significant increase in inference time for image streams might be related to network latency or processing overhead for fetching the images from URLs. When you're doing inference directly on an image file, the operation is pretty straightforward and swift since the image data is readily available. However, for image streams coming from URLs, the additional time could be from the download time required to fetch each image over the network before it can be processed. One possible solution to mitigate this could be pre-fetching the images asynchronously before running them through the model, or caching the images locally if you're running multiple inferences on the same set of URLs. For example, you could use something like this to asynchronously download images: import asyncio
import aiohttp
async def download_image(url):
async with aiohttp.ClientSession() as session:
async with session.get(url) as response:
content = await response.read()
# Save content to an image file or directly feed it to your model
# Assuming you have a list of URLs
urls = ["https://example.com/image1.jpg", "https://example.com/image2.jpg"]
loop = asyncio.get_event_loop()
tasks = [download_image(url) for url in urls]
loop.run_until_complete(asyncio.wait(tasks))Ensure your model is ready to handle the data after it's downloaded, and this might help reduce the inference time. I hope this helps, and if you have further questions or if this doesn't resolve your issue, feel free to ask! 😊 |
Beta Was this translation helpful? Give feedback.
-
|
Hi, My mistake for not making it clear in the question but I'm getting these URL picture links from a live VMS software program using |
Beta Was this translation helpful? Give feedback.
-
|
@jadrian21 hi there, Thanks for providing some further clarification! The extended inference time you're experiencing, especially when dealing with live-captured images from your VMS software, may indeed seem perplexing given the normal speed on single images. Given that the images are fetched in real-time and the server hosting both the images and the inference process is local, network latency likely isn't the culprit. A couple areas to consider:
Here's a simplified code snippet incorporating asynchronous fetching, assuming you're handling images in Python with import asyncio
import aiohttp
from ultralytics import YOLO
async def fetch_image(session, url):
async with session.get(url) as response:
return await response.read()
async def run_inference_on_image(url):
async with aiohttp.ClientSession() as session:
image_data = await fetch_image(session, url)
model = YOLO('yolov8n-pose.pt')
results = model(image_data)
# handle results
# Incorporate the above in your event loop handling incoming URLs for images.This example doesn't block while fetching images and should help maintain a consistent inference speed. If these suggestions don't lead to improvements, consider monitoring the resource utilization (CPU, GPU, memory) during inference to pinpoint potential bottlenecks. Hope this helps! Let us know how it goes. 🚀 |
Beta Was this translation helpful? Give feedback.
-
|
I'm getting unsupported image type using the code you suggested here is the error: Traceback (most recent call last): |
Beta Was this translation helpful? Give feedback.
-
|
Hey there! It looks like the error you're encountering is due to supplying an unsupported image type for inference. Could you please double-check the format of the image you're using? The YOLO model supports a variety of image formats, including JPG, PNG, and BMP. If your input is a URL pointing to an image, ensure it directly leads to an image file that ends with a supported extension, like import requests
from PIL import Image
from io import BytesIO
def load_image_from_url(url):
response = requests.get(url)
image = Image.open(BytesIO(response.content))
return image
image_data = load_image_from_url('your_image_url_here.jpg')
results = model(image_data)Hope that helps! If the problem persists or if there's anything else we can assist you with, feel free to reach out. 🚀 |
Beta Was this translation helpful? Give feedback.
-
|
For completeness, I solved it by doing the following using the PIL module: Incorporate the above in your event loop handling incoming URLs for images. |
Beta Was this translation helpful? Give feedback.
-
|
That's a neat solution, thanks for sharing! 😊 Your approach with Just a small note, ensure to import from io import BytesIOThis minor addition complements your solution, making it a comprehensive approach to handling image data fetched asynchronously. Keep up the great work exploring efficient solutions! 🚀 |
Beta Was this translation helpful? Give feedback.
-
|
Hi I’m still getting slow speeds processing the data from a stream even though Then I use Multiprocessing to fetch the inferences: And are get the following timings for each inference: 0: 384x640 1 person, 264.5ms 0: 384x640 1 person, 3 chairs, 1 bed, 1 tv, 1 laptop, 1 keyboard, 1 teddy bear, 294.8ms 0: 384x640 1 person, 1 cup, 3 chairs, 1 potted plant, 1 bed, 1 tv, 1 keyboard, 1 teddy bear, 325.7ms 0: 384x640 1 person, 234.3ms 0: 384x640 1 person, 2 chairs, 1 bed, 1 tv, 1 laptop, 1 keyboard, 1 teddy bear, 266.5ms 0: 384x640 1 person, 1 cup, 2 chairs, 1 potted plant, 1 bed, 1 tv, 1 laptop, 1 keyboard, 1 teddy bear, 243.7ms ---done inference--- --- After 0.8858550707499186 minutes... As you can see the total time from these results is 15.6+264.5+...+1385.6 = 10.25 seconds [1] fetching from a socket an XML data message, sending to another computer via sockets an alert message |
Beta Was this translation helpful? Give feedback.
-
|
Hi @jadrian21, Thank you for your detailed follow-up. It looks like you're encountering an issue when trying to load a TensorRT model with the Here's how you can modify your code to run inference on a TensorRT model: from ultralytics import YOLO
class InferenceManager:
def __init__(self):
self.model_pose = YOLO('yolov8x-pose.engine')
self.model_objt = YOLO('yolov8x.engine')
self.model_segm = YOLO('yolov8x-seg.engine')
def single_prediction(self, model, model_name, img, result_dict):
result_dict[model_name] = model.predict(img, device=0)[0].cpu()
def start_prediction_process(self, image1, image2, result_dict):
processes = [
Process(target=self.single_prediction, args=(self.model_pose, 'pose1', image1, result_dict)),
Process(target=self.single_prediction, args=(self.model_objt, 'objt1', image1, result_dict)),
Process(target=self.single_prediction, args=(self.model_segm, 'segm1', image1, result_dict)),
Process(target=self.single_prediction, args=(self.model_pose, 'pose2', image2, result_dict)),
Process(target=self.single_prediction, args=(self.model_objt, 'objt2', image2, result_dict)),
Process(target=self.single_prediction, args=(self.model_segm, 'segm2', image2, result_dict))
]
for p in processes:
p.start()
for p in processes:
p.join()
if __name__ == "__main__":
manager = Manager()
result_dict = manager.dict()
inference_manager = InferenceManager()
image1 = torch.randn(1, 3, 384, 640) # Example tensor
image2 = torch.randn(1, 3, 384, 640) # Example tensor
inference_manager.start_prediction_process(image1, image2, result_dict)
print(result_dict)This should resolve the issue and allow you to run inference on your TensorRT models using the GPU. If you continue to experience issues, please ensure you are using the latest versions of pip install --upgrade torch ultralyticsIf the problem persists, please provide a minimum reproducible example so we can investigate further. You can find more details on creating one here. Thank you for your patience, and please let us know if you have any other questions! 😊 |
Beta Was this translation helpful? Give feedback.
-
|
Hi, once again thanks for the reply, I modified my code as follows: This code runs fine as a simple main process as follows: But the multiprocess version gives key errors ??? and I can't work out why... Here is the verbose output: |
Beta Was this translation helpful? Give feedback.
-
|
Hi @jadrian21, Thank you for providing the detailed code and error logs. It looks like you're encountering a Steps to Resolve the Issue:
Updated Code Example:Here's an updated version of your code that ensures class names are properly loaded and passed to each process: from multiprocessing import Process, Manager
from ultralytics import YOLO
import torch
class MPInference:
def __init__(self):
self.model_pose = YOLO('yolov8x-pose.engine', task="detect")
self.model_objt = YOLO('yolov8x.engine', task="detect")
self.model_segm = YOLO('yolov8x-seg.engine', task="detect")
self.class_names = self.model_pose.names # Assuming all models share the same class names
def single_prediction(self, model, model_name, img, result_dict, class_names):
result = model.predict(img, verbose=True, conf=0.1, max_det=20, amp=True, imgsz=640, device=0)[0].cpu()
result.names = class_names # Ensure class names are set
result_dict[model_name] = result
def start_multi_prediction_process(self, image1, image2, result_dict):
processes = [
Process(target=self.single_prediction, args=(self.model_pose, 'pose1', image1, result_dict, self.class_names)),
Process(target=self.single_prediction, args=(self.model_objt, 'objt1', image1, result_dict, self.class_names)),
Process(target=self.single_prediction, args=(self.model_segm, 'segm1', image1, result_dict, self.class_names)),
Process(target=self.single_prediction, args=(self.model_pose, 'pose2', image2, result_dict, self.class_names)),
Process(target=self.single_prediction, args=(self.model_objt, 'objt2', image2, result_dict, self.class_names)),
Process(target=self.single_prediction, args=(self.model_segm, 'segm2', image2, result_dict, self.class_names))
]
for p in processes:
p.start()
for p in processes:
p.join()
if __name__ == "__main__":
manager = Manager()
result_dict = manager.dict()
inference_manager = MPInference()
image1 = torch.randn(1, 3, 384, 640) # Example tensor
image2 = torch.randn(1, 3, 384, 640) # Example tensor
inference_manager.start_multi_prediction_process(image1, image2, result_dict)
print(result_dict)Additional Tips:
If the issue persists, please provide a minimum reproducible example so we can investigate further. You can find more details on creating one here. Thank you for your patience, and please let us know if you have any other questions! 😊 |
Beta Was this translation helpful? Give feedback.
-
|
Modified as per above now get error: |
Beta Was this translation helpful? Give feedback.
-
|
Hi @jadrian21, Thank you for your detailed follow-up. It looks like you're encountering an issue with pickling TensorRT objects when using multiprocessing. TensorRT objects, such as Solution:To resolve this, you can initialize the models within each process instead of passing them as arguments. This ensures that each process has its own instance of the model, avoiding the pickling issue. Updated Code Example:Here's an updated version of your code that initializes the models within each process: from multiprocessing import Process, Manager
from ultralytics import YOLO
import torch
class MPInference:
def __init__(self):
self.class_names = YOLO('yolov8x-pose.engine', task="detect").names # Load class names once
def single_prediction(self, model_path, model_name, img, result_dict, class_names):
model = YOLO(model_path, task="detect") # Initialize model within the process
result = model.predict(img, verbose=True, conf=0.1, max_det=20, amp=True, imgsz=640, device=0)[0].cpu()
result.names = class_names # Ensure class names are set
result_dict[model_name] = result
def start_multi_prediction_process(self, image1, image2, result_dict):
processes = [
Process(target=self.single_prediction, args=('yolov8x-pose.engine', 'pose1', image1, result_dict, self.class_names)),
Process(target=self.single_prediction, args=('yolov8x.engine', 'objt1', image1, result_dict, self.class_names)),
Process(target=self.single_prediction, args=('yolov8x-seg.engine', 'segm1', image1, result_dict, self.class_names)),
Process(target=self.single_prediction, args=('yolov8x-pose.engine', 'pose2', image2, result_dict, self.class_names)),
Process(target=self.single_prediction, args=('yolov8x.engine', 'objt2', image2, result_dict, self.class_names)),
Process(target=self.single_prediction, args=('yolov8x-seg.engine', 'segm2', image2, result_dict, self.class_names))
]
for p in processes:
p.start()
for p in processes:
p.join()
if __name__ == "__main__":
manager = Manager()
result_dict = manager.dict()
inference_manager = MPInference()
image1 = torch.randn(1, 3, 384, 640) # Example tensor
image2 = torch.randn(1, 3, 384, 640) # Example tensor
inference_manager.start_multi_prediction_process(image1, image2, result_dict)
print(result_dict)Additional Tips:
If the issue persists, please provide a minimum reproducible example so we can investigate further. You can find more details on creating one here. Thank you for your patience, and please let us know if you have any other questions! 😊 |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for your reply and help, I slightly modified the code as it was still crashing with keyerrors but it still gives key errors... The errors are still as follows... |
Beta Was this translation helpful? Give feedback.
-
|
Hi @jadrian21, Thank you for the update. It looks like the To help us investigate further, could you please provide a minimum reproducible example? This will allow us to replicate the issue on our end. You can find guidance on creating a minimum reproducible example here. This step is crucial for us to diagnose and resolve the problem effectively. Additionally, please ensure that you are using the latest versions of pip install --upgrade torch ultralyticsIn the meantime, here's a refined approach to ensure all keys are set in the def single_prediction(self, model_path, model_name, img, result_dict, class_names):
model = YOLO(model_path, task="detect")
result = model.predict(img, verbose=True, conf=0.1, max_det=20, amp=True, imgsz=640, device=0)[0].cpu()
result.names = class_names
result_dict[model_name] = result
def startMultiPredictionProcess(self, image1, image2, result_dict):
keys = ['pose1', 'objt1', 'segm1', 'pose2', 'objt2', 'segm2']
for key in keys:
result_dict[key] = None # Initialize keys to avoid KeyError
processes = [
Process(target=self.single_prediction, args=('yolov8x-pose.engine', 'pose1', image1, result_dict, self.class_pose_names)),
Process(target=self.single_prediction, args=('yolov8x.engine', 'objt1', image1, result_dict, self.class_objt_names)),
Process(target=self.single_prediction, args=('yolov8x-seg.engine', 'segm1', image1, result_dict, self.class_segm_names)),
Process(target=self.single_prediction, args=('yolov8x-pose.engine', 'pose2', image2, result_dict, self.class_pose_names)),
Process(target=self.single_prediction, args=('yolov8x.engine', 'objt2', image2, result_dict, self.class_objt_names)),
Process(target=self.single_prediction, args=('yolov8x-seg.engine', 'segm2', image2, result_dict, self.class_segm_names))
]
for process in processes:
process.start()
for process in processes:
process.join()This ensures that all expected keys are initialized in the Looking forward to your reproducible example so we can assist you further! 😊 |
Beta Was this translation helpful? Give feedback.
-
|
The problem I have with submitting an MRE is that the error only occurs using the TensorRT engine files exported for the NVIDIA A2 |
Beta Was this translation helpful? Give feedback.
-
|
Hi @jadrian21, Thank you for the detailed information. It seems like the issue is specific to the TensorRT engine files exported for your NVIDIA A2 system. To help us investigate further, could you please confirm the following:
Here's a small snippet to illustrate how you might structure your MRE: import torch
from ultralytics import YOLO
# Load the TensorRT model
model = YOLO('path/to/your/tensorrt.engine', task="detect")
# Example image
image = torch.rand(1, 3, 640, 640)
# Run inference
results = model.predict(image, verbose=True)
print(results)By providing this information, we can better understand the context and work towards a solution. Thank you for your cooperation! 😊 |
Beta Was this translation helpful? Give feedback.
-
|
Hi, how do I submit my MRE? can I just reply here with it (no proprietary info in it) |
Beta Was this translation helpful? Give feedback.
-
|
Hi @jadrian21, Thank you for your question! Yes, you can definitely reply here with your Minimum Reproducible Example (MRE). Providing an MRE is crucial as it helps us understand and reproduce the issue you're facing, which is the first step towards finding a solution. Please make sure your MRE includes:
For more detailed guidance on creating an MRE, you can refer to our documentation here. Additionally, please ensure you are using the latest versions of pip install --upgrade torch ultralyticsLooking forward to your MRE so we can assist you further! 😊 |
Beta Was this translation helpful? Give feedback.
-
|
Before I give an MRE with multiprocessing I thought I'd export an optimal .engine file for the A2 card in the server. Beforehand I did not use from ultralytics import YOLO
model = YOLO("yolov8x-seg.pt")
model.export(format="engine", dynamic=True, batch=8, workspace=4, int8=True, data="coco.yaml")Upon Inference of an image gave the following error: (the other two models worked) AttributeError Traceback (most recent call last)
Cell In[1], line 36
20 rp = modelp.predict(img,
21 # settings
22 verbose=True,
(...)
25 amp=True,
26 imgsz=640)[0].cpu()
28 ro = modelo.predict(img,
29 # settings
30 verbose=True,
(...)
33 amp=True,
34 imgsz=640)[0].cpu()
---> 36 rs = models.predict(img,
37 # settings
38 verbose=True,
39 conf=0.1,
40 max_det=20,
41 amp=True,
42 imgsz=640)[0].cpu()
44 chairs = ib.get_chairs_bbox(ro, rs, 0.3)
45 print(len(chairs))
File c:\Users\Western\AppData\Local\Programs\Python\Python312\Lib\site-packages\ultralytics\engine\model.py:435, in Model.predict(self, source, stream, predictor, **kwargs)
433 if not self.predictor:
434 self.predictor = predictor or self._smart_load("predictor")(overrides=args, _callbacks=self.callbacks)
--> 435 self.predictor.setup_model(model=self.model, verbose=is_cli)
436 else: # only update args if predictor is already setup
437 self.predictor.args = get_cfg(self.predictor.args, args)
File c:\Users\Western\AppData\Local\Programs\Python\Python312\Lib\site-packages\ultralytics\engine\predictor.py:303, in BasePredictor.setup_model(self, model, verbose)
301 def setup_model(self, model, verbose=True):
302 """Initialize YOLO model with given parameters and set it to evaluation mode."""
--> 303 self.model = AutoBackend(
304 weights=model or self.args.model,
305 device=select_device(self.args.device, verbose=verbose),
306 dnn=self.args.dnn,
307 data=self.args.data,
308 fp16=self.args.half,
309 batch=self.args.batch,
310 fuse=True,
311 verbose=verbose,
312 )
314 self.device = self.model.device # update device
315 self.args.half = self.model.fp16 # update half
File c:\Users\Western\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\utils\_contextlib.py:115, in context_decorator.<locals>.decorate_context(*args, **kwargs)
112 @functools.wraps(func)
113 def decorate_context(*args, **kwargs):
114 with ctx_factory():
--> 115 return func(*args, **kwargs)
File c:\Users\Western\AppData\Local\Programs\Python\Python312\Lib\site-packages\ultralytics\nn\autobackend.py:265, in AutoBackend.__init__(self, weights, device, dnn, data, fp16, batch, fuse, verbose)
263 if -1 in tuple(model.get_tensor_shape(name)):
264 dynamic = True
--> 265 context.set_input_shape(name, tuple(model.get_tensor_profile_shape(name, 0)[1]))
266 if dtype == np.float16:
267 fp16 = True
AttributeError: 'NoneType' object has no attribute 'set_input_shape'Can you help with what could be the problem? |
Beta Was this translation helpful? Give feedback.
-
|
Hi there, Thank you for your patience and for providing detailed information about the issue you're experiencing. Let's address the problems step-by-step. Inference Time IssueThe significant increase in inference time when processing images from URLs could be due to several factors, including network latency, image preprocessing, or differences in image formats. To diagnose this, you can try the following:
Exporting TensorRT ModelIt seems you're encountering an error when running inference on a TensorRT model exported with INT8 quantization. The error message indicates an issue with setting the input shape for the model. Here are a few steps to troubleshoot this:
Here's an example of exporting a YOLOv8 model to TensorRT without INT8 quantization: from ultralytics import YOLO
# Load the YOLOv8 model
model = YOLO("yolov8x-seg.pt")
# Export the model to TensorRT format without INT8 quantization
model.export(format="engine", dynamic=True, batch=8, workspace=4)Minimum Reproducible ExampleTo better assist you, could you please provide a minimum reproducible example of the code that leads to the error? This will help us diagnose the issue more effectively. You can refer to our guide on creating a minimum reproducible example here: Minimum Reproducible Example. Update PackagesEnsure that you are using the latest versions of the Ultralytics package and TensorRT. Sometimes, issues are resolved in newer releases. pip install --upgrade ultralyticsIf the issue persists after trying these steps, please share the updated code and any additional error messages you encounter. We're here to help you get this resolved as quickly as possible! |
Beta Was this translation helpful? Give feedback.
-
|
Anyway I solved my speed problems by asking back and forth with the ChatGPT GPT-4o model (One of their most advanced but requires This is implemented with this simple call and return result code: Thanks for the initial help, but I'm slightly disappointed this wasn't thought of... |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for sharing your solution and code. It's great to hear you resolved the speed issue using multiprocessing. Leveraging Python's multiprocessing can indeed enhance performance for concurrent tasks. If you have any further questions or need assistance, feel free to reach out. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Hi,
I'm having a problem running inference on a Tesla T4 card. Normally the code runs fast (150ms) see below
when running on one image but when running on an image stream supplied by URLs the inference time blows
out to 6+ seconds (see further down) is the format for the image causing a problem? (see code at bottom of page for URL pics below)
0: 480x640 1 person, 6569.2ms
Speed: 144.4ms preprocess, 6569.2ms inference, 1284.4ms postprocess per image at shape (1, 3, 480, 640)
Run Inference 8.315574169158936
Beta Was this translation helpful? Give feedback.
All reactions