modes/val/

[giscus[bot]]

modes/val/

Guide for Validating YOLOv8 Models. Learn how to evaluate the performance of your YOLO models using validation settings and metrics with Python and CLI examples.

https://docs.ultralytics.com/modes/val/

[shubhamjanu]

This line of code "metrics.box.map50" gives error saying that "'NoneType' object has no attribute 'box' ".

My code is described below:

Load a YOLOv8 model

model = YOLO('yolov8n.pt')

Train the model

results_train = model.train(data='japan5.yaml', epochs=1, imgsz=600)
model = YOLO('runs/detect/train11/weights/best.pt') # load a custom model

Validate the model

metrics = model.val() # no arguments needed, dataset and settings remembered
map50=metrics.box.map50
print(map50)

Please suggest.

[glenn-jocher]

@shubhamjanu hello! 😊 Let's dive right into your queries:

1. For training with high-resolution images while maintaining the aspect ratio, YOLOv8's training mechanism automatically handles resizing and aspect ratios elegantly. However, dealing with very high-resolution images does require a balance. For your case, using imgsz=640 (or slightly higher if your GPU supports it) is a sensible choice. It scales down images, maintaining their aspect ratio. YOLOv8 interprets imgsz as [height, width]. The worse results with [1920,1024] could be due to several factors, including overfitting or not fine-tuning other hyperparameters accordingly.

2. Yes, resizing with the imgsz parameter handles annotations accordingly. YOLOv8 takes care of scaling the annotations, so you don't have to manually adjust your annotation files.

3. Whether downsampling or upsampling is better can depend on various factors, including the nature of your dataset and what you're trying to detect. Generally, slight upsampling (e.g., 600x600 to 608x608) can be beneficial as it introduces minimal distortion while adhering to the model's architectural constraints. However, extensive experimentation with your specific dataset is the best way to determine the ideal approach.

4. The IoU threshold differences during training (0.7) and validation (0.6) are predefined settings based on empirical performance metrics. If your goal is to calculate mAP at IoU=0.5, setting iou=0.5 during both training and validation is a reasonable approach. This aligns the model's detection criteria closely with your evaluation metric and can provide more relevant performance insights.

Remember, experimenting with these settings in the context of your specific dataset and objectives is key to achieving the best results. Happy experimenting! 🚀

[DoFulangChen]

hi, when i use the pycharm, the code is (not cmd line)
model.train(data=r"datasets\data.yaml", epochs=5, batch=1, device='cpu', task='segment', imgsz=[1920, 1200])

one warning comes out
WARNING ⚠️ updating to 'imgsz=1920'. 'train' and 'val' imgsz must be an integer, while 'predict' and 'export' imgsz may be a [h, w] list or an integer, i.e. 'yolo export imgsz=640,480' or 'yolo export imgsz=640'

how can i train my own dataset ? img_width = 1920, img_height = 1200, many tks

[glenn-jocher]

Hey there! 👋 It looks like you've encountered a warning due to a slight misunderstanding in specifying the image size for training in PyCharm.

For training, YOLOv8 expects the imgsz argument to be a single integer, representing both the height and width since it scales the images keeping their aspect ratio intact. This means when you provide a single integer (like imgsz=640), it maintains the aspect ratio for you automatically. If you're seeing worse results at a specific size, consider different dimensions close to your target but as a single value, or experiment with the rect parameter as you mentioned.

The warning you encountered suggests a syntax more aligned with export or predict tasks, which can accept [height, width] lists. For training your dataset that has images of resolution 1920x1200, start with something manageable for your GPU like imgsz=640 and see if your hardware can handle higher resolutions gradually.

To answer your question, for training, you should adjust the imgsz to a single value that ensures your hardware supports it, without explicit width and height distinction. The annotation adjustments are taken care of by YOLOv8 based on the resizing.

Regarding whether to downsample or upsample, both have benefits and drawbacks, and the choice heavily depends on your specific problem and dataset. Downsampling retains more original content, but might miss fine details, whereas upsampling might introduce artifacts. Given the two options, I'd slightly lean towards downsampling for less distortion, but encourage testing both to see what works better for your case.

The IoU parameter differences during training and validation reflect typical use cases, but for mAP calculation at IoU=0.5, you should indeed adjust the IoU threshold to 0.5 during both training and validation for consistency with your evaluation criteria.

Hope this helps clarify things! Let me know if there's anything else I can do for you. Happy training! 😊

[DoFulangChen]

i see, thanks for your answer, regards

[glenn-jocher]

Hey there! 😊 Glad you found the answer helpful. For training your dataset with varying resolutions while preserving aspect ratios and avoiding out-of-memory issues, you might consider trying adaptive resizing with smaller imgsz values that fit your GPU capabilities, like imgsz=640. YOLOv8 indeed auto-scales annotations, so you're all set on that front!

Regarding image resolutions, the decision between downsampling and upsampling (e.g., 576 vs. 608) generally depends on your specific dataset and task. Experimenting is key, but starting with downsampling might be slightly more efficient to retain original content without artificially introducing pixels.

For calculating mAP at IoU=0.5, setting iou=0.5 for both training and validation phases would align your training closer with your evaluation metric.

Remember, YOLOv8 will take care of aspect ratios and annotations for you during training. Just specify your desired imgsz (as a single integer for training) and experiment to find what works best for your setup. Happy training! 🚀

[devandress]

[pderrenger]

Hey there! 👋 It sounds like you're having trouble locating the predict file and the best.pt model after training. No worries, let's sort this out!

After training with YOLOv8, the best.pt model (which is the best performing model on your validation set during training) should be saved in your runs/train/exp*/weights/ directory. The directory might have a different name if you've set a custom --project or --name during training.

As for the predict functionality, it's not a separate file but a mode you can use with the YOLO CLI or Python API. To make predictions using your trained model, you can use the following command if you're using the CLI:

yolo detect predict model=runs/train/exp*/weights/best.pt source='path/to/your/image/or/video'

Or, if you're using Python:

from ultralytics import YOLO

# Load your trained model
model = YOLO('runs/train/exp*/weights/best.pt')

# Make predictions on an image or video
results = model(['path/to/your/image/or/video'])

# To view results
results.show()

Make sure to replace 'path/to/your/image/or/video' with the actual path to your image or video file. Also, adjust the path to best.pt based on where your trained model is saved.

If you're still having issues or if there's anything else I can help you with, feel free to ask. Happy coding! 😊

[madasuvenky]

When I convert the YOLO v8 model weights to int16 and validate, I'm getting 0 accuracy, but with float32 model weights, I'm getting 0.87 accuracy.

[pderrenger]

Hi there! 👋

Thank you for reaching out. The discrepancy you're observing between the int16 and float32 model weights is quite significant. Here are a few steps and considerations to help diagnose and potentially resolve the issue:

1. Reproducible Example: To better understand and reproduce the issue, could you please provide a minimal reproducible example? This will help us pinpoint the problem more effectively. You can refer to our guide on creating a minimum reproducible example.

2. Latest Versions: Ensure you are using the latest versions of all relevant packages, including Ultralytics YOLOv8. Sometimes, issues are resolved in newer releases.

3. Quantization Impact: Int16 quantization can significantly impact model accuracy, especially if the model was not specifically trained or fine-tuned for lower precision. You might want to try FP16 (half-precision) quantization instead, which often maintains better accuracy compared to int16.

4. Validation Metrics: When validating, make sure to use appropriate metrics and settings. Here’s a quick example of how you can validate your model using Python:

   from ultralytics import YOLO
   
   # Load your model
   model = YOLO("path/to/your/model.pt")
   
   # Validate the model
   metrics = model.val()
   print(f"mAP50-95: {metrics.box.map}")
   print(f"mAP50: {metrics.box.map50}")
   print(f"mAP75: {metrics.box.map75}")

   And using CLI:

   yolo val model=path/to/your/model.pt

5. Quantization-Aware Training: If you haven't already, consider using quantization-aware training (QAT) to better prepare your model for lower precision inference. This can help mitigate the accuracy drop when converting to int16.

6. Alternative Quantization: If int16 is a requirement, you might need to explore other quantization techniques or frameworks that better support int16 without significant accuracy loss.

If you continue to experience issues, please share more details about your setup and any error messages you encounter. This will help us assist you more effectively.

Looking forward to your response! 😊

[madasuvenky]

I need int16 accuracy

[pderrenger]

Hi there! 👋

Thank you for reaching out. Achieving high accuracy with int16 quantization can be challenging, as it often introduces more significant precision loss compared to float32. Here are a few steps to help you improve your int16 model accuracy:

1. Quantization-Aware Training (QAT): If you haven't already, consider using quantization-aware training. QAT simulates the effects of quantization during the training process, helping the model adapt to lower precision and potentially improving accuracy.

2. Validation Settings: Ensure your validation settings are optimized. Here’s a quick example of how to validate your model using Python:

   from ultralytics import YOLO
   
   # Load your model
   model = YOLO("path/to/your/int16_model.pt")
   
   # Validate the model
   metrics = model.val()
   print(f"mAP50-95: {metrics.box.map}")
   print(f"mAP50: {metrics.box.map50}")
   print(f"mAP75: {metrics.box.map75}")

   And using CLI:

   yolo val model=path/to/your/int16_model.pt

3. Check Latest Versions: Make sure you are using the latest versions of all relevant packages, including Ultralytics YOLOv8. Sometimes, updates include fixes and improvements that can help with quantization issues.

4. Alternative Quantization: If int16 is a strict requirement, you might need to explore other quantization techniques or frameworks that better support int16 without significant accuracy loss.

5. Hyperparameter Tuning: Fine-tune your model’s hyperparameters to see if you can achieve better performance with int16 weights. Adjusting parameters like learning rate, batch size, and others can sometimes yield better results.

If you can provide a minimal reproducible example, it would greatly help in diagnosing the issue more effectively. You can refer to our guide on creating a minimum reproducible example.

Feel free to reach out if you have any more questions or need further assistance. Happy coding! 😊

[eddmar1993]

After the training of YOLOv8, I got my metrics results in csv file. However, I did validation. But I cannot get my results in csv format. Only images (png & jpeg). How to get validation results in csv?

[glenn-jocher]

Hey there! 😊 To get your validation results in CSV format, you can utilize the save_json argument during validation, which will save the results in a JSON file. From there, you can easily convert the JSON file to CSV using Python. Here's a quick example:

from ultralytics import YOLO
import json
import pandas as pd

# Validate the model and save results to JSON
model = YOLO('path/to/your/model.pt')
model.val(save_json=True)

# Assuming your JSON results are saved as 'results.json'
with open('results.json') as f:
    data = json.load(f)

# Convert JSON to DataFrame then save as CSV
df = pd.DataFrame(data)
df.to_csv('validation_results.csv', index=False)

This will give you a CSV file with your validation results. Hope this helps! Let me know if you have any more questions.

[RudeNinja]

Hey, I am trying to get metrics such as recall and also trying to save some images in validation. I am using coco.yaml. Any idea how I might do it?

[glenn-jocher]

Hey there! 👋

For calculating recall and saving validation images using coco.yaml, you can simply run the validation process with the save_json and plots arguments set to True. This will save the results in JSON format (which includes recall among other metrics) and also save plots of predictions vs. ground truth for visual evaluation.

Here's how you can do it:

yolo detect val model=yolov8n.pt data=coco.yaml save_json=True plots=True

This command will validate your model and save the detailed metrics and images in the runs directory. You can find recall in the saved JSON file and the validation images in the plots directory.

Hope this helps! Let me know if you have any other questions. 😊

[madasuvenky]

When I convert the YOLO v8 model weights to int16 and validate, I'm getting 0 accuracy, but with float32 model weights, I'm getting 0.87 accuracy.

[glenn-jocher]

Hi there! 👋

Thank you for reaching out. The issue you're experiencing with the int16 model weights resulting in 0 accuracy is likely due to the quantization process. Quantizing to int16 can sometimes lead to significant loss in model precision, especially if not done carefully.

Here are a few steps you can take to troubleshoot and potentially resolve this issue:

1. Verify Quantization Process: Ensure that the quantization process is correctly implemented. Sometimes, additional steps such as calibration with representative data are required to maintain model accuracy.

2. Check Compatibility: Verify that the int16 quantization is supported and optimized for the hardware and framework you are using. Some frameworks may not fully support int16 quantization, leading to poor performance.

3. Use FP16 Instead: If int16 quantization is causing issues, consider using FP16 (half-precision) instead. FP16 often provides a good balance between model size reduction and maintaining accuracy. You can enable FP16 by setting the half argument to True during validation:

   yolo detect val model=yolov8n.pt data=coco.yaml half=True

4. Update Packages: Ensure you are using the latest versions of all relevant packages. Sometimes, bugs and issues are resolved in newer releases.

5. Provide a Reproducible Example: If the issue persists, providing a minimum reproducible example can help us diagnose the problem more effectively. You can find guidelines on creating a reproducible example here.

Here's an example of how you might validate your model with FP16 precision:

from ultralytics import YOLO

# Load a model
model = YOLO("yolov8n.pt")  # or your custom model

# Validate the model with FP16 precision
metrics = model.val(data="coco.yaml", half=True)
print(metrics.box.map)  # mAP50-95

Feel free to explore more about validation settings and metrics in the Ultralytics documentation.

I hope this helps! Let me know if you have any further questions. 😊

[madasuvenky]

Thankyou but i need int16 accuracy

[glenn-jocher]

Hi there! 👋

Thank you for your patience. Achieving high accuracy with int16 quantization can be challenging due to the precision loss that occurs during the quantization process. Here are a few steps you can take to improve the accuracy of your int16 quantized model:

1. Calibration with Representative Data: Ensure that you are using a representative dataset for calibration during the quantization process. This helps the model maintain accuracy by better approximating the range of values it will encounter during inference.

2. Quantization-Aware Training (QAT): If possible, consider using quantization-aware training. This technique simulates quantization during the training process, allowing the model to adjust its parameters to better handle the reduced precision.

3. Verify Quantization Implementation: Double-check the implementation of your int16 quantization to ensure it is correctly applied. Sometimes, small errors in the quantization process can lead to significant drops in accuracy.

4. Use FP16 as an Alternative: If int16 quantization continues to cause issues, you might want to consider using FP16 (half-precision) quantization instead. FP16 often provides a good balance between model size reduction and maintaining accuracy. You can enable FP16 by setting the half argument to True during validation:

   yolo detect val model=yolov8n.pt data=coco.yaml half=True

5. Update Packages: Ensure you are using the latest versions of all relevant packages. Sometimes, bugs and issues are resolved in newer releases.

6. Provide a Reproducible Example: If the issue persists, providing a minimum reproducible example can help us diagnose the problem more effectively. You can find guidelines on creating a reproducible example here.

Here's an example of how you might validate your model with FP16 precision:

from ultralytics import YOLO

# Load a model
model = YOLO("yolov8n.pt")  # or your custom model

# Validate the model with FP16 precision
metrics = model.val(data="coco.yaml", half=True)
print(metrics.box.map)  # mAP50-95

Feel free to explore more about validation settings and metrics in the Ultralytics documentation.

I hope this helps! Let me know if you have any further questions. 😊

[qdvan3546]

I want to get parameters of a class in metrics and save it to csv. Thank you very much.

[pderrenger]

Hi there! 👋 To save class-specific metrics like mAP, precision, and recall to a CSV file after validation, follow these steps:

Python API Example

After running validation, you can access class-specific metrics from the metrics object and save them using pandas:

import pandas as pd
from ultralytics import YOLO

# Validate your model
model = YOLO('yolov8n.pt')  # or your custom model
metrics = model.val()

# Assuming you want mAP for each class
map50_95_per_class = metrics.box.maps.tolist()

# Create a DataFrame
df = pd.DataFrame({
    'class': model.names,
    'mAP50-95': map50_95_per_class
})

# Save to CSV
df.to_csv('class_metrics.csv', index=False)

CLI Approach

If you're looking to perform post-processing after using the CLI for validation, you'd need to parse the output logs or JSON file (--save_json for generating a JSON file) and then manually write to a CSV using a similar approach or any preferred scripting language.

Remember, if you choose to save results directly from validation with --save_json, the resulting file can be easily parsed and converted into a CSV with your preferred data handling tools like pandas or even Excel.

Let me know if this helps or if you have any more questions! 😊

[HafizRaheem]

hey

i want to do the evaluate my model i already get the detection images using ignition gazebo and ros2 but now i want to evaluate my model like accuracy or MAP or recall how to do it i did not have any dataset or any annotations i am using pretrained yolov8 model and using coco8.yaml

[HafizRaheem]

for option 2 Run Predictions on Real-Time Images. what it mean for this sentence (could be a ROS2 subscriber saving images to disk temporarily) before start the detection i have to save images after the ros2 but before the detection start using yolo and then pass those images to the yolov8 for detection

and for predict does it give me the precision or recall
that is the main thing i have to do it other than that i have images with scores
file:///home/hafiz/ros2_ws/processed_images/processed_10.jpg

[HafizRaheem]

or first i have to train my model and then do everything download the dataset from the kaggle and trian it and do everything

[glenn-jocher]

Hey there! 👋

Absolutely, if you've already got a pretrained model and have newly annotated images in COCO format, you’re on a fantastic track. Since the images are already detected using Ignition Gazebo and ROS2, and now that you've annotated some of them for evaluation, you’re ready to move on to the validation part using Ultralytics YOLOv8.

For validation, you don't need to re-train your model unless you're aiming to fine-tune it on very specific data from your environment. Given you're using a pretrained model, here’s how you can proceed:

1. Prepare your dataset YAML file, ensuring it points to your annotated images and COCO-style annotations. Your YAML might look something like this (adjust paths as necessary):

val: path/to/your/images
annotations: path/to/your/annotations.json

2. Run validation to evaluate your model against your manually annotated dataset. This will give you precision, recall, and mAP metrics, providing insight into how well your model is performing.

yolo detect val model=path/to/your/pretrained/model.pt data=path/to/your/dataset.yaml

This step uses the annotations you've created to compare against the model's predictions, allowing you to assess the model's accuracy.

Since you're handling real-time images and have already annotated them, this validation process will enable you to evaluate the YOLO model's effectiveness in your specific application scenario.

For real-time detection, there's typically no need for a separate dataset aside from monitoring the model's general performance or unless you're using it for continuous learning. So, you're correct; for real-time detection, you proceed with the model prediction directly on incoming images from Ignition Gazebo and ROS2.

Please let me know if you need further guidance or have more questions. Best of luck with your project and your upcoming deadline! 😊

[HafizRaheem]

Thank You but confused how to do it this part i share my code do i have to do this in sperate or do it in same file that i share i have some images with bounding box and scores. now i don't need to rerun the detection or my node so i have annotaton json file and images that i use and draw bounding box using cvat and save as coco format just used those for precison and recall.
sorry i know i am asking stupid question but i have no choice

[glenn-jocher]

Hey there! 😄 It looks like you're on the right track with creating annotations for your images in COCO format using CVAT. Now, since you have both images and annotations, you don't need to rerun detections for validation purposes. You can use your annotations and images directly to calculate precision, recall, and mAP metrics.

Here’s the simplified process:

1. Ensure your annotated dataset (images + COCO format annotations) is organized correctly. Your .yaml file for the dataset should correctly point to the location of your images and annotations.

2. Validate your model using the dataset to get precision, recall, and mAP. Here's how you can do it:

   from ultralytics import YOLO
   
   # Load your model
   model = YOLO('path/to/your/pretrained/model.pt')
   
   # Validate the model
   metrics = model.val(data='path/to/your/dataset.yaml')
   print(metrics.box.map, metrics.box.map50, metrics.box.map75)

Or via CLI,

yolo detect val model=path/to/your/pretrained/model.pt data=path/to/your/dataset.yaml

3. No need for separate detection runs for validation. The val process will internally use your model to predict on the images specified in your dataset YAML and compare those predictions against your annotations to compute the evaluation metrics.

4. Prediction and Validation are separate steps. Predictions are for visual inspection or application integration (using detected bounding boxes and scores). Validation (with .val()) computes how well your model is performing statistically (precision, recall, mAP).

For real-time detections coming from Ignition Gazebo and ROS2 to YOLOv8, you're essentially running predictions. If you wish to evaluate these real-time detections, you would need a way to capture the inferred annotations and have a ground truth to compare against, which might require building a custom process.

No questions are stupid; we all start somewhere! 👍 If you're under a tight deadline and need more detailed help or file reviews, consider reaching out directly on GitHub issues or discussions specific to the Ultralytics YOLO repository. The community and developers are pretty responsive there. Best of luck with your project and deadline!

[UsmanRashid12]

Hello i have two questions.
First one: i have trained model on custom dataset for object detection. After testing the model is giving false positives i want to calculate the false positives? How to calculate the false positives that how many times it is giving me false positive.
Second one: I want to what save_hybrid does? and how to implement it what are the benefits of it.

[UsmanRashid12]

Hello thankyou for your reply. To the first answer i have this already but wanted to see the FP directly. Secondly what if i want to run evaluations on my whole model and then make a confusion matrix on it. Like run the model on all my dataset including all the images of train test valid and then create a confusion matrix of it. The issue i face is when running inference on each jpeg file it saves the image just by save=True but the evaluations such as mask, bbox, keypoints, names and orig img. when i try to save them it gives me error even i cant convert them to JSON because of object error in list. so how do you recommend i do this.

Issue with my model is it is detecting false positives with a high percentage so i want to see what the problem really is between those two wrongly detected classes.

[UsmanRashid12]

Also there is not much information available about the save_hybrid=True argument ive searched it online that how does it shows us the predictions with the model against actual labels. Because i get results in batches of images , confusion matrix, and others like P-curve, R_curve. after val

[pderrenger]

Hello! Thanks for reaching out. 🙌

For the save_hybrid functionality, by setting save_hybrid=True during the validation process, the model will save a hybrid version of labels. This means it will combine your model predictions with the original labels in your validation dataset. This can immensely help in analyzing the model's performance by visually comparing its predictions against the ground truth directly on the images. You can see these comparisons in the resulting batch of images, confusion matrices, P and R curves, etc., provided after running yolo detect val with save_hybrid=True. It's a powerful tool for diagnosing model behavior, especially in identifying false positives or understanding class-wise performance.

To address the issue of saving evaluations like masks, bboxes, keypoints, names, and original images, it sounds like you're looking for a structured output. Typically, these details are part of the Python object returned by validation or prediction methods and might not be directly serializable to JSON due to their complex structures (e.g., tensors or custom objects).

For a comprehensive evaluation, including generating a confusion matrix that visually represents model performance across all your dataset images (train, test, validate), you'd likely benefit from aggregating predictions across your dataset in a structured form like a CSV or a database, then using a separate script or tool to calculate and visualize the confusion matrix.

If you're encountering serialization issues, consider converting these objects to more JSON-friendly structures (like converting tensors to lists) before attempting to convert to JSON. Here’s a quick snippet for converting a prediction result to a list (Pythonic way), which might give you an idea:

# Assuming 'results' is the output from model validation or prediction
predictions = [result.xyxy[0].tolist() for result in results]

For identifying why your model is giving false positives between two classes, diving into the save_hybrid results can provide visual clues. Furthermore, systematically reviewing the confusion matrix and the specific images associated with those false positives can reveal patterns or specific features the model might be misinterpreting.

I hope this helps! If you have any more questions or need further assistance, feel free to ask. Happy coding! 😊

[UsmanRashid12]

Hello thankyou so much for this detailed answer i used save_hybrid feature. i have one more question

If i want to see the wrongly predicted image in the model how can i do that. For example my model is wrongly predicting an image giving me a false positive can i see through validation or any argument that the model predicted which image wrong?

[pderrenger]

@UsmanRashid12 hey there! 😀 Really glad the save_hybrid feature was helpful for you! To identify and inspect wrongly predicted images or false positives more directly, you might consider running validation with detailed logging enabled or utilizing the save_hybrid=True feature to visually inspect outputs.

For validation, running the command below generates a detailed report, including a confusion matrix that can highlight where your model tends to predict wrong classes:

yolo detect val model=path/to/your/model.pt data=your_dataset.yaml

For save_hybrid=True, unfortunately, the YOLOv8 documentation is sparse on this. Generally, it combines model predictions with ground-truth annotations on saved output images, allowing you to visually compare predictions against truth directly for each image.

To further drill down on specific false positives or to build a custom confusion matrix from your entire dataset, consider collating predictions programmatically:

from ultralytics import YOLO

model = YOLO('path/to/your/model.pt')

# For each image in your dataset...
for path in dataset_paths:
    results = model.predict(path)
    # Process each result

You can structure predictions alongside labels for all images to analyze/model performance comprehensively, export them to JSON or another format, and then use that data to specifically identify and examine false positives more closely.

Hope this helps! Let me know if there's anything else you're curious about. 🌟

[UsmanRashid12]

Hello i have two questions kindly help me out.

when i have custom trained model. if i load the model and then to detect use" !yolo detect val model=/weights/best.pt data=/data.yaml save=True save_json=True conf=0.85 iou=0.5 split=val " can i give it a new yaml file other than the one which model was trained. For example i used an old dataset to train the model. Now i have new data on which i want to validation of the model i upload data in valid folder and give its yaml file to the model to load those images and validate against the model. (THOSE ARE NEW IMAGES ADDED AFTER TRAINING IN VALID) will it give me accurate results on the new dataset against the model trained? I added 10k images to my Valid folder and want to detection on those new images and see the validation of the model.

Second question is that does conf threshold matters while doing validation? and can you please explain how is the confusion matrix is build after validation because its prediction against classes so it goes and compares with the ground truth? Please explain thankyou

[pderrenger]

@UsmanRashid12 hello! I'm happy to help with your questions. 😊

1. Validating on a New Dataset: Yes, you can specify a new YAML file that points to a different dataset for validation, such as your new data.yaml with the updated valid folder images. YOLOv8 models adapt to the dataset specified during the validation phase. So, if your model was trained on an older dataset and you now have a new set of images, simply adjust your data.yaml to point to these new images for validation. This approach will give you insights into how your model performs on the new dataset. However, keep in mind, if the new dataset is significantly different or more challenging, the accuracy might vary.

Example command adjustment:

!yolo detect val model=/weights/best.pt data=/new_data.yaml save=True save_json=True conf=0.85 iou=0.5 split=val

2. Confidence Threshold and Confusion Matrix: Yes, the confidence (conf) threshold matters during validation because it filters out detections below the specified confidence level, affecting the final metrics like precision, recall, and mAP. Lowering the threshold might increase the number of detected objects but can also introduce more false positives. Raising it can reduce false positives but might also miss some true positives.

The confusion matrix is constructed by comparing the predicted class labels against the true class labels for all the validations or test examples. It helps in understanding the performance of the model for each class, indicating how often it confuses two classes. For building it after validation, each prediction is matched with ground truth based on IoU criteria. True Positives (TP), False Positives (FP), and False Negatives (FN) are then counted for each class which goes into making the confusion matrix.

Hope this clarifies your queries! Let me know if there's anything else you want to know.

[TersteegC]

Hi, I want to use the validate function to finetune my trained model and see how it performs against real life pictures. There is however a difference in parameters available between "predict" mode and val "mode". For instance, I would like to use specific settings for Augment and retina_mask as part of the validation. Is this possible? And out of curiosity, why are the default settings for Val different from Predict?

[glenn-jocher]

Hey there! ✨ Thanks for reaching out with your query.

To tailor your validation settings for Augment and retina_mask, you can directly modify these parameters in your dataset's YAML file or provide them as additional arguments in your custom script. These options, however, are not directly available through the CLI for the val mode. Changes to parameters like these are usually done within the training script or the dataset configuration to affect the validation phase indirectly.

Regarding defaults differences between val and predict modes – these modes serve different purposes and thus have distinct default settings optimized for their specific use-cases. Val mode focuses on model evaluation against a known dataset with ground truth for performance metrics computation, hence might prefer higher accuracy over inference speed. In contrast, predict mode is tailored towards real-time inference, possibly preferring speed and real-world applicability.

Here's a quick Python snippet to kickstart customization for validation (ensure your dataset YAML reflects your augmentation desires):

from ultralytics import YOLO

# Load your custom model
model = YOLO('path/to/your/model.pt')

# Validate the model with custom settings
validation_results = model.val(data='your_dataset.yaml', imgsz=640, batch=16, conf=0.25, iou=0.6)

Remember, Augment and custom settings may require adjustments within your training/validation Python script or dataset YAML rather than through CLI commands. Hope this helps clear things up! 🚀

[SimCromarty]

Hi, I am completing a vision project for an automated industrial packing cell. Currently I have trained 3 custom models for the different tasks. I want to test the performance of each with different scenarios like bright, dim, natural, artificial lighting and varied backgrounds or similar objects. What would be the best method to do so? Would it be best to create these standard test datasets, link with yaml then use the val function? Or use predict and count correct / incorrect detections and work out metrics from there? Once I have done this I also want to investigate performance with different model architecture sizes and training times so has to be easily repeatable. How is best to implement basically, thanks in advance!

[pderrenger]

@SimCromarty hello! 🌟 For evaluating your models under different lighting conditions and backgrounds, using the val function with standard test datasets linked via YAML is indeed a good approach. This allows you to leverage YOLOv8's built-in metrics like mAP50, mAP75, and others for a comprehensive evaluation. Here's a quick example on how you might set up your val step:

yolo val model=path/to/your/model.pt data=your_test_dataset.yaml

For repeated experiments across various model architectures and training durations, setting up separate YAML files for each scenario can streamline the process. This ensures that you can easily switch between datasets and model configurations without manually counting detections.

Additionally, remember that YOLOv8 models remember their training configurations, making it straightforward to repeat validation steps with different settings. Modifying the imgsz, batch, and other parameters in your dataset's YAML file allows for flexible adaptation to varied scenarios.

Hope this helps! If you have any more questions or need further assistance, feel free to ask. Good luck with your project! 🚀

[Keletom1]

Hi there!

I have got a similar question and am still confused on how to evaluate my results.

So i have 16 different datasets and i want to find out what impact which dataset has on a model. I trained one YOLOv8-seg-model on each dataset and now want to validate these models and compare their results.
Of course, I could just look at the precision, recall, and F1-score curves, but I would prefer to store these values in a TXT or CSV file for easier comparison. Unfortunately, setting save_json=True doesn't help because it doesn't include these specific values.

Is there a simple way to have these values printed out in such a file (without all the curve values)?

This would really help a lot!

Thanks in advance,
Tom :)

[glenn-jocher]

Hi Tom! For exporting precision, recall, and F1-score metrics from YOLOv8-seg validation, you can access these values programmatically through the Python API and save them to a CSV file. After running model.val(), use:

import csv

metrics = model.val()
with open('metrics.csv', 'w') as f:
    writer = csv.writer(f)
    writer.writerow(['Metric', 'Value'])
    writer.writerow(['Precision', metrics.box.p])
    writer.writerow(['Recall', metrics.box.r])
    writer.writerow(['F1-Score', metrics.box.f1])

This gives you a clean file with just the core metrics. For multiple models, wrap this in a loop. Let us know if you need more specifics!

[slhermite]

Hi, an iou (1) threshold is used during detection to prevent duplicate bounding boxes.
But another threshold ior (2) is used during validation to determine when a predicted bounding box corresponds to the ground truth, this allows us to "decide" whether a prediction is true or false positive.
If I've understood correctly, in metrics = model.val(ior=0.6) you fill in the threshold (1) and metrics contains precision and recall. My question is as follows: With what threshold ior (2) are precision and recall calculated? Can you choose this threshold (2) in model.val ?

[glenn-jocher]

Hi there!

Great question! The IoU threshold for the confusion matrix is set slightly lower at 0.45 to provide a more lenient criterion for considering detections as matches to the ground truth. This can help in analyzing scenarios where the bounding boxes are not perfectly aligned but are close enough to be considered correct detections, thus giving a broader perspective on the model's performance.

This setting helps in understanding the model's behavior under slightly less strict conditions, which can be particularly useful in practical applications where perfect alignment of predicted and actual bounding boxes is rare.

If you have more questions or need further clarification, feel free to ask. Happy to help! 😊

[learn01one]

hello,can everyone change the IoU threshold for calculating precision and recall metrics (iou2)?

[glenn-jocher]

Hello!

Currently, the IoU threshold for calculating precision and recall metrics (iou2) during validation is fixed and not directly adjustable through the model.val() function. This threshold is set to ensure consistency in how model performance is evaluated across different scenarios and datasets.

If you need to experiment with different IoU thresholds for validation metrics, you might consider modifying the source code or using post-processing scripts that can re-evaluate the predictions against varying IoU thresholds.

Thank you for reaching out with your question! If you have more inquiries or need further assistance, feel free to ask. 😊

[Mattshanevdberg]

Hi, Thanks for this information.

I have a follow up question for the IoU (1) - the NMS one to reduce duplicates. How is this calculated or worked out? ie what does IoU of 0.6 actually mean in terms of removing duplicate bounding boxes? Does it mean that only if the bounding boxes overlap by more than 60% they are removed?

Note, I am only performing single class detection.

Thank you so much for the assistance!

[glenn-jocher]

Hi, thanks for your question.

The IoU threshold for NMS (iou1) is used to filter out duplicate bounding boxes. An IoU of 0.6 means that if two bounding boxes overlap by more than 60%, the one with the lower confidence score will be removed. This helps in reducing redundant detections and keeping only the most confident predictions.

For single-class detection, this threshold ensures that overlapping boxes for the same object are minimized, improving the clarity of your results.

If you have any more questions, feel free to ask!

[HumairaBano]

hi,
i am validating yolov8 on pidray dataset using the following code but got this error. I am unable to resolve it as I could not find any difference in class labels. there are 12 classes.
from ultralytics import YOLO
path = "/content/drive/MyDrive/datasets/pidray/yolov8-pidray/weights/yolov8_pidray.pt"
model = YOLO(path)
metrics=model.val(data="/content/drive/MyDrive/datasets/pidray/yolov8-pidray/data/pidray.yaml")

the error is
Ultralytics YOLOv8.1.47 🚀 Python-3.10.12 torch-2.2.1+cu121 CPU (Intel Xeon 2.20GHz)
YOLOv8s summary (fused): 168 layers, 11130228 parameters, 0 gradients, 28.5 GFLOPs
val: Scanning /content/drive/MyDrive/datasets/pidray/labels/hidden.cache... 1241 images, 3764 backgrounds, 0 corrupt: 100%|██████████| 5005/5005 [00:00<?, ?it/s]
Class Images Instances Box(P R mAP50 mAP50-95): 100%|██████████| 313/313 [1:09:00<00:00, 13.23s/it]

KeyError Traceback (most recent call last)
in <cell line: 1>()
----> 1 history=model.val(data="/content/drive/MyDrive/datasets/pidray/yolov8-pidray/data/pidray.yaml")

7 frames
/usr/local/lib/python3.10/dist-packages/ultralytics/utils/metrics.py in plot_mc_curve(px, py, save_dir, names, xlabel, ylabel, on_plot)
476 if 0 < len(names) < 21: # display per-class legend if < 21 classes
477 for i, y in enumerate(py):
--> 478 ax.plot(px, y, linewidth=1, label=f"{names[i]}") # plot(confidence, metric)
479 else:
480 ax.plot(px, py.T, linewidth=1, color="grey") # plot(confidence, metric)

KeyError: 11

[glenn-jocher]

Hey there! 🚀 It looks like you're encountering a KeyError during validation, which often indicates a mismatch between the class labels defined in your dataset and those the model is expecting. Since you mentioned having 12 classes, ensure your pidray.yaml file accurately reflects all your class labels and that they are zero-indexed (i.e., the first class is 0, the second is 1, etc.).

Here's a quick check you can do:

• Open your pidray.yaml and ensure the nc: (number of classes) is set to 12 and the names: list includes all your classes indexed correctly.

If your pidray.yaml looks correct, the issue might be with the model expecting a different number of classes. This can happen if the model was trained on a dataset with a different class structure. Make sure you're using the correct pre-trained model or that your custom model was trained with the same class setup.

As a simple code snippet for validating, your setup seems fine. Just ensure the paths and class definitions align perfectly.

Hope this helps! If you're still stuck, could you share the class definitions from your pidray.yaml? That might provide more clues. 🕵️‍♂️

[HumairaBano]

model is trained on same pidray dataset. It is github project https://github.com/haksorus/yolov8-pidray.
yes sure the .yaml file is here

path: /content/drive/MyDrive/datasets/pidray
train: images/train
val: images/hidden
test: images/hidden

names:
0: baton
1: pliers
2: hammer
3: powerbank
4: scissors
5: wrench
6: gun
7: bullet
8: sprayer
9: handcuffs
10: knife
11: lighter

[glenn-jocher]

Hey there! 👋 It looks like your pidray.yaml is set up correctly with 12 classes, which matches the class indices you've provided. The KeyError: 11 suggests an issue accessing the last class, "lighter". This could be due to how the classes are interpreted during the validation process.

Since the model and the dataset are correctly aligned, ensure that:

• Your dataset annotations are consistent with the class labels in pidray.yaml.
• There's no missing/corrupt label file in the dataset that might cause this error during validation.

Remember, YOLOv8 seamlessly integrates the configuration from your training for validation, so mismatches are usually about dataset labeling or class indices. If the issue persists, revisiting the dataset to confirm all annotations are correct and complete for each class might help.

For clarity, validation is as simple as:

from ultralytics import YOLO

# Assuming your model and dataset are correctly set
model = YOLO('/content/drive/MyDrive/datasets/pidray/yolov8-pidray/weights/yolov8_pidray.pt')
metrics = model.val(data="/content/drive/MyDrive/datasets/pidray/yolov8-pidray/data/pidray.yaml")

# This should run without the KeyError if classes match dataset annotations

I hope this points you in the right direction! Let us know if there's anything else we can help with. 🚀

[YunYeTang]

hi,
when I use yolov8-obb to validate DOTAv1, I found that confusion matrix is empty. Does it mean that confusion matrix is not supported in OBB validation?

[glenn-jocher]

Hi there! 👋

At the moment, confusion matrices might not be directly supported for OBB validation in the way they are for standard bounding box detections. This could explain why you're encountering an empty confusion matrix during your DOTAv1 validation using yolov8-obb.

However, evaluating the model's performance can still be done using other metrics provided during validation, such as mAP. If you need more detailed insights or have a specific evaluation need, you might consider manually computing confusion matrices based on the raw prediction and ground truth data obtained during validation.

Here's a quick snippet on how you could potentially start to manually calculate a confusion matrix (for a hypothetical scenario):

from sklearn.metrics import confusion_matrix
y_true = [...]  # True labels
y_pred = [...]  # Predicted labels by your model
conf_mat = confusion_matrix(y_true, y_pred)
print(conf_mat)

Remember, this requires you to post-process the model outputs and ground truth labels to fit into a format suitable for confusion matrix computation.

We're always looking to improve, so feedback like yours is invaluable. Feel free to share any more thoughts or questions you might have! 🚀

[YunYeTang]

thank u very much

[glenn-jocher]

Hey there! 🚀

Absolutely, thank you for reaching out!

Regarding your observation about the confusion matrix being empty during DOTAv1 validation with yolov8-obb, it's essential to note that certain specialized metrics or visualizations might need specific support or additional setup procedures when dealing with oriented bounding boxes (OBB). At the moment, direct confusion matrix visualization support may not be explicitly designed for OBB validation in the documentation.

But, worry not! For a comprehensive evaluation, you can still rely on other metrics that are provided during the validation process, such as mAP. If a confusion matrix is critical for your analysis, consider manually extracting prediction and ground truth data to compute it as needed.

If there's anything more specific you'd like assistance with or any further questions, feel free to ask. Cheers! 🌟

[ghost]

I use my own post-processing logic on the txts that i get from running inference the validation set. So now I have the post processed txts for the validation set and the original validation txts that i used for training. How to compute the val performance results using the txt files ?

[pderrenger]

@atharva-hude-asu hey there! 👋 It sounds like you've already taken some great steps with your post-processing and have your processed txt files ready to go. To calculate the validation performance using these txt files, you typically need to compare your post-processed predictions against the ground truth annotations in your validation set.

One straightforward approach is to use the evaluation script provided in many object detection repositories, which might require slight modifications to match your data format. However, since you're interested in Ultralytics YOLO, I'd recommend feeding your post-processed results directly back into the YOLO val command and letting it handle the performance calculations.

Here's a basic idea of how to proceed:

1. Ensure that your post-processed txt files follow the same format as the YOLO prediction files, where each line corresponds to a detection formatted as [class_id] [confidence] [x_center] [y_center] [width] [height], with values normalized to the image width and height.

2. Use these txt files in a manner consistent with YOLO expectations for validation datasets. You might need a custom script to align your post-processed results with the expected input for validation.

Unfortunately, if the val mode directly doesn't support external result inputs, you might consider calculating key metrics like mAP, precision, and recall using external scripts or tools designed for evaluating object detection models (many are available online).

Lastly, if you have access to a custom setup that calculates these metrics, simply adjust it to work with your format. This discussion might provide some insights or direction on handling custom evaluation scripts.

If there's anything more specific you'd need help with, please feel free to ask! Happy coding! 🚀

[Huo-Yukang]

[pderrenger]

@Huo-Yukang hello! 😊 To obtain APS (AP small), APM (AP medium), and APL (AP large) metrics through your trained model weight file, you'll simply need to run a validation using the model. Here's how you can do it:

If you're using Python:

from ultralytics import YOLO

# Load your trained model
model = YOLO('path/to/your/trained_weights.pt')

# Validate the model
metrics = model.val()  # Ensure you're validating on a dataset that has been seen during training for accurate metrics.

# Access APS, APM, and APL
print(f"APS: {metrics.box.aps}, APM: {metrics.box.apm}, APL: {metrics.box.apl}")

Or, if you prefer using the CLI:

yolo detect val model=path/to/your/trained_weights.pt

(Note: The CLI will display a comprehensive metrics summary including APS, APM, and APL in the output)

This process evaluates your trained model's performance across different object sizes — small, medium, and large — giving you insightful metrics on how well your model detects objects of varying scales. 📊

If you have any more questions, feel free to ask. Happy validating!

[pderrenger]

Hey there! 😊 It seems like you encountered an AttributeError because aps, apm, and apl metrics might not be directly accessible attributes of the Metrics object after validation. Instead, you can extract similar size-specific AP metrics using the COCO API if you're working with COCO datasets or any custom evaluation script tailored for your specific dataset format.

For a closer look at how metrics like mAP are calculated and possibly dive into size-specific AP calculations, using COCO tools directly might offer more insights.

For COCO datasets, after saving the results in the COCO format (save_json=True during validation), you could use something like this with pycocotools:

from pycocotools.coco import COCO
from pycocotools.cocoeval import COCOeval

# Assuming val.json is your validation set annotations in COCO format
# and results.json is what you saved using save_json=True during validation
cocoGt=COCO('val.json')  
cocoDt=cocoGt.loadRes('results.json')

cocoEval = COCOeval(cocoGt,cocoDt,'bbox')
cocoEval.evaluate()
cocoEval.accumulate()
cocoEval.summarize()

# Access size-specific AP metrics
print(f"APS: {cocoEval.stats[4]}, APM: {cocoEval.stats[5]}, APL: {cocoEval.stats[6]}")

Please note, this example uses COCO evaluation tools directly for size-specific metrics, which might differ slightly from how Ultralytics processes these metrics internally. For the differences you're observing in mAP calculations, ensure the evaluation settings (like IoU thresholds and object sizes) match between your Ultralytics validation process and the pycocotool process. Adjustments to parameters may bring the evaluations more in line with each other. 👍

[sea190]

but the generated results.json in validation has a big problem, The image_id in results.json does not match the image_id in the annotation file.

[glenn-jocher]

Ensure that the dataset configuration file correctly maps image IDs between your annotation file and the images used during validation, as mismatches here can cause the issue you're describing.

[dthief55]

When i tried to evaluation model by command below:

model = YOLO('yolov8n.pt')
metrics = model.val(data=dataset, epochs=100, imgsz=640, plots=True)

Why there was an error appear on the results section (i was using Google Colab)

Error was below:

Ultralytics 8.3.41 🚀 Python-3.10.12 torch-2.5.1+cu121 CUDA:0 (Tesla T4, 15102MiB)
YOLOv8n summary (fused): 168 layers, 3,151,904 parameters, 0 gradients, 8.7 GFLOPs

AssertionError Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/ultralytics/data/base.py in get_img_files(self, img_path)
124 # self.img_files = sorted([x for x in f if x.suffix[1:].lower() in IMG_FORMATS]) # pathlib
--> 125 assert im_files, f"{self.prefix}No images found in {img_path}. {FORMATS_HELP_MSG}"
126 except Exception as e:

AssertionError: val: No images found in /content/drive/MyDrive/Colab Notebooks/[Skripsi] People Tracking with YOLO and Deep SORT/dataset/crowd_detection_mall_dataset/valid/images. Supported formats are:
images: {'pfm', 'mpo', 'webp', 'png', 'bmp', 'jpeg', 'heic', 'tif', 'tiff', 'jpg', 'dng'}
videos: {'ts', 'gif', 'mpg', 'webm', 'mp4', 'mpeg', 'avi', 'mkv', 'wmv', 'm4v', 'asf', 'mov'}

The above exception was the direct cause of the following exception:

FileNotFoundError Traceback (most recent call last)
9 frames
/usr/local/lib/python3.10/dist-packages/ultralytics/data/base.py in get_img_files(self, img_path)
125 assert im_files, f"{self.prefix}No images found in {img_path}. {FORMATS_HELP_MSG}"
126 except Exception as e:
--> 127 raise FileNotFoundError(f"{self.prefix}Error loading data from {img_path}\n{HELP_URL}") from e
128 if self.fraction < 1:
129 im_files = im_files[: round(len(im_files) * self.fraction)] # retain a fraction of the dataset

FileNotFoundError: val: Error loading data from /content/drive/MyDrive/Colab Notebooks/[Skripsi] People Tracking with YOLO and Deep SORT/dataset/crowd_detection_mall_dataset/valid/images
See https://docs.ultralytics.com/datasets for dataset formatting guidance.

I was using dataset from Roboflow Universe, link: https://universe.roboflow.com/mall-dataset/human-counter-kmx6g/dataset/1

How can i solve this kind of problem? Actually the format is already correct because it's from Roboflow, please someone help me to solve this.

[glenn-jocher]

@dthief55 the error indicates that no images were found in the specified validation directory. Please double-check the path (/valid/images) and ensure it contains your validation images in supported formats (jpg, png, etc.). Confirm your dataset is properly structured and matches the expected YOLO dataset format for validation. If issues persist, re-download and verify the dataset provided by Roboflow.

[maomao123-git]

hello！please help me, I have some questions:

1. I compared two models. In F1_curve.png, if all classes of two models are 1, is the model with higher confidence better? model A : all classes 1.00 at 0.56; model B: all classes 1.00 at 0.50, model A is better than model B? only estimation in F1_curve.
2. as shown in Q1, do the P_curve has the similar conclusion?
3. how I can get the F1 score? if all classes 1.00 at 0.56 in the F1_curve.png, is the 1.00 f1 score ?
4. I am confused in the f1-confidence curve and precision-confidence curve. I think the higher confidence value is better if value of all classes are the same.
   Thanks!

[glenn-jocher]

@maomao123-git 1) For the F1_curve, the confidence threshold itself does not determine which model is better. Instead, focus on the highest F1 score, as it represents the optimal balance between precision and recall. A higher threshold simply indicates stricter confidence requirements.
2) The P_curve can provide similar insights, but it focuses solely on precision, so conclusions may differ when considering recall.
3) The 1.00 in the F1_curve indicates the maximum F1 score achieved, irrespective of the confidence value.
4) Higher confidence thresholds reduce false positives but may miss true positives, affecting recall. The "better" threshold depends on your application needs.

For more details on validation metrics, refer to this guide.

[maomao123-git]

Thanks for your reply.
It is possible my dataset is simple causing the same map@0.5. All of them are 0.995. The difference in F1_curve and P_curve figures is the confidence., as shown in previous question.
In F1_curve, as the confidence value close to 1.00, the f1 value will decrease sharply. I can understood it. In all classes 1.00 at 0.80, the confidence value is 0.80. Is it the maximum confidence value when all classes value become 1.00. This is to say, maximum confidence threshold is 0.80.
If the confidence value higher, meanwhile, f1 value do not decrease, is it better? model A: all classes 1.00 at 0.80 model B: all classes 1.00 at 0.78

[glenn-jocher]

Thank you for your detailed follow-up! Regarding your question, the confidence value at which "all classes 1.00" occurs in the F1_curve indicates the threshold where the F1 score is maximized for all classes. However, a higher confidence threshold (e.g., 0.80 vs. 0.78) is not inherently better unless it aligns with your application goals. The key is to focus on the F1 score itself rather than the confidence threshold, as the score balances precision and recall effectively. If F1 does not decrease at higher thresholds, that indicates robustness, which could be beneficial depending on your use case. For further insights, please refer to YOLO Performance Metrics.

[BroenLin]

"yolo val" outputs "p","r","mAP50" and "mAP50:95".
When calculating "p" and "r", what is the threshold setting?
I output the yolov10 result in the json format, but failed to calculate the same "p" and "r" as YOLO automatically outputs.
I tried "iou threshold"=0.5 and "conf threshold=0.001", but the calculated 'p' is too low and 'r' is too high.

[glenn-jocher]

The p (Precision) and r (Recall) metrics during validation are calculated using a confidence threshold of 0.001 and an IoU threshold of 0.6 by default. If your calculated values differ from the automatic YOLO output, ensure your evaluation process matches these thresholds and uses the same dataset and settings. For detailed argument control, refer to the Validation Guide.

[Ares-cz]

I used the save_json = True to save the predictions.json for further analysis. But the image_id seems like a mapping? They are not the images' path.
And in my training, the dataset yaml file setted as the txt file within train/val/test images path
So that how can I use the prediciton.json, are there any official tools?
Many thanks in advance.

[glenn-jocher]

@Ares-cz for mapping JSON image_id to image paths, you can cross-reference the order of images in your validation dataset's text file (specified in your YAML). The IDs correspond to the sequential index of images in your dataset. While we don't provide dedicated mapping tools, you can easily create a path list using:

with open('path/to/val.txt') as f:
    image_paths = f.read().splitlines()

Then access paths via image_paths[image_id].

[AbhayVG]

how to Visualize the ground truth and predicted bounding boxes on some samples from the test dataset.

[glenn-jocher]

@AbhayVG to visualize ground truth and predicted bounding boxes during validation, set the plots=True argument when running validation. This will save comparison plots in your runs/val/ directory. For example:

Python:

model.val(plots=True)

CLI:

yolo val model=

[AbhayVG]

Thank you so much

[glenn-jocher]

You're welcome! For future visualizations, remember that setting plots=True during validation generates comparison plots in runs/val/ showing both ground truth and predictions. Let us know if you need any clarification on the metrics or visualization outputs! 🚀

[Petros626]

when I try to validate my model the AutoBatch is not accepted:

from ultralytics import YOLO
model = YOLO('/home/heizung1/ultralytics_yolov8-obb_ob_kitti/ultralytics/kitti_bev_yolo/run_94_Adam_88.8_87.2/weights/best.pt') # val. Pytorch model FP32
metrics = model.val(data='/home/heizung1/ultralytics_yolov8-obb_ob_kitti/ultralytics/cfg/datasets/kitti_bev.yaml', imgsz=640, 
                    batch=-1, save_json=False, save_hybrid=False, conf=0.001, iou=0.5, max_det=300, half=False,
                    device='0', dnn=False, plots=False, rect=False, split='val', project='kitti_bev_yolo', name='val')

error:
ValueError: batch_size should be a positive integer value, but got batch_size=-1

[glenn-jocher]

@Petros626 for AutoBatch functionality in validation, please use the CLI interface as shown in the Val mode documentation: yolo val model=path/to/model.pt batch=-1. The Python API requires explicit positive integer batch sizes. Let us know if you need further clarification!

[Petros626]

batch=-1 for validation isn't supported. It's for train() only.

[glenn-jocher]

Thanks for reaching out! The batch=-1 AutoBatch feature is indeed exclusive to the CLI yolo val command as shown in the Val mode documentation. For Python validation, please use a positive integer batch size like batch=16. Let us know if you need further assistance! 🚀

[SunyuKy]

I used YOLOv8 for quantization and performed inference on my validation set using my chip, obtaining some inference results. How can I use Ultralytics to separately call the post-processing function and obtain performance metrics such as mAP50-95?

[glenn-jocher]

@SunyuKy to obtain validation metrics like mAP50-95 with custom post-processing, you can manually instantiate the validator and process your results. Here's a concise example using the YOLO Val API:

from ultralytics import YOLO

model = YOLO('yolo11n.pt')  # Load your quantized model
validator = model.validator(args={'data': 'coco8.yaml'})  # Set dataset config
validator(model=model.model)  # Run validation with your chip's inference results
print(validator.metrics.box.map)  # Get mAP50-95

This approach lets you decouple inference and validation while maintaining access to all standard metrics. For full implementation details, see the Ultralytics Val mode documentation.

[aryakpt]

from ultralytics import YOLO

Load a model

model = YOLO("yolov8n.pt") # or your custom model

Validate the model with FP16 precision

metrics = model.val(data="coco.yaml")
print(metrics.box.map) # mAP50-95

how can I get a metrics if i use cli like this:

yolo detect val model=yolov8n.pt data=coco.yaml

[aryakpt]

from ultralytics import YOLO

model = YOLO("yolov8n.pt")  # or your custom model

# Validate the model with FP16 precision
metrics = model.val(data="coco.yaml")
print(metrics.box.map)  # mAP50-95

how can I get a metrics if i use cli like this:

yolo detect val model=yolov8n.pt data=coco.yaml

[glenn-jocher]

@aryakpt when using the CLI validation command, metrics like mAP50-95 are automatically printed to your terminal after validation completes. The results will appear in your console output under the "Metrics" section of the validation report. For more details on CLI output formatting, see the Val mode documentation.

[417dq]

My task is a single-object detection task. I encountered a problem in val. When conf = 0.1 and 0.25, the recall value of 0.25 is higher. In addition, the output recall value seems to be different from the value directly calculated by the confusion matrix. Is this normal? Or is there something wrong

[BedoHatem]

I have a couple of models (a mix of object detection and segmentation), so I want to run them on the same validation dataset to compare them properly. The exact tasks for the seg and obj-det models are the same at a certain level, so I'm aware that seg models give a B-Box and Mask, while obj-det gives just a B-Box.

With that said, will a segmentation dataset work on both seg and obj-det models ?
Or should i make a seg and a obj-det version of the same dataset, so i can use it to equally validate the models ?

Thanks in Advance

[BedoHatem]

I tested it. It seems you can use a segmentation dataset to validate an obj-det model, as the seg datasets include B-Boxes and Masks of the classes.

[glenn-jocher]

@BedoHatem yes, you're correct! Segmentation datasets can be used to validate object detection models because they contain the bounding box annotations needed for detection evaluation. The segmentation masks are simply ignored when the dataset is used with a detection model.

This flexibility is an advantage of the YOLO annotation format - the additional segmentation data doesn't interfere with detection evaluation. For the most accurate comparison between models, using the same validation dataset ensures you're evaluating performance on exactly the same ground truth data.

@417dq regarding your recall values at different confidence thresholds - the situation you described (higher recall at conf=0.25 than at conf=0.1) is unusual but possible in certain edge cases. This can happen if the higher confidence threshold filters out false negatives that were occurring at the lower threshold due to NMS or other post-processing steps. As for the difference between calculated recall and what's shown in the confusion matrix, this likely occurs because the validation metrics are calculated using a different IoU threshold than what's used for the confusion matrix visualization (which is typically 0.45). The metrics output from val() are the more accurate representation of your model's performance.

[caisamuels]

How do you limit the evaluation to specific class IDs similarly to the classes argument used for track?

[e-roe]

Hello,
I was wondering if it's possible to get the confusion matrix from model.val().
If not, is there another way get it?
Thanks in advance!

[e-roe]

Ops,
I found it:
confusion = metrics.confusion_matrix.matrix
Thank you
PS. Is there a way to delete questions?

[glenn-jocher]

@e-roe you're right! The confusion matrix can be accessed via metrics.confusion_matrix.matrix after running model.val(). This gives you the raw matrix data for further analysis.

Regarding deletion of questions - unfortunately GitHub discussions don't allow users to delete their own comments once posted. Only repository administrators can remove comments. But no worries, finding your own solution and sharing it actually helps others who might have the same question in the future!

If you want to analyze the confusion matrix in more detail, you can also save the validation plots using plots=True when running validation:

metrics = model.val(plots=True)
# Access confusion matrix
confusion = metrics.confusion_matrix.matrix

This will generate visualization plots including the confusion matrix in your output directory.

[danegarrido]

Hi, is there an easy way to get information about each image processed during validation? I’d like to retrieve the filenames of the images where the model failed, so I can review them afterward.

[danegarrido]

Hi, is there an easy way to get information about each image processed during validation? I’d like to retrieve the filenames of the images where the model failed, so I can review them afterward.

[Petros626]

Doubled

[glenn-jocher]

@danegarrido to retrieve information about specific images where your model failed during validation, you can use a few approaches:

1. Save detailed results to JSON: Use the save_json=True parameter during validation which creates a predictions.json file with per-image results:

from ultralytics import YOLO

model = YOLO('yolov8n.pt')
metrics = model.val(data='your_dataset.yaml', save_json=True)

2. Save text outputs: Use save_txt=True to save individual detection results for each image:

metrics = model.val(data='your_dataset.yaml', save_txt=True)

3. Create plots for visual inspection: Enable plots=True to generate visualizations of predictions vs. ground truth:

metrics = model.val(data='your_dataset.yaml', plots=True)

After validation, you can analyze the saved JSON or text files to identify images with low detection metrics or high error rates. The JSON file contains image IDs mapped to predictions which can be correlated with your dataset's images.

If you need to implement a more custom approach to specifically flag "failed" images (e.g., images with precision or recall below a certain threshold), you might need to modify the validator code to track and output this information.

[lauravchrobak]

When I run validate without setting plots=True I still get the f1, PR curve plots in the val folder. Potentially the default for this flag is actually true despite it being false in the table above?