Unexplained behavior of the Validation of a Classification model #13653
Comments
|
👋 Hello @Himmelsw4nderer, thank you for your interest in Ultralytics YOLOv8 🚀! We recommend a visit to the Docs for new users where you can find many Python and CLI usage examples and where many of the most common questions may already be answered. If this is a 🐛 Bug Report, please provide a minimum reproducible example to help us debug it. If this is a custom training ❓ Question, please provide as much information as possible, including dataset image examples and training logs, and verify you are following our Tips for Best Training Results. Join the vibrant Ultralytics Discord 🎧 community for real-time conversations and collaborations. This platform offers a perfect space to inquire, showcase your work, and connect with fellow Ultralytics users. InstallPip install the pip install ultralyticsEnvironmentsYOLOv8 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):
Status
If this badge is green, all Ultralytics CI tests are currently passing. CI tests verify correct operation of all YOLOv8 Modes and Tasks on macOS, Windows, and Ubuntu every 24 hours and on every commit. |
|
@Himmelsw4nderer hi there, Thank you for providing a detailed description of the issue and the relevant code snippets. Let's work through this step by step to understand and resolve the problem you're encountering with the validation of your classification model. Initial Checks
AnalysisFrom your description, it seems like there are inconsistencies between the training and validation results, particularly with the confusion matrix. Here are a few points to consider:
Suggested Steps
Example Code for Built-in ValidationHere’s an example of how to use the built-in validation method: from ultralytics import YOLO
# Load your trained model
model = YOLO('runs/classify/train33/weights/best.pt')
# Validate the model on the validation dataset
results = model.val(data='datasets/fold_4/val')
print(results)Additional ResourcesFor more detailed information on validation, you can refer to the Ultralytics documentation on model validation. If the issue persists after these checks, please provide any additional logs or error messages that might help in diagnosing the problem further. Feel free to reach out with any more questions or updates on your progress. We're here to help! 😊 |
|
Hi, I was already on the newest versions of ultralytics and torch Using: results in the same confusion matrix as my custom script: So only the |
|
Hi @Himmelsw4nderer, Thank you for the update and for providing the additional details. It's great to hear that you're using the latest versions of Given that both the built-in validation method and your custom script yield the same confusion matrix, it seems the issue might be related to how the Here are a few steps to help diagnose and potentially resolve the issue:
Here’s an example of how you might add logging to your custom validation script: import os
import cv2
import numpy as np
from sklearn.metrics import precision_score, recall_score, f1_score, accuracy_score, confusion_matrix
from ultralytics import YOLO
# Load your trained YOLO classification model
model = YOLO('runs/classify/train33/weights/best.pt')
def load_images_from_folder(folder):
images = []
labels = []
class_names = os.listdir(folder)
for class_name in class_names:
class_folder = os.path.join(folder, class_name)
if os.path.isdir(class_folder):
for filename in os.listdir(class_folder):
img_path = os.path.join(class_folder, filename)
img = cv2.imread(img_path)
if img is not None:
img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
images.append(img)
labels.append(class_name)
return images, labels, class_names
def validate_model(folder):
images, true_labels, class_names = load_images_from_folder(folder)
class_names = sorted(class_names)
y_true = []
y_pred = []
confidences = []
for img, true_label in zip(images, true_labels):
results = model(img)
for result in results:
probs = result.probs.cpu().numpy()
class_idx = np.argmax(probs.data)
predicted_label = class_names[class_idx]
confidence = probs.data[class_idx]
y_true.append(true_label)
y_pred.append(predicted_label)
confidences.append(confidence)
# Logging
print(f"True Label: {true_label}, Predicted Label: {predicted_label}, Confidence: {confidence}")
y_true_idx = [class_names.index(label) for label in y_true]
y_pred_idx = [class_names.index(label) for label in y_pred]
avg_confidence = np.mean(confidences)
precision = precision_score(y_true_idx, y_pred_idx, average='weighted')
recall = recall_score(y_true_idx, y_pred_idx, average='weighted')
f1 = f1_score(y_true_idx, y_pred_idx, average='weighted')
accuracy = accuracy_score(y_true_idx, y_pred_idx)
conf_matrix = confusion_matrix(y_true_idx, y_pred_idx)
print(f'Average Confidence: {avg_confidence:.2f}')
print(f'Precision: {precision:.2f}')
print(f'Recall: {recall:.2f}')
print(f'F1 Score: {f1:.2f}')
print(f'Accuracy: {accuracy:.2f}')
print('Confusion Matrix:')
print(conf_matrix)
# Specify the folder containing class subfolders with images
data_folder = 'datasets/fold_4/val'
validate_model(data_folder)This should help you identify any inconsistencies in the predictions and ground truth labels. If the issue persists, please let us know, and we can further investigate. Thank you for your patience and cooperation! |
Search before asking
YOLOv8 Component
Val, Predict
Bug
For my studies, I tried implementing and training a classification on the following dataset: https://www.kaggle.com/datasets/grassknoted/asl-alphabet
I used YOLOv8n-cls and trained for different amounts of epochs, the confusion Matrix was always something like that:
Which obviously doesn't look good at all.
I watched the loss values and the val/loss values, which at some point always come to a val/loss around 2.4259:
I found that based on the dataset already quite unusual. So I tried removing the background class and trained again, just because it always kind of predicted the background class.
I got good classification results. At that point, I wanted to calculate more metrics to find out how I can make the training better. The custom script used the normal predict method and generated for the same model as in the first confusion matrix and the same dataset the following confusion Matrix:
The code for the training and the code for the validation + the code for the custom validation is included below
Environment
Ultralytics YOLOv8.2.32 🚀 Python-3.12.3 torch-2.3.1+cu121 CUDA:0 (NVIDIA GeForce RTX 3070 Ti, 8084MiB)
Setup complete ✅ (24 CPUs, 31.2 GB RAM, 77.1/895.6 GB disk)
OS Linux-6.8.12-200.fsync.fc39.x86_64-x86_64-with-glibc2.38
Environment Linux
Python 3.12.3
Install pip
RAM 31.17 GB
CPU 13th Gen Intel Core(TM) i7-13700KF
CUDA 12.1
matplotlib ✅ 3.9.0>=3.3.0
opencv-python ✅ 4.10.0.82>=4.6.0
pillow ✅ 10.3.0>=7.1.2
pyyaml ✅ 6.0.1>=5.3.1
requests ✅ 2.32.3>=2.23.0
scipy ✅ 1.13.1>=1.4.1
torch ✅ 2.3.1>=1.8.0
torchvision ✅ 0.18.1>=0.9.0
tqdm ✅ 4.66.4>=4.64.0
psutil ✅ 5.9.8
py-cpuinfo ✅ 9.0.0
pandas ✅ 2.2.2>=1.1.4
seaborn ✅ 0.13.2>=0.11.0
ultralytics-thop ✅ 0.2.8>=0.2.5
Minimal Reproducible Example
train.py
validation.py
custom_val.py
Additional
No response
Are you willing to submit a PR?
The text was updated successfully, but these errors were encountered: