How to make a confusion matrix in YOLOv5 step by step?

[husnan622]

Search before asking

• I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

I've done model training using YOLOv5 and got pretty good performance. Therefore I want to make a confusion matrix for my needs. But I don't know how to make it and I've tried several tutorials and I still fail. Please help to explain step by step how to make a confusion matrix on YOLOv5 🙏🏻

Additional

No response

[github-actions]

👋 Hello @husnan622, thank you for your interest in YOLOv5 🚀! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset images, training logs, screenshots, and a public link to online W&B logging if available.

For business inquiries or professional support requests please visit https://ultralytics.com or email support@ultralytics.com.

Requirements

Python>=3.7.0 with all requirements.txt installed including PyTorch>=1.7. To get started:

git clone https://github.com/ultralytics/yolov5  # clone
cd yolov5
pip install -r requirements.txt  # install

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training, validation, inference, export and benchmarks on MacOS, Windows, and Ubuntu every 24 hours and on every commit.

[glenn-jocher]

@husnan622 val.py makes confusion matrices automatically. See val.py for usage examples.

[husnan622]

@glenn-jocher I've tried what you suggested regarding val.py but I haven't been able to bring up the confusion matrix image, I don't know what's missing from the command I made

and val.py is still scanning the valid dataset even though I have modified val.py to be task=test

[glenn-jocher]

@husnan622 check your runs/val/exp2 directory, confusion matrix is in there.

[glenn-jocher]

@husnan622 if your data.yaml has a test: key then yes you can run python val.py --task test to use your test split.

[husnan622]

Thanks @glenn-jocher

Are the weights that I use correct?

and usually the code in the confusion matrix is contained actual data path & prediction data path for example y_true and y_pred, where can i find it

[glenn-jocher]

@husnan622 you can use any weights you want as long as they are trained on your --data data.yaml

You can access the confusion matrix code in utils/metrics.py:

yolov5/utils/metrics.py

Lines 126 to 220 in 7845cea

	class ConfusionMatrix:
	# Updated version of https://github.com/kaanakan/object_detection_confusion_matrix
	def __init__(self, nc, conf=0.25, iou_thres=0.45):
	self.matrix = np.zeros((nc + 1, nc + 1))
	self.nc = nc # number of classes
	self.conf = conf
	self.iou_thres = iou_thres
	def process_batch(self, detections, labels):
	"""
	Return intersection-over-union (Jaccard index) of boxes.
	Both sets of boxes are expected to be in (x1, y1, x2, y2) format.
	Arguments:
	detections (Array[N, 6]), x1, y1, x2, y2, conf, class
	labels (Array[M, 5]), class, x1, y1, x2, y2
	Returns:
	None, updates confusion matrix accordingly
	"""
	if detections is None:
	gt_classes = labels.int()
	for gc in gt_classes:
	self.matrix[self.nc, gc] += 1 # background FN
	return
	detections = detections[detections[:, 4] > self.conf]
	gt_classes = labels[:, 0].int()
	detection_classes = detections[:, 5].int()
	iou = box_iou(labels[:, 1:], detections[:, :4])
	x = torch.where(iou > self.iou_thres)
	if x[0].shape[0]:
	matches = torch.cat((torch.stack(x, 1), iou[x[0], x[1]][:, None]), 1).cpu().numpy()
	if x[0].shape[0] > 1:
	matches = matches[matches[:, 2].argsort()[::-1]]
	matches = matches[np.unique(matches[:, 1], return_index=True)[1]]
	matches = matches[matches[:, 2].argsort()[::-1]]
	matches = matches[np.unique(matches[:, 0], return_index=True)[1]]
	else:
	matches = np.zeros((0, 3))
	n = matches.shape[0] > 0
	m0, m1, _ = matches.transpose().astype(int)
	for i, gc in enumerate(gt_classes):
	j = m0 == i
	if n and sum(j) == 1:
	self.matrix[detection_classes[m1[j]], gc] += 1 # correct
	else:
	self.matrix[self.nc, gc] += 1 # true background
	if n:
	for i, dc in enumerate(detection_classes):
	if not any(m1 == i):
	self.matrix[dc, self.nc] += 1 # predicted background
	def tp_fp(self):
	tp = self.matrix.diagonal() # true positives
	fp = self.matrix.sum(1) - tp # false positives
	# fn = self.matrix.sum(0) - tp # false negatives (missed detections)
	return tp[:-1], fp[:-1] # remove background class
	@TryExcept('WARNING ⚠️ ConfusionMatrix plot failure')
	def plot(self, normalize=True, save_dir='', names=()):
	import seaborn as sn
	array = self.matrix / ((self.matrix.sum(0).reshape(1, -1) + 1E-9) if normalize else 1) # normalize columns
	array[array < 0.005] = np.nan # don't annotate (would appear as 0.00)
	fig, ax = plt.subplots(1, 1, figsize=(12, 9), tight_layout=True)
	nc, nn = self.nc, len(names) # number of classes, names
	sn.set(font_scale=1.0 if nc < 50 else 0.8) # for label size
	labels = (0 < nn < 99) and (nn == nc) # apply names to ticklabels
	ticklabels = (names + ['background']) if labels else "auto"
	with warnings.catch_warnings():
	warnings.simplefilter('ignore') # suppress empty matrix RuntimeWarning: All-NaN slice encountered
	sn.heatmap(array,
	ax=ax,
	annot=nc < 30,
	annot_kws={
	"size": 8},
	cmap='Blues',
	fmt='.2f',
	square=True,
	vmin=0.0,
	xticklabels=ticklabels,
	yticklabels=ticklabels).set_facecolor((1, 1, 1))
	ax.set_ylabel('True')
	ax.set_ylabel('Predicted')
	ax.set_title('Confusion Matrix')
	fig.savefig(Path(save_dir) / 'confusion_matrix.png', dpi=250)
	plt.close(fig)
	def print(self):
	for i in range(self.nc + 1):
	print(' '.join(map(str, self.matrix[i])))

[husnan622]

Thank you so much for your help @glenn-jocher

[husnan622]

@glenn-jocher Previously I was able to generate a confusion matrix using val.py, the results are like the following image:

But I want a confusion matrix that only displays True Positive (TP), True Negative (TN), False Positive (FP) and False Negative (FN), for example in the following image:

How to generate a confusion matrix that only displays True Positive (TP), True Negative (TN), False Positive (FP) and False Negative (FN)?

[glenn-jocher]

@husnan622 set normalize=False in ConfusionMatrix()

[husnan622]

What is the background in the confusion matrix image?

How do I remove this section?

[github-actions]

👋 Hello, this issue has been automatically marked as stale because it has not had recent activity. Please note it will be closed if no further activity occurs.

Access additional YOLOv5 🚀 resources:

• Wiki – https://github.com/ultralytics/yolov5/wiki
• Tutorials – https://docs.ultralytics.com/yolov5
• Docs – https://docs.ultralytics.com

Access additional Ultralytics ⚡ resources:

• Ultralytics HUB – https://ultralytics.com/hub
• Vision API – https://ultralytics.com/yolov5
• About Us – https://ultralytics.com/about
• Join Our Team – https://ultralytics.com/work
• Contact Us – https://ultralytics.com/contact

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLOv5 🚀 and Vision AI ⭐!

[charanlsa]

@glenn-jocher I got the confusion matrix as shown,

For this matrix the TP is nil(no value), FP is 1 and TN is 0.94 what these indicates exactly and may I know how to improve the values like TP has to get some value, FP has to decrease??

I really appreciate your help. Thanks in advance..

[syamghali]

@glenn-jocher : For YOLOv5, how do I increase the font size of numbers in the confusion matrix?

[glenn-jocher]

@syamghali to increase the font size of the numbers in the confusion matrix in YOLOv5, you can modify the plot_confusion_matrix() function in the utils/plots.py file. Specifically, you can change the fontsize parameter in the heatmap function call on line 74. The default value is 14; you can increase it to the desired size. However, please note that while increasing the font size may make the numbers in the plot more readable, it may also reduce the space available for the plot, which could make the plot less visually informative.

[muhanadabdul]

pls. What is the background mean in the confusion matrix image? and how can I remove it?

[glenn-jocher]

@muhanadabdul the background area in the confusion matrix image represents values that are not part of the confusion matrix itself. This area is typically used to display legends or colorbars. To remove the background area, you can modify the plot_confusion_matrix() function in the utils/plots.py file of YOLOv5. Specifically, you can remove the code that generates the legend or colorbar or modify the relevant parameters to adjust their size and location. However, please note that removing or modifying these sections may make the plot less informative or harder to read, especially for users who are less familiar with the specific plot.

[muhanadabdul]

Dear, thank you for explanation, but I am afraid I do not understand what the values in the background label mean to the validation phase

the values with the red circle in the image above

[glenn-jocher]

@muhanadabdul hello, the values you have highlighted in the image represent the number of samples that were not considered in the validation phase. These samples may have been excluded from the validation set for a variety of reasons, such as missing annotations or insufficient image quality. The values can be useful to understand the size of the validation set relative to the full dataset and to identify any potential imbalances or biases in the dataset. However, they are generally not included in the confusion matrix or other validation metrics, as they do not represent true positive or negative results. Please let me know if you have any further questions or concerns.

[muhanadabdul]

Dear, thanks for your explanation, now it's clear, but just have another question. the two highlighted values in the image below are related to the same class "mobile_use" Based on your explanation I am confused in understanding their meaning if we let the 0.08 value represent the number of samples that were not considered in the validation phase, what the other value (0.06) mean?

Is there any way to find which images the model excluded from the validation set?
Also, can we hide these two columns (V, H) and their values from the resulting confusion matrix

[glenn-jocher]

@muhanadabdul hello! The value of 0.08 represents the number of samples that were not included in the validation set for the "mobile_use" class. The value of 0.06 represents the percentage of samples for this class that were not included in the validation set. This percentage is calculated as the number of excluded samples (0.08) divided by the total number of samples (1.34) for this class. Regarding whether it's possible to find which images were excluded from the validation set, this depends on how the data was split and stored. If you have access to the code or procedure that was used to split the data, you may be able to identify the excluded images. If the image filenames or IDs are included in the dataset, you may also be able to cross-reference them with the validation set to identify the excluded images. Finally, regarding the question of hiding the V and H columns and their values from the confusion matrix, you can modify the plot_confusion_matrix() function in the utils/plots.py file of YOLOv5. Specifically, you can remove the code that generates the V and H columns, or modify the relevant parameters to adjust their size and location. Please note that modifying the confusion matrix in this way may make it less informative or harder to read, especially for users who are less familiar with the plot.

[muhanadabdul]

Fantastic, many thanks for your dear.

[glenn-jocher]

@muhanadabdul hello! You're very welcome. If you have any further questions or concerns, please don't hesitate to ask. We're here to help!

[charanlsa]

Dear Glenn I implemented the detection but at the time of execution the accuracy was not good so I used with hyperparameters and used for detection and I also used few(10%) negative images in the training but the confusion matrix is as shown, What exactly it describes glenn and How to decrease the 1 over there and How should I increase the accuracy. PLease reply asap as I'm doing the project . [image: image.png]

…

On Wed, 31 May 2023, 08:29 Glenn Jocher, ***@***.***> wrote: @muhanadabdul <https://github.com/muhanadabdul> hello! You're very welcome. If you have any further questions or concerns, please don't hesitate to ask. We're here to help! — Reply to this email directly, view it on GitHub <#10365 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A53QIF6R5QOXGOLDPVUYAZ3XI2XYLANCNFSM6AAAAAASRDZ2NM> . You are receiving this because you commented.Message ID: ***@***.***>

[glenn-jocher]

@charanlsa dear user,

Thank you for reaching out. The confusion matrix you have shared represents the performance of your YOLOv5 model on the validation set. The rows correspond to the ground truth classes, and the columns correspond to the predicted classes. Each element in the matrix represents the number of validation samples that were assigned to a specific ground truth class and predicted class. The diagonal elements represent the number of correct predictions for each class, and the off-diagonal elements represent the number of incorrect predictions.

If you are noticing a low accuracy, one possible approach to increasing it is to adjust the hyperparameters of your YOLOv5 model. Some hyperparameters you might consider tuning include the learning rate, batch size, and number of training iterations. Additionally, you may want to consider increasing the size or diversity of your training set to boost overall performance.

Regarding the specific class where you are noticing a high number of false positives (i.e., ground truth class 1 and predicted class 0), you may want to consider adjusting the class weights or focal loss coefficients to emphasize this class during training. You may also want to examine the annotations for this class carefully to ensure that they are accurate and complete.

I hope this information is helpful. Please let me know if you have any further questions or concerns, and I'll be happy to assist you. Good luck with your project!

Best regards,
Glenn Jocher

[muhanadabdul]

plot_confusion_matrix()

Dear, I did not find the plot_confusion_matrix() function in the utils/plots.py ?

[ryecries]

Hi, sorry for the late reply. so when i edit my metrics.py, do I have to run val.py again? Pada tanggal Rab, 20 Sep 2023 pukul 07.20 dadin852 ***@***.***> menulis:

…

Hello, are you suggesting that you'd like to remove the background FP/FN labels? My approach involves saving the matrix data in metrics.py and then editing it to create the confusion_matrix.png. In metrics.py : def plot(self, save_dir='', names=()): try: import seaborn as sn array = self.matrix / ((self.matrix.sum(0).reshape(1, self.nc + 1) + 1E-6) if False else 1 ) # normalize # Save the matrix data open(f'{my_save_dir}/matrix.txt', 'a+').write(np.array2string(array, separator=',')) array[array < 0.005] = np.nan # don't annotate (would appear as 0.00) ... — Reply to this email directly, view it on GitHub <#10365 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AO376HQNGIBNVDVMXGEHRPTX3IZF5ANCNFSM6AAAAAASRDZ2NM> . You are receiving this because you commented.Message ID: ***@***.***>

[glenn-jocher]

@ryecries no worries about the delay! After editing the metrics.py file, you don't necessarily need to run val.py again. The metrics.py file is responsible for generating the confusion matrix based on the validation results that have already been computed.

Once you have made the desired changes in the metrics.py file, you can run the plot() method to generate the confusion matrix. This can be done independently from the validation process. The matrix will be saved as matrix.txt, and you can use this data to create your custom confusion matrix visualization.

Let me know if you have any further questions or need any assistance!

Cheers!

[RyanTNN]

hello. I have a question: I trained the YOLOv5 and I got the confusion matrix result (confusion matrix picture). but I want to re-create the confusion matrix picture because I want to change the font size or text in the confusion matrix picture. I always run training again to get the new confusion matrix result. are there any ways to create a new confusion matrix without training? can I use the weight result to re-create the confusion matrix picture? Thank you!

[glenn-jocher]

@RyanTNN hi there!

It is not necessary to run the training again to create a new confusion matrix with different font size or text. You can create a new confusion matrix using the existing model weights.

The confusion matrix is generated based on the predictions and ground truth labels during the evaluation process. If you want to change the font size or text in the confusion matrix picture, you can modify the code responsible for plotting the matrix.

In the metrics.py file, you can adjust the font size or other visualization settings within the plot() method. You can explore the matplotlib documentation for customizing the appearance of the confusion matrix.

Once you have made the desired modifications, you can run the evaluation again using the trained model weights by running the val.py script. This will generate a new confusion matrix with the updated visualization settings.

I hope this helps! Let me know if you have any further questions or need any clarifications.

[RyanTNN]

@RyanTNN hi there!

It is not necessary to run the training again to create a new confusion matrix with different font size or text. You can create a new confusion matrix using the existing model weights.

The confusion matrix is generated based on the predictions and ground truth labels during the evaluation process. If you want to change the font size or text in the confusion matrix picture, you can modify the code responsible for plotting the matrix.

In the metrics.py file, you can adjust the font size or other visualization settings within the plot() method. You can explore the matplotlib documentation for customizing the appearance of the confusion matrix.

Once you have made the desired modifications, you can run the evaluation again using the trained model weights by running the val.py script. This will generate a new confusion matrix with the updated visualization settings.

I hope this helps! Let me know if you have any further questions or need any clarifications.

okay. I got it. Thank you for your time.

[glenn-jocher]

@RyanTNN You're welcome! I'm glad I could help. If you have any more questions or need further assistance, feel free to ask. Happy coding!

[Joshnavarma]

@husnan622 val.py makes confusion matrices automatically. See val.py for usage examples.

what is the file used for plotting confusion matrix for yolov7

[Joshnavarma]

@glenn-jocher . could you please help me get confusion matrix and accuracy score for yolov7 on custom dataset

[Joshnavarma]

Why my confusion matrix is not showing values? the image is the from exp folder after training.

[glenn-jocher]

@Joshnavarma It looks like the image link you provided is broken. To better assist you, could you please provide more details or a valid link for the confusion matrix image? Thank you!

[Joshnavarma]

please find the attached image.

Apart from this i have one qestion that my yolov7 gave 85.2 % mAP on custom dataset fo 6k images(50 epochs, batch size 16). is it a acceptable mAP score and why it is giving that big vlaue. the reason i have the question is when i train my model with 100 epochs, batchsize=32 it gave around 64%mAP. now i doubt if i did anything wrong in my yolov7 training. i have used yolov7-training.pt file for trainng my custom data. am doing masters in data nalytics and this is my final year project i want to justify why the score has that big difference.

Appreciate your help. Thanks.

[glenn-jocher]

@Joshnavarma Thank you for sharing the confusion matrix image. It seems that the provided link is not accessible. If you could upload the image to a public image hosting service (such as Imgur or PostImage) and share the new link, I would be happy to take a look.

Regarding your question about the mAP score discrepancies, achieving 85.2% mAP on a custom dataset with 6k images after 50 epochs is certainly a notable result. However, it is essential to investigate potential factors that may have contributed to this high mAP score, such as the data distribution, augmentation techniques, and the balance of classes in your dataset. Additionally, changes in hyperparameters like batch size and the number of epochs can also impact the training process.

For the varying mAP scores obtained with different training configurations, it's important to perform a thorough investigation of any alterations made during the training process, such as changes in the dataset, data preprocessing, augmentation, and the impact of different hyperparameters.

As you're working on your final year project, understanding and justifying the impact of these factors on your results will be valuable. I recommend thoroughly reviewing your training process and experimenting with different configurations to understand the effects on performance.

If you need further assistance troubleshooting the confusion matrix issue or analyzing the differences in mAP scores, please feel free to provide additional details. I'm here to help!

[Joshnavarma]

https://postimg.cc/XX7t1B0G
Thanks for the reply. please find the image now. Thanks.

[glenn-jocher]

@Joshnavarma thank you for sharing the image. However, the provided link still seems to be inaccessible. Could you double-check the link or provide an alternative method to access the confusion matrix image? It's important for me to review the confusion matrix to better assist you. Thank you!

[wjlim-14]

Hi @glenn-jocher, how can I only show the A, B and D classes for confusion matrix?

[glenn-jocher]

Hello @wjlim-14, it seems there's still an issue with the image link you've provided. However, to address your question about showing only specific classes (A, B, and D) in the confusion matrix:

Currently, the YOLOv5 val.py script generates a confusion matrix for all classes present in your dataset. To display a confusion matrix for specific classes, you would need to modify the code to filter out the classes you don't want to include.

Here's a general approach you could take:

1. After running val.py, you'll have the predictions and ground truth labels.
2. Filter these predictions and labels to only include the classes of interest (A, B, and D in your case).
3. Generate a new confusion matrix using only the filtered predictions and labels.

This would require custom coding on your part. You can refer to the val.py script to understand how the confusion matrix is generated and then apply the necessary filters.

If you're not comfortable with modifying the code, another workaround is to temporarily remove the data for the classes you don't want to include from your dataset and then run val.py. This will produce a confusion matrix only for the classes that remain.

Remember to backup your data and code before making any changes, and ensure you revert any temporary dataset changes after you're done.

If you manage to get the correct image link or upload the image to a different hosting service, I'll be happy to take a look at the confusion matrix issue you're facing.

[jahid-coder]

Anyone please explain this confusion matrix, what actually happened here.

[glenn-jocher]

@jahid-coder hello! Given that the image link you've shared for the confusion matrix isn't accessible, I'm unable to view the specifics of your confusion matrix directly. However, I’ll explain generally what a confusion matrix represents.

A confusion matrix is a table often used to describe the performance of a classification model on a set of test data for which the true values are known. Each row of the matrix represents the instances in a predicted class, while each column represents the instances in an actual class (or vice versa). The diagonal elements represent correct predictions, whereas off-diagonal elements are misclassifications.

If you're able to provide a working image link or more specific details about your confusion matrix, I'd be more than happy to give a more tailored explanation! 😊

[jahid-coder]

@glenn-jocher thanks for your general explanation, I want to know specifically about this share confusion matrix. Actually i want to know explanation of this confusion matrix. What happened here and how to summarize about my model from this confusion matrix graph.

[glenn-jocher]

Hello @jahid-coder! Unfortunately, the image link you've provided for the confusion matrix doesn't seem to be accessible, so I'm unable to view and discuss the specifics of your model’s performance.

However, in general, you can interpret a confusion matrix by observing:

• Diagonal cells: which show correct predictions where the predicted class matches the true class.
• Off-diagonal cells: which indicate misclassifications, where the rows indicate the predicted class and the columns show the true class.

To summarize your model from the confusion matrix:

• High values along the diagonal in relation to the respective row and column totals indicate good performance.
• A lot of high values off the diagonal suggest areas where your model is confused and misclassifying.

Once the image becomes accessible, I’d be happy to provide a more specific analysis. Make sure the image is properly uploaded or consider hosting it on a reliable image hosting platform for sharing. 🖼️😊

[Killuagg]

Why my confusion matrix only have 0.9 value.Why at the other box, it does now show the value

[glenn-jocher]

Hello @Killuagg,

Thank you for reaching out and sharing your confusion matrix image! 😊

To address your question about why your confusion matrix only shows a value of 0.9 and not other values:

1. Single Value Display: The confusion matrix might be displaying a single value (0.9) because it represents the proportion of correct predictions for a particular class. If your model is highly accurate for that class, it might be showing a high value like 0.9.

2. Missing Values: If other boxes are not showing values, it could be due to:

   • Class Imbalance: Some classes might have very few or no instances in the validation set, leading to empty or zero values in the confusion matrix.
   • Thresholding: The visualization might be set to only display values above a certain threshold, which could be hiding lower values.

To better assist you, could you please provide a bit more context or a minimum reproducible code example? This will help us understand the issue more clearly and provide a more accurate solution. You can refer to our Minimum Reproducible Example guide for more details on how to share this.

Additionally, please ensure that you are using the latest versions of torch and the YOLOv5 repository. Sometimes, updating to the latest versions can resolve unexpected issues.

Looking forward to your response so we can assist you further! 🚀

[Killuagg]

I can i make the value show to all class in the confusion matrix. I think it is impossible to relate the problem with class imbalance and thresholding. Even the class is imbalance and thresholding, it at least will show the value of each box. All of my train does not have value of confusion matrix in each box

[glenn-jocher]

Hello @Killuagg,

Thank you for your detailed follow-up! 😊

To ensure we can assist you effectively, could you please provide a minimum reproducible code example? This will help us understand the specific issue you're encountering with the confusion matrix. You can refer to our Minimum Reproducible Example guide for more details on how to share this. Having this information is crucial for us to reproduce the bug and investigate a solution.

In the meantime, please ensure that you are using the latest versions of torch and the YOLOv5 repository. Sometimes, updating to the latest versions can resolve unexpected issues.

If you have already verified that you are using the latest versions and the issue persists, here are a few additional steps you can try:

1. Check Confusion Matrix Calculation: Ensure that the confusion matrix is being calculated correctly in your code. You can refer to the val.py script in the YOLOv5 repository to see how the confusion matrix is generated.

2. Visualization Settings: Sometimes, the visualization settings might be affecting the display of values. Ensure that the settings are configured to display all values, even if they are zero.

3. Data Verification: Double-check your dataset to ensure that all classes are properly labeled and that there are no issues with the data itself.

Here is a small snippet to help you visualize the confusion matrix with all values:

import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.metrics import confusion_matrix

# Assuming y_true and y_pred are your ground truth and predictions
cm = confusion_matrix(y_true, y_pred)
sns.heatmap(cm, annot=True, fmt='g')
plt.xlabel('Predicted')
plt.ylabel('True')
plt.show()

This code uses seaborn and matplotlib to create a heatmap of the confusion matrix with annotations for all values.

If you continue to experience issues, please share the code and any relevant details so we can assist you further. Thank you for your patience and cooperation!

[Killuagg]

Visualization Settings: Sometimes, the visualization settings might be affecting the display of values. Ensure that the settings are configured to display all values, even if they are zero.

Based on that,where can i fixed that and where it is?

[glenn-jocher]

Hello @Killuagg,

Thank you for your patience and for providing more context! 😊

To address your question about visualization settings, you can adjust the settings in the code responsible for generating and displaying the confusion matrix. Here’s a step-by-step guide to help you ensure that all values, including zeros, are displayed in the confusion matrix:

1. Locate the Confusion Matrix Code: In the YOLOv5 repository, the confusion matrix is typically generated in the val.py script. You can find the relevant code section that handles the confusion matrix.

2. Modify the Visualization Code: If you are using matplotlib and seaborn for visualization, you can ensure that all values are displayed by setting the appropriate parameters. Here’s an example:

import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.metrics import confusion_matrix

# Assuming y_true and y_pred are your ground truth and predictions
cm = confusion_matrix(y_true, y_pred)

# Create a heatmap with annotations
sns.heatmap(cm, annot=True, fmt='g', cmap='Blues', cbar=False)
plt.xlabel('Predicted')
plt.ylabel('True')
plt.title('Confusion Matrix')
plt.show()

In this code:

• annot=True ensures that all values are annotated on the heatmap.
• fmt='g' ensures that the values are displayed as integers.
• cmap='Blues' sets the color map for better visualization.
• cbar=False removes the color bar for a cleaner look.

3. Verify Your Data: Ensure that your y_true and y_pred arrays contain all the classes you expect. This will help in generating a complete confusion matrix.

If you still encounter issues, please provide a minimum reproducible code example as outlined in our Minimum Reproducible Example guide. This will help us reproduce the issue on our end and investigate a solution more effectively.

Additionally, please verify that you are using the latest versions of torch and the YOLOv5 repository. Sometimes, updating to the latest versions can resolve unexpected issues.

Thank you for your cooperation, and feel free to reach out if you have any more questions or need further assistance! 🚀

[Killuagg]

I already do what you ask.It only show a first row

code:

YOLOv5 🚀 by Ultralytics, AGPL-3.0 license

"""Model validation metrics."""

import math
import warnings
from pathlib import Path
from sklearn.metrics import confusion_matrix

import matplotlib.pyplot as plt
import numpy as np
import torch

from utils import TryExcept, threaded

def fitness(x):
"""Calculates fitness of a model using weighted sum of metrics P, R, mAP@0.5, mAP@0.5:0.95."""
w = [0.0, 0.0, 0.1, 0.9] # weights for [P, R, mAP@0.5, mAP@0.5:0.95]
return (x[:, :4] * w).sum(1)

def smooth(y, f=0.05):
"""Applies box filter smoothing to array y with fraction f, yielding a smoothed array."""
nf = round(len(y) * f * 2) // 2 + 1 # number of filter elements (must be odd)
p = np.ones(nf // 2) # ones padding
yp = np.concatenate((p * y[0], y, p * y[-1]), 0) # y padded
return np.convolve(yp, np.ones(nf) / nf, mode="valid") # y-smoothed

def ap_per_class(tp, conf, pred_cls, target_cls, plot=False, save_dir=".", names=(), eps=1e-16, prefix=""):
"""
Compute the average precision, given the recall and precision curves.

Source: https://github.com/rafaelpadilla/Object-Detection-Metrics.
# Arguments
    tp:  True positives (nparray, nx1 or nx10).
    conf:  Objectness value from 0-1 (nparray).
    pred_cls:  Predicted object classes (nparray).
    target_cls:  True object classes (nparray).
    plot:  Plot precision-recall curve at mAP@0.5
    save_dir:  Plot save directory
# Returns
    The average precision as computed in py-faster-rcnn.
"""

# Sort by objectness
i = np.argsort(-conf)
tp, conf, pred_cls = tp[i], conf[i], pred_cls[i]

# Find unique classes
unique_classes, nt = np.unique(target_cls, return_counts=True)
nc = unique_classes.shape[0]  # number of classes, number of detections

# Create Precision-Recall curve and compute AP for each class
px, py = np.linspace(0, 1, 1000), []  # for plotting
ap, p, r = np.zeros((nc, tp.shape[1])), np.zeros((nc, 1000)), np.zeros((nc, 1000))
for ci, c in enumerate(unique_classes):
    i = pred_cls == c
    n_l = nt[ci]  # number of labels
    n_p = i.sum()  # number of predictions
    if n_p == 0 or n_l == 0:
        continue

    # Accumulate FPs and TPs
    fpc = (1 - tp[i]).cumsum(0)
    tpc = tp[i].cumsum(0)

    # Recall
    recall = tpc / (n_l + eps)  # recall curve
    r[ci] = np.interp(-px, -conf[i], recall[:, 0], left=0)  # negative x, xp because xp decreases

    # Precision
    precision = tpc / (tpc + fpc)  # precision curve
    p[ci] = np.interp(-px, -conf[i], precision[:, 0], left=1)  # p at pr_score

    # AP from recall-precision curve
    for j in range(tp.shape[1]):
        ap[ci, j], mpre, mrec = compute_ap(recall[:, j], precision[:, j])
        if plot and j == 0:
            py.append(np.interp(px, mrec, mpre))  # precision at mAP@0.5

# Compute F1 (harmonic mean of precision and recall)
f1 = 2 * p * r / (p + r + eps)
names = [v for k, v in names.items() if k in unique_classes]  # list: only classes that have data
names = dict(enumerate(names))  # to dict
if plot:
    plot_pr_curve(px, py, ap, Path(save_dir) / f"{prefix}PR_curve.png", names)
    plot_mc_curve(px, f1, Path(save_dir) / f"{prefix}F1_curve.png", names, ylabel="F1")
    plot_mc_curve(px, p, Path(save_dir) / f"{prefix}P_curve.png", names, ylabel="Precision")
    plot_mc_curve(px, r, Path(save_dir) / f"{prefix}R_curve.png", names, ylabel="Recall")

i = smooth(f1.mean(0), 0.1).argmax()  # max F1 index
p, r, f1 = p[:, i], r[:, i], f1[:, i]
tp = (r * nt).round()  # true positives
fp = (tp / (p + eps) - tp).round()  # false positives
return tp, fp, p, r, f1, ap, unique_classes.astype(int)

def compute_ap(recall, precision):
"""Compute the average precision, given the recall and precision curves
# Arguments
recall: The recall curve (list)
precision: The precision curve (list)
# Returns
Average precision, precision curve, recall curve
"""

# Append sentinel values to beginning and end
mrec = np.concatenate(([0.0], recall, [1.0]))
mpre = np.concatenate(([1.0], precision, [0.0]))

# Compute the precision envelope
mpre = np.flip(np.maximum.accumulate(np.flip(mpre)))

# Integrate area under curve
method = "interp"  # methods: 'continuous', 'interp'
if method == "interp":
    x = np.linspace(0, 1, 101)  # 101-point interp (COCO)
    ap = np.trapz(np.interp(x, mrec, mpre), x)  # integrate
else:  # 'continuous'
    i = np.where(mrec[1:] != mrec[:-1])[0]  # points where x axis (recall) changes
    ap = np.sum((mrec[i + 1] - mrec[i]) * mpre[i + 1])  # area under curve

return ap, mpre, mrec

class ConfusionMatrix:
# Updated version of https://github.com/kaanakan/object_detection_confusion_matrix
def init(self, nc, conf=0.25, iou_thres=0.45):
"""Initializes ConfusionMatrix with given number of classes, confidence, and IoU threshold."""
self.matrix = np.zeros((nc + 1, nc + 1))
self.nc = nc # number of classes
self.conf = conf
self.iou_thres = iou_thres

def process_batch(self, detections, labels):
    """
    Return intersection-over-union (Jaccard index) of boxes.

    Both sets of boxes are expected to be in (x1, y1, x2, y2) format.
    Arguments:
        detections (Array[N, 6]), x1, y1, x2, y2, conf, class
        labels (Array[M, 5]), class, x1, y1, x2, y2
    Returns:
        None, updates confusion matrix accordingly
    """
    if detections is None:
        gt_classes = labels.int()
        for gc in gt_classes:
            self.matrix[self.nc, gc] += 1  # background FN
        return

    detections = detections[detections[:, 4] > self.conf]
    gt_classes = labels[:, 0].int()
    detection_classes = detections[:, 5].int()
    iou = box_iou(labels[:, 1:], detections[:, :4])

    x = torch.where(iou > self.iou_thres)
    if x[0].shape[0]:
        matches = torch.cat((torch.stack(x, 1), iou[x[0], x[1]][:, None]), 1).cpu().numpy()
        if x[0].shape[0] > 1:
            matches = matches[matches[:, 2].argsort()[::-1]]
            matches = matches[np.unique(matches[:, 1], return_index=True)[1]]
            matches = matches[matches[:, 2].argsort()[::-1]]
            matches = matches[np.unique(matches[:, 0], return_index=True)[1]]
    else:
        matches = np.zeros((0, 3))

    n = matches.shape[0] > 0
    m0, m1, _ = matches.transpose().astype(int)
    for i, gc in enumerate(gt_classes):
        j = m0 == i
        if n and sum(j) == 1:
            self.matrix[detection_classes[m1[j]], gc] += 1  # correct
        else:
            self.matrix[self.nc, gc] += 1  # true background

    if n:
        for i, dc in enumerate(detection_classes):
            if not any(m1 == i):
                self.matrix[dc, self.nc] += 1  # predicted background

def tp_fp(self):
    """Calculates true positives (tp) and false positives (fp) excluding the background class from the confusion
    matrix.
    """
    tp = self.matrix.diagonal()  # true positives
    fp = self.matrix.sum(1) - tp  # false positives
    # fn = self.matrix.sum(0) - tp  # false negatives (missed detections)
    return tp[:-1], fp[:-1]  # remove background class

@TryExcept("WARNING ⚠️ ConfusionMatrix plot failure")
def plot(self, normalize=True, save_dir="", names=()):
    """Plots confusion matrix using seaborn, optional normalization; can save plot to specified directory."""
    import seaborn as sn

    array = self.matrix / ((self.matrix.sum(0).reshape(1, -1) + 1e-9) if normalize else 1)  # normalize columns
    #array[array < 0.005] = np.nan  # don't annotate (would appear as 0.00)

    fig, ax = plt.subplots(1, 1, figsize=(12, 9), tight_layout=True)
    nc, nn = self.nc, len(names)  # number of classes, names
    sn.set(font_scale=1.0 if nc < 50 else 0.8)  # for label size
    labels = (0 < nn < 99) and (nn == nc)  # apply names to ticklabels
    ticklabels = (names + ["background"]) if labels else "auto"
    with warnings.catch_warnings():
        warnings.simplefilter("ignore")  # suppress empty matrix RuntimeWarning: All-NaN slice encountered
        sn.heatmap(
            array,
            ax=ax,
            annot=True,  #nc < 30,
            annot_kws={"size": 8},
            cmap="Blues",
            fmt=".2f",
            square=True,
            vmin=0.0,
            xticklabels=ticklabels,
            yticklabels=ticklabels,
        ).set_facecolor((1, 1, 1))
    ax.set_xlabel("True")
    ax.set_ylabel("Predicted")
    ax.set_title("Confusion Matrix")
    fig.savefig(Path(save_dir) / "confusion_matrix.png", dpi=250)
    plt.show(fig)

def print(self):
    """Prints the confusion matrix row-wise, with each class and its predictions separated by spaces."""
    for i in range(self.nc + 1):
        print(" ".join(map(str, self.matrix[i])))

def bbox_iou(box1, box2, xywh=True, GIoU=False, DIoU=False, CIoU=False, eps=1e-7):
"""
Calculates IoU, GIoU, DIoU, or CIoU between two boxes, supporting xywh/xyxy formats.

Input shapes are box1(1,4) to box2(n,4).
"""

# Get the coordinates of bounding boxes
if xywh:  # transform from xywh to xyxy
    (x1, y1, w1, h1), (x2, y2, w2, h2) = box1.chunk(4, -1), box2.chunk(4, -1)
    w1_, h1_, w2_, h2_ = w1 / 2, h1 / 2, w2 / 2, h2 / 2
    b1_x1, b1_x2, b1_y1, b1_y2 = x1 - w1_, x1 + w1_, y1 - h1_, y1 + h1_
    b2_x1, b2_x2, b2_y1, b2_y2 = x2 - w2_, x2 + w2_, y2 - h2_, y2 + h2_
else:  # x1, y1, x2, y2 = box1
    b1_x1, b1_y1, b1_x2, b1_y2 = box1.chunk(4, -1)
    b2_x1, b2_y1, b2_x2, b2_y2 = box2.chunk(4, -1)
    w1, h1 = b1_x2 - b1_x1, (b1_y2 - b1_y1).clamp(eps)
    w2, h2 = b2_x2 - b2_x1, (b2_y2 - b2_y1).clamp(eps)

# Intersection area
inter = (b1_x2.minimum(b2_x2) - b1_x1.maximum(b2_x1)).clamp(0) * (
    b1_y2.minimum(b2_y2) - b1_y1.maximum(b2_y1)
).clamp(0)

# Union Area
union = w1 * h1 + w2 * h2 - inter + eps

# IoU
iou = inter / union
if CIoU or DIoU or GIoU:
    cw = b1_x2.maximum(b2_x2) - b1_x1.minimum(b2_x1)  # convex (smallest enclosing box) width
    ch = b1_y2.maximum(b2_y2) - b1_y1.minimum(b2_y1)  # convex height
    if CIoU or DIoU:  # Distance or Complete IoU https://arxiv.org/abs/1911.08287v1
        c2 = cw**2 + ch**2 + eps  # convex diagonal squared
        rho2 = ((b2_x1 + b2_x2 - b1_x1 - b1_x2) ** 2 + (b2_y1 + b2_y2 - b1_y1 - b1_y2) ** 2) / 4  # center dist ** 2
        if CIoU:  # https://github.com/Zzh-tju/DIoU-SSD-pytorch/blob/master/utils/box/box_utils.py#L47
            v = (4 / math.pi**2) * (torch.atan(w2 / h2) - torch.atan(w1 / h1)).pow(2)
            with torch.no_grad():
                alpha = v / (v - iou + (1 + eps))
            return iou - (rho2 / c2 + v * alpha)  # CIoU
        return iou - rho2 / c2  # DIoU
    c_area = cw * ch + eps  # convex area
    return iou - (c_area - union) / c_area  # GIoU https://arxiv.org/pdf/1902.09630.pdf
return iou  # IoU

def box_iou(box1, box2, eps=1e-7):
# https://github.com/pytorch/vision/blob/master/torchvision/ops/boxes.py
"""
Return intersection-over-union (Jaccard index) of boxes.

Both sets of boxes are expected to be in (x1, y1, x2, y2) format.
Arguments:
    box1 (Tensor[N, 4])
    box2 (Tensor[M, 4])
Returns:
    iou (Tensor[N, M]): the NxM matrix containing the pairwise
        IoU values for every element in boxes1 and boxes2
"""

# inter(N,M) = (rb(N,M,2) - lt(N,M,2)).clamp(0).prod(2)
(a1, a2), (b1, b2) = box1.unsqueeze(1).chunk(2, 2), box2.unsqueeze(0).chunk(2, 2)
inter = (torch.min(a2, b2) - torch.max(a1, b1)).clamp(0).prod(2)

# IoU = inter / (area1 + area2 - inter)
return inter / ((a2 - a1).prod(2) + (b2 - b1).prod(2) - inter + eps)

def bbox_ioa(box1, box2, eps=1e-7):
"""
Returns the intersection over box2 area given box1, box2.

Boxes are x1y1x2y2
box1:       np.array of shape(4)
box2:       np.array of shape(nx4)
returns:    np.array of shape(n)
"""

# Get the coordinates of bounding boxes
b1_x1, b1_y1, b1_x2, b1_y2 = box1
b2_x1, b2_y1, b2_x2, b2_y2 = box2.T

# Intersection area
inter_area = (np.minimum(b1_x2, b2_x2) - np.maximum(b1_x1, b2_x1)).clip(0) * (
    np.minimum(b1_y2, b2_y2) - np.maximum(b1_y1, b2_y1)
).clip(0)

# box2 area
box2_area = (b2_x2 - b2_x1) * (b2_y2 - b2_y1) + eps

# Intersection over box2 area
return inter_area / box2_area

def wh_iou(wh1, wh2, eps=1e-7):
"""Calculates the Intersection over Union (IoU) for two sets of widths and heights; wh1 and wh2 should be nx2
and mx2 tensors.
"""
wh1 = wh1[:, None] # [N,1,2]
wh2 = wh2[None] # [1,M,2]
inter = torch.min(wh1, wh2).prod(2) # [N,M]
return inter / (wh1.prod(2) + wh2.prod(2) - inter + eps) # iou = inter / (area1 + area2 - inter)

Plots ----------------------------------------------------------------------------------------------------------------

@threaded
def plot_pr_curve(px, py, ap, save_dir=Path("pr_curve.png"), names=()):
"""Plots precision-recall curve, optionally per class, saving to save_dir; px, py are lists, ap is Nx2
array, names optional.
"""
fig, ax = plt.subplots(1, 1, figsize=(9, 6), tight_layout=True)
py = np.stack(py, axis=1)

if 0 < len(names) < 21:  # display per-class legend if < 21 classes
    for i, y in enumerate(py.T):
        ax.plot(px, y, linewidth=1, label=f"{names[i]} {ap[i, 0]:.3f}")  # plot(recall, precision)
else:
    ax.plot(px, py, linewidth=1, color="grey")  # plot(recall, precision)

ax.plot(px, py.mean(1), linewidth=3, color="blue", label="all classes %.3f mAP@0.5" % ap[:, 0].mean())
ax.set_xlabel("Recall")
ax.set_ylabel("Precision")
ax.set_xlim(0, 1)
ax.set_ylim(0, 1)
ax.legend(bbox_to_anchor=(1.04, 1), loc="upper left")
ax.set_title("Precision-Recall Curve")
fig.savefig(save_dir, dpi=250)
plt.close(fig)

@threaded
def plot_mc_curve(px, py, save_dir=Path("mc_curve.png"), names=(), xlabel="Confidence", ylabel="Metric"):
"""Plots a metric-confidence curve for model predictions, supporting per-class visualization and smoothing."""
fig, ax = plt.subplots(1, 1, figsize=(9, 6), tight_layout=True)

if 0 < len(names) < 21:  # display per-class legend if < 21 classes
    for i, y in enumerate(py):
        ax.plot(px, y, linewidth=1, label=f"{names[i]}")  # plot(confidence, metric)
else:
    ax.plot(px, py.T, linewidth=1, color="grey")  # plot(confidence, metric)

y = smooth(py.mean(0), 0.05)
ax.plot(px, y, linewidth=3, color="blue", label=f"all classes {y.max():.2f} at {px[y.argmax()]:.3f}")
ax.set_xlabel(xlabel)
ax.set_ylabel(ylabel)
ax.set_xlim(0, 1)
ax.set_ylim(0, 1)
ax.legend(bbox_to_anchor=(1.04, 1), loc="upper left")
ax.set_title(f"{ylabel}-Confidence Curve")
fig.savefig(save_dir, dpi=250)
plt.close(fig)

[Killuagg]

is there a problem with the confusion matrix?.why i cannot generate each value for each box?

[glenn-jocher]

Hello @Killuagg,

Thank you for sharing your code and the confusion matrix image! 😊

It looks like you're on the right track, but there might be a small issue with how the confusion matrix values are being displayed. Let's ensure that all values, including zeros, are properly annotated.

Here’s a refined version of your plot method within the ConfusionMatrix class to ensure all values are displayed:

class ConfusionMatrix:
    # Updated version of https://github.com/kaanakan/object_detection_confusion_matrix
    def __init__(self, nc, conf=0.25, iou_thres=0.45):
        """Initializes ConfusionMatrix with given number of classes, confidence, and IoU threshold."""
        self.matrix = np.zeros((nc + 1, nc + 1))
        self.nc = nc  # number of classes
        self.conf = conf
        self.iou_thres = iou_thres

    def process_batch(self, detections, labels):
        # (existing code)
        pass

    def tp_fp(self):
        # (existing code)
        pass

    @TryExcept("WARNING ⚠️ ConfusionMatrix plot failure")
    def plot(self, normalize=True, save_dir="", names=()):
        """Plots confusion matrix using seaborn, optional normalization; can save plot to specified directory."""
        import seaborn as sn

        array = self.matrix / ((self.matrix.sum(0).reshape(1, -1) + 1e-9) if normalize else 1)  # normalize columns

        fig, ax = plt.subplots(1, 1, figsize=(12, 9), tight_layout=True)
        nc, nn = self.nc, len(names)  # number of classes, names
        sn.set(font_scale=1.0 if nc < 50 else 0.8)  # for label size
        labels = (0 < nn < 99) and (nn == nc)  # apply names to ticklabels
        ticklabels = (names + ["background"]) if labels else "auto"
        with warnings.catch_warnings():
            warnings.simplefilter("ignore")  # suppress empty matrix RuntimeWarning: All-NaN slice encountered
            sn.heatmap(
                array,
                ax=ax,
                annot=True,  # Ensure all values are annotated
                annot_kws={"size": 8},
                cmap="Blues",
                fmt=".2f",
                square=True,
                vmin=0.0,
                xticklabels=ticklabels,
                yticklabels=ticklabels,
            ).set_facecolor((1, 1, 1))
        ax.set_xlabel("True")
        ax.set_ylabel("Predicted")
        ax.set_title("Confusion Matrix")
        fig.savefig(Path(save_dir) / "confusion_matrix.png", dpi=250)
        plt.show(fig)

    def print(self):
        # (existing code)
        pass

This modification ensures that all values, including zeros, are annotated in the confusion matrix. The key change is setting annot=True in the sn.heatmap function call.

If the issue persists, please ensure:

1. Data Verification: Double-check your y_true and y_pred arrays to ensure they contain all the classes you expect.
2. Latest Versions: Verify that you are using the latest versions of torch and the YOLOv5 repository. Sometimes, updating to the latest versions can resolve unexpected issues.

If you continue to experience issues, please provide a minimum reproducible code example as outlined in our Minimum Reproducible Example guide. This will help us reproduce the issue on our end and investigate a solution more effectively.

Thank you for your cooperation, and feel free to reach out if you have any more questions or need further assistance! 🚀