Object Detection Label Errors Argument Shape Difficulty

[Burhan-Q]

I posted in the cleanlab community Slack #help channel a few days ago, but figured I should also open an issue since I haven't seen any replies. I'm working with an object detection dataset and facing some challenges with constructing the inputs for cleanlab.object_detection.filter.find_label_issues.

My understanding from the docs regarding the inputs, but maybe I have this wrong:

• ground truth labels, are in a list of dictionaries list[dict] with keys ["bboxes", "labels"] both of which are Numpy arrays
• predictions input is a list[np.ndarray] with (D, K, 1, 5) dimensions (D is detections per image, K is the number of classes, 1 entry per class, [xmin, ymin, xmax, ymax, conf])

Example data (368 classes):

gt = [
    {
        'bboxes': 
            array([[0.055833, 0.318125, 0.916667, 0.964375],
            [0.204167, 0.325   , 0.771667, 0.98    ],
            [0.220833, 0.38625 , 0.776667, 0.91875 ],
            [0.      , 0.430625, 0.999167, 0.999375]]),
        'labels': 
            array([[ 91],
            [226],
            [ -1],
            [ 69]])
    }
]
det = np.array(
    [ # [class index, xmin, ymin, xmax, ymax, conf]
        [2.26000000e+02, 2.18217492e-01, 3.70962501e-01, 7.79024482e-01, 9.99999523e-01, 6.29037023e-01],
        [6.90000000e+01, 7.95739472e-01, 6.41192496e-01, 9.99720514e-01, 9.84477460e-01, 3.43284994e-01],
        [9.10000000e+01, 2.08500028e-03, 4.19329971e-01, 6.62768960e-01, 9.96745944e-01, 5.77416003e-01]
    ],
)

# code to translate "det" into "preds" with shape (3, 368, 1, 5)

result = cleanlab.object_detection.filter.find_label_issues(gt, [preds])

From the example/tutorial, it looks like the array should be (3, 368, 0, 5) but I'm a bit confused by the $0$ interim dimension (maybe that's just uncommon to me). I've attempted several different ways to manipulate preds to be compatible, but I either end up with an IndexError or an AttributeError. I haven't had the time/opportunity to delve into or debug the source code, but I'm hoping that maybe I'm missing something obvious that someone can help point out. Much appreciated

[elisno]

Thanks for the question, @Burhan-Q! I'll admit that the formatting can take some time getting used to. There are a few things to address here:

Format of given labels

You're correct about the format for ground truth labels. They should be a list[dict] where each dictionary has keys "bboxes" and "labels", both containing NumPy arrays. A small correction is needed in your example:

1. The array for the "labels" key should be 1-dimensional, not 2-dimensional, as in your example.
2. It's unclear what the -1 label represents in your example. If it's meant to indicate a "no object" or background class, you might want to handle this differently depending on your use case.

The format should look something like this:

gt = [
    {
        'bboxes': np.array([
            [0.055833, 0.318125, 0.916667, 0.964375],
            [0.204167, 0.325   , 0.771667, 0.98    ],
            [0.      , 0.430625, 0.999167, 0.999375]
        ]),
        'labels': np.array([91, 226, 69])  # Note: 1D array, removed the -1 label
    }
]

Predictions Format

The format for predictions a bit trickier. It should be a list[np.ndarray], where:

• The list represents different images
• Each element of the list is a NumPy array of arrays (nested arrays)
• This array has K elements (one for each class in the dataset)
• Each of these K elements is itself a NumPy array of shape (M_K, 5), where:
  • M_K is the number of detections for that class K. It can be 0, i.e. an empty array with no predictions for that class.
  • The 5 columns represent your [xmin, ymin, xmax, ymax, conf]

It's important to note that we're dealing with arrays of arrays, not 3D arrays. Some elements in the top-level array may be empty arrays of shape (0, 5) for classes with no detections.

Here's how you can format your det array into the correct preds format for a single image:

# Number of classes
K = 368

# Create a single empty array to be shared
no_pred = np.empty((0, 5))

# Create an array where all entries initially point to the same empty, no-detection array
preds = np.array([no_pred] * K)

# Your example detection array
det = np.array(
    [
        [2.26000000e+02, 2.18217492e-01, 3.70962501e-01, 7.79024482e-01, 9.99999523e-01, 6.29037023e-01],
        [6.90000000e+01, 7.95739472e-01, 6.41192496e-01, 9.99720514e-01, 9.84477460e-01, 3.43284994e-01],
        [9.10000000e+01, 2.08500028e-03, 4.19329971e-01, 6.62768960e-01, 9.96745944e-01, 5.77416003e-01]
    ]
)

for row in det:
    class_index = int(row[0])  # Get the class index
    bbox_and_conf = row[1:]  # Get the rest of the row (bbox and pred_prob)

    # Check if the class index points to the shared empty array
    if preds[class_index] is no_pred:
        # Replace with a new array if this is the first detection for this class
        preds[class_index] = np.array([bbox_and_conf])
    else:
        # Stack the new detection onto the current array at the class index
        preds[class_index] = np.vstack((preds[class_index], bbox_and_conf))

Using find_label_issues

Now you can use the function like this:

result = find_label_issues(gt, [preds])

The [preds] is wrapped in a list because the function expects a list of prediction arrays, one for each image. In your case, a list with a single element.

Further notes

1. The shape (D, K, 1, 5) you mentioned isn't correct for this use case. The correct structure is an array of K arrays, each with shape (M_K, 5) as explained above.
2. The 0 dimension you saw in some examples represents a class with no detections for the given image.

Full example

import numpy as np
from cleanlab.object_detection.filter import find_label_issues

gt = [
    {
        'bboxes': np.array([
            [0.055833, 0.318125, 0.916667, 0.964375],
            [0.204167, 0.325   , 0.771667, 0.98    ],
            [0.      , 0.430625, 0.999167, 0.999375]
        ]),
        'labels': np.array([91, 226, 69])  # Note: 1D array, removed the -1 label
    }
]

# Number of classes
K = 368

# Create a single empty array to be shared
no_pred = np.empty((0, 5))

# Create an array where all entries initially point to the same empty, no-detection array
preds = np.array([no_pred] * K)

# Your example detection array
det = np.array(
    [
        [2.26000000e+02, 2.18217492e-01, 3.70962501e-01, 7.79024482e-01, 9.99999523e-01, 6.29037023e-01],
        [6.90000000e+01, 7.95739472e-01, 6.41192496e-01, 9.99720514e-01, 9.84477460e-01, 3.43284994e-01],
        [9.10000000e+01, 2.08500028e-03, 4.19329971e-01, 6.62768960e-01, 9.96745944e-01, 5.77416003e-01]
    ]
)

for row in det:
    class_index = int(row[0])  # Get the class index
    bbox_and_conf = row[1:]  # Get the rest of the row (bbox and pred_prob)

    # Check if the class index points to the shared empty array
    if preds[class_index] is no_pred:
        # Replace with a new array if this is the first detection for this class
        preds[class_index] = np.array([bbox_and_conf])
    else:
        # Stack the new detection onto the current array at the class index
        preds[class_index] = np.vstack((preds[class_index], bbox_and_conf))

result = find_label_issues(gt, [preds])
result
# array([False])

[Burhan-Q]

@elisno Thank you for the reply and example(s).

It's unclear what the -1 label represents in your example

This was a placeholder and I didn't mean to include it, but since it was included I figure I might as well ask. Suppose that I have a list of ground truth labels that has 750 classes, with some classes being hierarchical classes (like animal where mammal and cat are also included). In this scenario, I want to keep the "best" box from a prediction or ground truth, but only use the most specific label (in this case cat). I guessed that this would be something I would later have to address with my own logic, but am curious if there's a way in cleanlab to handle this.

---

One thing I'm seeing that might be a compatibility issue is with respect to your example for setting the value of the Numpy array. I tested your example on Windows using Python 3.11.6, numpy==1.26.4, and cleanlab==2.6.6 (not that it is used at this point).

    # Check if the class index points to the shared empty array
    if preds[class_index] is no_pred:
        # Replace with a new array if this is the first detection for this class
        preds[class_index] = np.array([bbox_and_conf])
    else:
        # Stack the new detection onto the current array at the class index
        preds[class_index] = np.vstack((preds[class_index], bbox_and_conf))

1. The conditional if preds[class_index] is no_pred evaluates as False for me when running this example (even on the very first assignment).
2. Next, the line(s) for assignment preds[class_index] = np.array([bbox_and_conf]) or preds[class_index] = np.vstack((preds[class_index], bbox_and_conf)) both do not populate any data into the array when running this example.

[Burhan-Q]

If my understanding of your explanation is correct, the array would likely need to be constructed using something like:

K = 368  # Number of classes

# example detection array
det = np.array(
    [
        [2.26000000e+02, 2.18217492e-01, 3.70962501e-01, 7.79024482e-01, 9.99999523e-01, 6.29037023e-01],
        [6.90000000e+01, 7.95739472e-01, 6.41192496e-01, 9.99720514e-01, 9.84477460e-01, 3.43284994e-01],
        [9.10000000e+01, 2.08500028e-03, 4.19329971e-01, 6.62768960e-01, 9.96745944e-01, 5.77416003e-01]
    ]
)
det_counts = np.bincount(det[:, 0].astype(int), minlength=K)
preds = [np.zeros((dc, 5)) for dc in det_counts]
classes = np.unique(det[:, 0]).astype(int)

# NOTE not possible to make `preds` into an array as-is AFAIK, as the dimensions will be mismatched, 
# would need to make all sub-arrays the same size (using max detected class count)

for _cls in classes: # iterate classes for faster array construction
    ...

This would mean that the final result (if not using the max detected class count method) for preds would be a list[list[np.ndarray]]. The first list would be for each image, each nested list would be for detections, where each class detection would be a Numpy array of varying dimensionality.

The only "natural" way that I could see that a list[np.ndarray[np.ndarray]] could be constructed, would be if there was a single detection per-class for each image. This is unlikely to occur, since in many cases there are multiples of the same object detected in an image.

I'll share that this is not my first attempt to work with cleanlab and although it was quite some time ago, I know that multi-class object detection data was feasible to use with find_label_issues. When I did this previously, it was much less complex to process the data as well (I think I was only 5 months into working with Python at the time).

Maybe there's a critical factor I'm missing here about how to construct the Numpy arrays, or that the Numpy version I'm using isn't capable of these operations? I'll try to flesh out my example and test it out to see if it works and post the result. Appreciate anyone pointing out something I missed or where I might be wrong in my understanding/conclusion.

[Burhan-Q]

Well, my conclusion was incorrect and list[list[np.ndarray]] doesn't work.

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    is_issue = _find_label_issues(
               ^^^^^^^^^^^^^^^^^^^
  File ".venv/Lib/site-packages/cleanlab/object_detection/filter.py", line 138, in _find_label_issues
    auxiliary_inputs = _get_valid_inputs_for_compute_scores(ALPHA, labels, predictions)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/Lib/site-packages/cleanlab/object_detection/rank.py", line 512, in _get_valid_inputs_for_compute_scores
    min_possible_similarity = _get_min_possible_similarity(alpha, predictions, labels)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/Lib/site-packages/cleanlab/object_detection/rank.py", line 432, in _get_min_possible_similarity
    pred_bboxes, pred_labels, _ = _separate_prediction(prediction)
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/Lib/site-packages/cleanlab/object_detection/rank.py", line 312, in _separate_prediction
    boxes, labels, pred_probs = _separate_prediction_single_box(prediction)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/Lib/site-packages/cleanlab/object_detection/rank.py", line 288, in _separate_prediction_single_box
    bboxes = [box[:4] for box in boxes]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/Lib/site-packages/cleanlab/object_detection/rank.py", line 288, in <listcomp>
    bboxes = [box[:4] for box in boxes]
              ~~~^^^^
TypeError: 'float' object is not subscriptable

[n-patricia]

@elisno

Hi, I just wanna to confirm, should the prediction have the same number of objects with ground truth? If the predictions return more predicted objects (false negative) then we need to remove them? Because I get this error message
*** ValueError: labels and predictions length needs to match

If I want to use cleanlab to analyse false negative and false positive objects, how to implement it with cleanlab?

Best,
Novi

[aditya1503]

@elisno

Hi, I just wanna to confirm, should the prediction have the same number of objects with ground truth? If the predictions return more predicted objects (false negative) then we need to remove them? Because I get this error message *** ValueError: labels and predictions length needs to match

If I want to use cleanlab to analyse false negative and false positive objects, how to implement it with cleanlab?

Best, Novi

@elisno

Hi, I just wanna to confirm, should the prediction have the same number of objects with ground truth? If the predictions return more predicted objects (false negative) then we need to remove them? Because I get this error message *** ValueError: labels and predictions length needs to match

If I want to use cleanlab to analyse false negative and false positive objects, how to implement it with cleanlab?

Best, Novi

Hi Novi : predictions do not need to have the same number of objects as ground truth, and you should not remove extra predictions. The error only means that the number of images in labels and predictions must match, not the number of boxes. labels[I] and predictions[I] refer to image indices.