YOLO Conformal Prediction

[alyasltd]

Hi ! i wanted to apply CP to object detection using a yolo finetuned model as a predictor, from what i understood i need to wrapped my predictor (so a class containing a predict method that's doing inference of the model).
The thing is that :
`# Perform conformal prediction
conformal_predictor = SplitBoxWise(predictor=wrapped_predictor, method="multiplicative", train=False)

conformal_predictor.fit(
X_fit=X_fit,
y_fit=y_fit,
X_calib=X_calib,
y_calib=y_calib
)

alpha = 0.2
y_pred, box_inner, box_outer = conformal_predictor.predict(X_test=image_paths, alpha=alpha)`

gives this error : ValueError: X_fit and X_calib must contain the same number of features. (but in your example here, i ran it and X_fit & X_calib lenght is different)

In my case X contains images paths of my database, and Y a padded list of bounded boxes as a numpy array, if somebody understand please tell me ! have a nice day :)

[M-Mouhcine]

Hi @alyasltd,

I agree, the error message is not very clear in this case. Let me explain why your code isn't working and provide a way to make it functional.

• First, the predictor model must output only the bounding box (bbox) coordinates in the specific format $[x1, y1, x2, y2]$, where $(x1, y1)$ represents the top-left corner and $(x2, y2)$ represents the bottom-right corner of the predicted bounding box. However, YOLO typically outputs additional information, including an objectness score and class probabilities. Therefore, you need to process YOLO's outputs to extract only the bbox coordinates in the required format.

• Second, for each image in the calibration set, YOLO may predict multiple bounding boxes $[bbox_1, bbox_2, ..., bbox_i]$ if several objects are detected. Each of these predicted bounding boxes must be mapped to a true bounding box to compute the nonconformity score (i.e., the discrepancy between the predicted and true coordinates):
  $bbox_1 \rightarrow bbox^{true}_1, \dots, bbox_i \rightarrow bbox^{true}_i$.
  Assigning each predicted box to a true box is a non-trivial but essential step for calibration. This process is achieved using the Hungarian matching algorithm, which relies on the Intersection over Union (IoU) metric. This approach ensures that only true positive bounding boxes are retained for calibration. During inference, the probabilistic guarantees focus on ensuring that true positives cover the ground truth. However, false negatives (missed detections) may still occur and are not addressed by this calibration. A theory overview of conformal object detection is provided in the theory overview page.

For simplicity, we crafted the toy example you mentionned in a way that skips the step of matching true and predicted bounding boxes. This is not the case in real-world scenarios. Therefore, you need to process the outputs of your YOLO model and use a matching algorithm to link predictions with ground truth. A full example is provided in our Conformal Object Detection Tutorial. In this tutorial, you’ll find a wrapper class, ObjectDetectionAPI, for a DETR model from Hugging Face, which includes all necessary processing steps (compiled in ObjectDetectionAPI.query).

Finally, due to the processing overhead introduced by the object prediction wrapper, we recommend using an IdPredictor instead of a BasePredictor. The IdPredictor allows you to directly provide your predictions ($y_{pred}$) when calling the fit method on the conformal predictor:

# Code snippet from the COD tutorial

from deel.puncc.api.prediction import IdPredictor
from deel.puncc.object_detection import SplitBoxWise

# Create the proxy for the object detection model
api_model = IdPredictor()

# Get matching true and predicted bbox coordinates from calibration data through the wrapper object
y_calib_api, y_calib_coco, _, _ = object_detection_api.query(calib_dataset, n_instances=150)

# Instantiate the conformal predictor
conformal_predictor = SplitBoxWise(api_model, method="multiplicative", train=False)

# Fit the conformal predictor, directly provide the predicted bboxes rather than image inputs
conformal_predictor.fit(X_calib=y_calib_api, y_calib=y_calib_coco)

# Inference
...

All the steps are detailed in the tutorial. Let me know if you need further clarification.

[alyasltd]

Hi ! sorry for the late respond, thank you so much for your very clear answer ! I just implemented the solution, I did a class yolowrapperapi, to wrapp the model as you suggested and then used the idpredictor ! Here are the results ;)

[M-Mouhcine]

Hi @alyasltd. Glad it worked for you :) !