YOLO Detector

Built for the LFX Mentorship 2023 03-Sep-Nov Challenge - #2702 Using CV2 to:

• Load the weights
• Convert the pre-processed image to a blob
• Do a forward pass
• Apply non-maxima suppression to the predictions

Required Packages

apt-get install -y libopencv-dev

pip install -r requirements.txt

Python Functions to be ported to Rust

Preprocessing:

cv2.normalize - To normalize images - potentially exists opencvmini

cv2.resize - To Resize images - to be added to opencv mini ? - Could also be exposed via imageops::resize

cv2.dnn.blobFromImage - convert image to blob for CV2

cv2.dnn.readNet - Loads network structure + pre-trained weights - This would need to accomadate a list of neural network formats (onnx, torch model, tflite ...)

Post Processing:

cv2.dnn.NMSBoxes - potentially exists media-pipe-nms

For drawing directly to input image
cv2.getTextSize

cv2.putText

np.max - gets max value along array

np.argmax - gets indicies of maximum value along an axis

draw_detections(image, boxes, scores, class_ids) - potentially exists media-pipe-draw-detections

rescale_boxes(boxes) - Could become a convenience function once bounding boxes are extracted extract_boxes(predictions) - Could become a convenience function once a forward pass has been made

It potentially makes sense to have a single function to take the output tensor of the forward pass of a YOLO and process it into something that can be consumed by the program / applied directly to the image. Something with a function signature:

fn process_yolo_tensor(tensor:YOLOTensor, conf_threshold: f64, iou_threshold: f64 ) -> HashSet<(BoundingBox,Class)>;

It could be comprised of all the post processing functions above, and the steps taken in the process_output function in the program.

Helper Functions:

cv2.imshow cv2.waitKey cv2.destroyAllWindows We may want to decouple this to the actual Yolo crate, as these functions will require peripherals and some idea of a windowing system. So depending on the design of the YOLO Crate, it may make sense to put these elsewhere.

Closing remarks

As much of this project as possible should leverage existing rust libraries that can target WASM