YOLOV8 port to keras-hub

[oarriaga]

This PR ports YOLOV8 from keras-cv to keras-hub (#176). All necessary YOLOV8 functions are now found inside keras-hub:

• Add CIOU loss.
• Add missing masking functionality in the bounding_boxes module.
• Add multibackend non maximum supression layer.
• Add label encoder.
• Build basic abstract object detector task class.
• Add YOLOV8 backbone and detector.

Missing steps include:

• Upload previous presets to Kaggle.
• Remove skipping tests with presets.
• Add colab with basic functionality.
• Add weight transfer script from keras-cv to keras-hub.
• Add training script.

[divyashreepathihalli]

Thanks for the PR @oarriaga left some initial comments

[fchollet]

Thanks for the updates! What's the current progress of the PR? Is it nearly ready to merge?

[fchollet]

Where is to_dense actually located?

[oarriaga]

In keras-cv to_dense was located outside of the training loop when building a tf.data pipeline.

[divyashreepathihalli]

the bounding box code is now moved to keras repo

[divyashreepathihalli]

anything that is bbox relates should be in Keras and if this models needs something new specifically just add it in the model folder

[mattdangerw]

I think we need a plan here. Bounding box support is coming into core Keras.

Why does this need to be public? Where will it be called by a user? How can we use the new Keras bounding box functionality?

We can add utilities to KerasHub. We can add layers (non-max, ciou loss) if we think they are generally useful outside of yolov8 and written in a way that could be usable by other models. We can even do temp things like add a symbol to KerasHub we are going to add to core Keras.

But we should have a clear picture of where we want everything to end up. And like the expected usage we are putting out. Will bounding_box.to_dense(bounding_boxes) even make sense with the core Keras way of handling bounding boxes?

[oarriaga]

Hi, thank you! I think it’s nearly ready. Right now, I’m validating the model’s expected convergence with PASCAL, which has turned out to be more challenging than anticipated. After that, the only remaining step will be to go through your comments and incorporate any additional input from Divya.

[divyashreepathihalli]

did another pass. Can you add numerics verification colab in the PR description?

[oarriaga]

Hi @divyashreepathihalli:

• Here is the link to a notebook with the optimization including the visualization of the results.
• Here is the link to a notebook with the automatic preset download, forward pass, and visualization of results.
• Here is the link to a notebook comparing the inference time between this implementation and the original one.

[ctorney]

Hi @oarriaga the task alignment score is using CIoU here
but I think should be using IoU - see eqn. 9 of https://arxiv.org/abs/2108.07755 or clamp the CIoU to 0 as they seem to do here

I raised this issue keras-team/keras-cv#2253 a while ago as negative ciou scores were resulting in strange assignments

[parthsdoshi]

I'm confused, the marketing for Keras 3 here seems to mention YOLOv8 is already present? Am I missing something?

Edit: I guess the marketing states in both keras-cv + KerasHub

[Lundez]

@parthsdoshi it works in inference but not training because of the RaggedArray requirements. keras_cv has a butt-load of requirement on tensorflow. Based on experience playing around with it.

[oarriaga]

@parthsdoshi @Lundez both KerasCV and KerasHub are available for training and inference. The previous shared notebooks provided an example in the VOC dataset.

[Lundez]

@oarriaga your example is using TensorFlow, could you please share how I do training with PyTorch backend on the keras_cv model?

[oarriaga]

All the internal components of the model are framework independent. However, the training pipeline does use tf.data. One should be able to modify the training pipeline with a PyDataset, or a custom python generator to manage ragged tensors accordingly.

[Lundez]

What I found with my custom dataset design (PyDataset) is that keras_cv required tf tensors for some bbox functionality and if I recall also something related to loss function.

I'll try again, but spent a lot of time on this trying to get pytorch backend working during training. Without success