Future scope of 3D Modeling in KerasCV.

[innat]

In coming days, keras_cv might want to support 3D modeling of the general computer vision task, such as 3D classification, 3D object detection, 3D semantic segmentation. This type of modeling is very crucial for robotics and self-driving technology. I read a nice google-ai-blog regarding TensorFlow 3D, we might want to take a look into this in near future.

• TensorFlow 3D

The TensorFlow 3D codebase is an open source framework built on top of TensorFlow 2 and Keras that makes it easy to construct, train and deploy 3D Object Detection, 3D Semantic Segmentation and 3D Instance Segmentation models.

[bhack]

@tanzhenyu as we have recently been introducing many custom c++ ops for 3d (CPU only), have we evaluated the approach adopted by https://github.com/google-research/visu3d?

It is based on:
https://github.com/google-research/dataclass_array

See some nice Colabs at:
https://github.com/google-research/visu3d#documentation

/cc @Conchylicultor

[bhack]

@tanzhenyu What do you think?

[tanzhenyu]

I think TF3D is a good value add, and it'd be really nice if they can contribute to KerasCV.
visu3d is somewhat different, it's a hybrid framework that supports TF/PT/Jax

@aaroey WDYT?

[bhack]

@tanzhenyu What I meant about visu3d is its approach to 3d without custom c++ ops.

[tanzhenyu]

@tanzhenyu What I meant about visu3d is its approach to 3d without custom c++ ops.

Oh, can you help summarize the relationship between dataclass and custom op? I'm not sure I followed the argument

[bhack]

If you see they have already some 3d transformation defined (also with a nice visualizzation) :

https://colab.research.google.com/github/google-research/visu3d/blob/main/docs/transform.ipynb

I guess if this approach could be extend to some of our 3d ops that introduced c++ custom code:

https://github.com/keras-team/keras-cv/tree/master/keras_cv/custom_ops/kernels

[shalindesigns2023]

With the power of deep learning and computer vision, KerasCV has the potential to revolutionize 3D modeling.

In particular, KerasCV's compatibility with TensorFlow and PyTorch opens up exciting possibilities for advanced 3D modeling techniques. This can lead to more accurate object recognition, 3D reconstruction, and point cloud analysis.

Moreover, KerasCV's user-friendly interface democratizes 3D modeling, making it accessible to beginners and experts alike. This fosters collaboration and innovation in the field.

While there's room for further research and development, the future of 3D modeling in KerasCV looks promising. Let's share our thoughts on this exciting topic!