diff --git a/docs/source/_static/images/tutorials/fov.jpeg b/docs/source/_static/images/tutorials/fov.jpeg deleted file mode 100644 index f70286598..000000000 Binary files a/docs/source/_static/images/tutorials/fov.jpeg and /dev/null differ diff --git a/docs/source/_static/images/tutorials/isp.jpg b/docs/source/_static/images/tutorials/isp.jpg new file mode 100644 index 000000000..3ae088d3b Binary files /dev/null and b/docs/source/_static/images/tutorials/isp.jpg differ diff --git a/docs/source/components/nodes/color_camera.rst b/docs/source/components/nodes/color_camera.rst index ab9c8787d..4f36e7148 100644 --- a/docs/source/components/nodes/color_camera.rst +++ b/docs/source/components/nodes/color_camera.rst @@ -69,6 +69,12 @@ For IMX378 (12MP), the **post-processing** works like this: │ ISP ├────────────────►│ video ├───────────────►│ preview │ └─────┘ max 3840x2160 └─────────┘ and cropping └──────────┘ +.. image:: /_static/images/tutorials/isp.jpg + +The image above is the ``isp`` output frame from the ColorCamera (12MP from IMX378). The blue rectangle represents the cropped 4K +``video`` output, and the yellow rectangle represents a cropped ``preview`` output when the preview size is set to 1:1 aspect ratio +(eg. when using 300x300 MobileNet-SSD NN model). + Usage ##### diff --git a/docs/source/tutorials/dispaying_detections.rst b/docs/source/tutorials/dispaying_detections.rst index 01bd2fa8f..6e1263b1b 100644 --- a/docs/source/tutorials/dispaying_detections.rst +++ b/docs/source/tutorials/dispaying_detections.rst @@ -31,7 +31,7 @@ draw bounding boxes to it. For bounding boxes to match the frame, :code:`preview #################### A problem that we often encounter with models is that their aspect ratio is :code:`1:1`, not eg. :code:`16x9` as our camera resolution. -This means that some of the FOV will be lost. In our :ref:`How to maximize FOV` tutorial we showcased that changing aspect ratio will +This means that some of the FOV will be lost. In our :ref:`Maximizing FOV` tutorial we showcased that changing aspect ratio will preserve the whole aspect ratio of the camera, but it will "squeeze"/"stretch" the frame, as you can see below. `Demo code here `__. diff --git a/docs/source/tutorials/maximize_fov.rst b/docs/source/tutorials/maximize_fov.rst index 29ea5da1d..f6a73e076 100644 --- a/docs/source/tutorials/maximize_fov.rst +++ b/docs/source/tutorials/maximize_fov.rst @@ -1,35 +1,60 @@ -How to maximize FOV -=================== +Maximizing FOV +============== -By default, when you are using :code:`preview` output from :ref:`ColorCamera`, the DepthAI will crop the -frames to get the desired aspect ratio. For example, if you are using Mobilenet-SSD model, you need -:code:`300x300` frames. DepthAI will crop 1080P frames to :code:`1080x1080` and then resize them to :code:`300x300`. -This means you will lose some part of the image. +In this tutorial we will look at how you can use the full FOV of the image sensor. -If you would like to maximize the FOV of the image, you can either: +When you are using ``preview`` output from :ref:`ColorCamera`, DepthAI will crop the +frames by default to get the desired aspect ratio. ``preview`` stream derives from ``video`` stream, which is cropped (16:9 aspect ratio, +max 4k resolution) from the ``isp`` stream, which has the full FOV. -#. Change the aspect ratio (stretch the image) -#. Apply letterboxing to the image +.. image:: /_static/images/tutorials/isp.jpg + +The image above is the ``isp`` output from the :ref:`ColorCamera` (12MP from IMX378). The blue rectangle represents the cropped 4K +``video`` output, and the yellow rectangle represents a cropped ``preview`` output when the preview size is set to 1:1 aspect ratio +(eg. when using 300x300 MobileNet-SSD NN model). + +In other words, you **need to use ISP output** from the :ref:`ColorCamera` **to maximize the image FOV**. A challenge +occures when your NN model expects different aspect ratio (eg. 1:1) compared to isp output (eg. 4:3). Let's say we have +a MobileNet-SSD which requires 300x300 frames (1:1 aspect ratio) - we have a few options: + +#. :ref:`Stretch the ISP frame ` to to the 1:1 aspect ratio of the NN +#. :ref:`Apply letterboxing ` to the ISP frame to get 1:1 aspect ratio frame +#. :ref:`Crop the ISP frame ` to 1:1 aspect ratio and lose some FOV Change aspect ratio ******************* -Use :code:`camRgb.setPreviewKeepAspectRatio(False)`. This means the aspect ratio will not be preserved and the image -will be "stretched". This might be problematic for some off-the-shelf NN models, so model fine-tuning it might be required. -`Usage example here `__. +**Pros: Preserves full FOV. Cons: Due to stretched frames, NNs accuracy might decrease.** -.. image:: https://user-images.githubusercontent.com/18037362/144095838-d082040a-9716-4f8e-90e5-15bcb23115f9.gif - :target: https://youtu.be/8X0IcnkeIf8 +Changing aspect ratio (**stretching**) can be used Use :code:`camRgb.setPreviewKeepAspectRatio(False)`. This means the aspect +ratio will not be preserved and the image will be "stretched". This might be problematic for some off-the-shelf NN models, so some fine-tuning might be required. +`Usage example here `__. + +.. image:: https://user-images.githubusercontent.com/18037362/180607962-e616cdc7-fcad-4bc8-a15f-617b89a2c047.jpg Letterboxing ************ -`Letterboxing `__ the frames. This method will decrease -the size of the image and apply "black bars" above and below the image, so the aspect ratio is preserved. You can +**Pros: Preserves full FOV. Cons: Smaller "frame" means less features might decrease NN accuracy.** + +`Letterboxing `__ approach will apply "black bars" above and below +the image to the full FOV (isp) frames, so the aspect ratio will be preserved. You can achieve this by using :ref:`ImageManip` with :code:`manip.setResizeThumbnail(x,y)` (for Mobilenet :code:`x=300,y=300`). The downside of using this method is that your actual image will be smaller, so some features might not be preserved, which can mean the NN accuracy could decrease. -`Usage example here `__. +`Usage example here `__. + +.. image:: https://user-images.githubusercontent.com/18037362/180607958-0db7fb34-1221-42a1-b889-10d1f9793912.jpg + +Cropping +******** + +**Pros: No NN accuracy decrease. Cons: Frame is cropped, so it's not full FOV.** + +Cropping the full FOV (isp) frames to match the NN aspect ratio can be used to get the best NN accuracy, but this +decreases FOV. +`Usage example here `__. -.. image:: /_static/images/tutorials/fov.jpeg +.. image:: https://user-images.githubusercontent.com/18037362/180607873-6a476ea4-55e0-4557-a93e-a7cadcd80725.jpg +.. include:: /includes/footer-short.rst