Segmentation Sensor

[AmrElsersy]

🎉 New feature

related to #134

Summary

Segmentation Camera Sensor that allow users to use it easily in SDF, and It publishes the sensor data image to ign-transport

Depends on

SDF #592
Rendering #329

Demo

Semantic

Instance

Test it

Will add a tutorial later.

Format

        <sensor name="segmentation_camera" type="segmentation">
          <camera>
            <horizontal_fov>1.047</horizontal_fov>
            <image>
              <width>320</width>
              <height>240</height>
            </image>
            <clip>
              <near>0.1</near>
              <far>100</far>
            </clip>
          </camera>
          <always_on>1</always_on>
          <update_rate>30</update_rate>
          <visualize>true</visualize>
          <topic>camera</topic>
          <colored>true</colored>
          <segmentation_type>semantic</segmentation_type>
        </sensor>

It requires visuals annotating in the sdf via Gazeb #853

Checklist

• Signed all commits for DCO
• Added tests
• Added example and/or tutorial
• Updated documentation (as needed)
• Updated migration guide (as needed)
• codecheck passed (See contributing)
• All tests passed (See test coverage)
• While waiting for a review on your PR, please help review another open pull request to support the maintainers

[AmrElsersy]

The sensor supports now publishing both colored and labels maps together instead of publishing them individually

[adlarkin]

Can we add one more box that is partially hidden? Perhaps it can be poking out of the right of the left-most box. It'd be nice to add test coverage to make sure that for partially hidden objects, we only see the part that is visible in the segmented image

[AmrElsersy]

It is already tested in the rendering::SegmentationCamera

[adlarkin]

Hmm, it seems like there is some overlap/redundancy in testing between ign-sensors and ign-rendering, but I'm not sure what kind of tests should go in which repository for a given sensor and its rendering capabilities... @iche033 would you mind chiming in to help clarify which tests belong to which repository?

[adlarkin]

Hmm, it seems like there is some overlap/redundancy in testing between ign-sensors and ign-rendering, but I'm not sure what kind of tests should go in which repository for a given sensor and its rendering capabilities... @iche033 would you mind chiming in to help clarify which tests belong to which repository?

[adlarkin]

I'm unable to build this PR in a colcon workspace. I believe I am including all of the dependent PRs (gazebosim/sdformat#592, gazebosim/gz-rendering#329, gazebosim/gz-msgs#165), but I see the following error output when building:

Starting >>> ignition-sensors6
[Processing: ignition-sensors6]                                    
--- stderr: ignition-sensors6                                       
../../lib/libignition-sensors6-segmentation_camera.so.6.0.0~pre1: undefined reference to `std::filesystem::__cxx11::path::_M_split_cmpts()'
../../lib/libignition-sensors6-segmentation_camera.so.6.0.0~pre1: undefined reference to `std::filesystem::__cxx11::directory_iterator::directory_iterator(std::filesystem::__cxx11::path const&, std::filesystem::directory_options, std::error_code*)'
../../lib/libignition-sensors6-segmentation_camera.so.6.0.0~pre1: undefined reference to `std::filesystem::__cxx11::directory_iterator::operator++()'
../../lib/libignition-sensors6-segmentation_camera.so.6.0.0~pre1: undefined reference to `std::filesystem::__cxx11::directory_iterator::operator*() const'
collect2: error: ld returned 1 exit status
make[2]: *** [bin/INTEGRATION_segmentation_camera] Error 1
make[1]: *** [test/integration/CMakeFiles/INTEGRATION_segmentation_camera.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
make: *** [all] Error 2
---
Failed   <<< ignition-sensors6 [42.8s, exited with code 2]

I am also seeing similar issues in #136. It looks like it's only an issue with the integration tests, which makes me think that this behavior started in ed5c136, after merging in the changes from #90. I was able to go back to commit e2c5b52 (the commit before the merge mentioned above) and build this PR just fine.

[iche033]

I'm unable to build this PR in a colcon workspace

std::filesystem::directory_iterator was added in this commit 21b5d57

How about switching to use DirIter from ign-common? Example usage here

[adlarkin]

LGTM! We'll need to wait for gazebosim/gz-rendering#419 to be merged before merging this, since the label map behavior and integration test fails without it

[chapulina]

I think we should leave the tutorials without numbers on their filenames. That doesn't scale well

[adlarkin]

I think we should leave the tutorials without numbers on their filenames

So, instead of 05_segmentation_camera.md, just segmentation_camera.md?

[chapulina]

So, instead of 05_segmentation_camera.md, just segmentation_camera.md?

Yup 👍 I see some other files being moved too. That can be reverted

[adlarkin]

Okay, all numbers have been removed from the tutorial files in 8642b60. We will probably need to make a release of ign-rendering before merging this, since gazebosim/gz-rendering#419 was just merged. CI on this PR will be red until an ign-rendering release is made, since gazebosim/gz-rendering#419 fixes the label map behavior and integration test failures.

Feedback has been addressed

[codecov]

Codecov Report

Merging #133 (8642b60) into main (e99a8bc) will decrease coverage by 2.37%.
The diff coverage is 52.80%.

@@            Coverage Diff             @@
##             main     #133      +/-   ##
==========================================
- Coverage   77.90%   75.52%   -2.38%     
==========================================
  Files          24       25       +1     
  Lines        2394     2644     +250     
==========================================
+ Hits         1865     1997     +132     
- Misses        529      647     +118     

Impacted Files	Coverage Δ
src/SegmentationCameraSensor.cc	52.80% <52.80%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e99a8bc...8642b60. Read the comment docs.