New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using a different image size for linemod detection for KinectV2 #28

Open
JimmyDaSilva opened this Issue Mar 22, 2016 · 13 comments

Comments

Projects
None yet
6 participants
@JimmyDaSilva

JimmyDaSilva commented Mar 22, 2016

Linemod in opencv is using a linearMemoryPyramid with levels {5,8} by default:
https://github.com/Itseez/opencv_contrib/blob/master/modules/rgbd/src/linemod.cpp#L1834:L1840

5 and 8 work fine for 640x480 images, but fails with every image sizes provided by the KinectV2 (512x424, 960x540, 1920x1080).
The problem has been discussed there:
opencv/opencv#4593

So I am trying to not use getDefaultLINEMOD but create my own detector using these lines in linemod_detect.cpp:

static const int T_LVLS[] = {3, 4};
std::vector< cv::Ptr<cv::linemod::Modality> > modalities;
modalities.push_back(new cv::linemod::ColorGradient());
modalities.push_back(new cv::linemod::DepthNormal());
detector_ = new cv::linemod::Detector(modalities, std::vector<int>(T_LVLS, T_LVLS +2));

instead of
detector_ = getDefaultLINEMOD()

For some image sizes the assertion then passes and the code runs. Unfortunately no objects are found...

@vrabaud I guess you know a bit more about the image processing behind. Do think this is something that actually work? Which values for T would you pick? How many pyramid levels do you think are necessary ?

Thanks for the help !
Jimmy

@nlyubova

This comment has been minimized.

Show comment
Hide comment
@nlyubova

nlyubova Mar 24, 2016

Contributor

Hi,

I've done it once but it is not completely adapted for different images scales, check it in ecto_image_pipeline

nlyubova/ecto_image_pipeline@e855706

Contributor

nlyubova commented Mar 24, 2016

Hi,

I've done it once but it is not completely adapted for different images scales, check it in ecto_image_pipeline

nlyubova/ecto_image_pipeline@e855706

@nlyubova

This comment has been minimized.

Show comment
Hide comment
@nlyubova

nlyubova Mar 24, 2016

Contributor

Regarding "For some image sizes the assertion then passes and the code runs. Unfortunately no objects are found..."

there can be another problem. You could try to train with the same image size that you will use for recognition (it worked for me), for example set in the training config file ;
renderer_width: 320
renderer_height: 240

Contributor

nlyubova commented Mar 24, 2016

Regarding "For some image sizes the assertion then passes and the code runs. Unfortunately no objects are found..."

there can be another problem. You could try to train with the same image size that you will use for recognition (it worked for me), for example set in the training config file ;
renderer_width: 320
renderer_height: 240

@JimmyDaSilva

This comment has been minimized.

Show comment
Hide comment
@JimmyDaSilva

JimmyDaSilva Mar 24, 2016

I got some positive results yesterday. I will come back on this later. But thanks for the help ! :)

JimmyDaSilva commented Mar 24, 2016

I got some positive results yesterday. I will come back on this later. But thanks for the help ! :)

@nlyubova

This comment has been minimized.

Show comment
Hide comment
@nlyubova

nlyubova Mar 24, 2016

Contributor

Did you try to change the threshold?

Contributor

nlyubova commented Mar 24, 2016

Did you try to change the threshold?

@nlyubova

This comment has been minimized.

Show comment
Hide comment
@nlyubova

nlyubova Mar 24, 2016

Contributor

Setting images size in training config definitely helps, especially if training and test images are very different in size (that was twice in my case)

Contributor

nlyubova commented Mar 24, 2016

Setting images size in training config definitely helps, especially if training and test images are very different in size (that was twice in my case)

@JimmyDaSilva

This comment has been minimized.

Show comment
Hide comment
@JimmyDaSilva

JimmyDaSilva Mar 24, 2016

@nlyubova Don't worry more. It works great :)
You were absolutely right about adding the renderer_width and renderer_height in the training script. It changes everything!
I am now playing with the T_LVLS values in my code above for linemod_detect.cpp to get the best and fastest results.
I will post a new PR once I am done.

I didn't have to add the renderer_width and renderer_height for the detection script... so I don't really understand how this works.
But anyway it seems to work now !

JimmyDaSilva commented Mar 24, 2016

@nlyubova Don't worry more. It works great :)
You were absolutely right about adding the renderer_width and renderer_height in the training script. It changes everything!
I am now playing with the T_LVLS values in my code above for linemod_detect.cpp to get the best and fastest results.
I will post a new PR once I am done.

I didn't have to add the renderer_width and renderer_height for the detection script... so I don't really understand how this works.
But anyway it seems to work now !

@JimmyDaSilva

This comment has been minimized.

Show comment
Hide comment
@JimmyDaSilva

JimmyDaSilva Mar 24, 2016

Hooray ! Linemod on Kinect2 is coming for ORK :)
linemod_kinect2

JimmyDaSilva commented Mar 24, 2016

Hooray ! Linemod on Kinect2 is coming for ORK :)
linemod_kinect2

@nlyubova

This comment has been minimized.

Show comment
Hide comment
@nlyubova

nlyubova Mar 24, 2016

Contributor

wow! good job! so what id you change?

Contributor

nlyubova commented Mar 24, 2016

wow! good job! so what id you change?

@JimmyDaSilva

This comment has been minimized.

Show comment
Hide comment
@JimmyDaSilva

JimmyDaSilva Mar 24, 2016

In linemod_detect.cpp:

// detector_ = cv::linemod::getDefaultLINEMOD();
static const int T_LVLS[] = {4, 15};
std::vector< cv::Ptr<cv::linemod::Modality> > modalities;
modalities.push_back(new cv::linemod::ColorGradient());
modalities.push_back(new cv::linemod::DepthNormal());
detector_ = new cv::linemod::Detector(modalities, std::vector<int>(T_LVLS, T_LVLS +2));

And as you advised I have added in training.ork:

renderer_width: 960
renderer_height: 540

These params are for QHD images only.
For SD, I set T={2,4} renderer_width: 512 renderer_height: 424
Detection for HD takes for ever and never finishes.

JimmyDaSilva commented Mar 24, 2016

In linemod_detect.cpp:

// detector_ = cv::linemod::getDefaultLINEMOD();
static const int T_LVLS[] = {4, 15};
std::vector< cv::Ptr<cv::linemod::Modality> > modalities;
modalities.push_back(new cv::linemod::ColorGradient());
modalities.push_back(new cv::linemod::DepthNormal());
detector_ = new cv::linemod::Detector(modalities, std::vector<int>(T_LVLS, T_LVLS +2));

And as you advised I have added in training.ork:

renderer_width: 960
renderer_height: 540

These params are for QHD images only.
For SD, I set T={2,4} renderer_width: 512 renderer_height: 424
Detection for HD takes for ever and never finishes.

@giacomodabisias

This comment has been minimized.

Show comment
Hide comment
@giacomodabisias

giacomodabisias Mar 30, 2016

Good Job.
I still believe that the code should be fixed in such a way that you can easily insert size and data type in order to use it with any depth camera. I will have a look soon and see if I can contribute.

giacomodabisias commented Mar 30, 2016

Good Job.
I still believe that the code should be fixed in such a way that you can easily insert size and data type in order to use it with any depth camera. I will have a look soon and see if I can contribute.

@mikearmstrong800

This comment has been minimized.

Show comment
Hide comment
@mikearmstrong800

mikearmstrong800 Apr 12, 2017

Thanks Jimmy and Nlyubova!

Thanks to you I get great linemod detection of the can with kinect2_bridge on ROS Kinetic on Ubuntu 16.04.

Mike

mikearmstrong800 commented Apr 12, 2017

Thanks Jimmy and Nlyubova!

Thanks to you I get great linemod detection of the can with kinect2_bridge on ROS Kinetic on Ubuntu 16.04.

Mike

@wmy101

This comment has been minimized.

Show comment
Hide comment
@wmy101

wmy101 May 3, 2017

Hi, I have tried the modifications to the linemod and training files but am still having trouble detecting the coke can, majority of the time I just see "publishing to topic:/recognized_object_array". Very rarely I could catch a glimpse of the mesh, when it mistaken a random object for coke. If I use Kinect1 however, it detects fine. Just not sure what I am doing wrong with Kinectv2?

wmy101 commented May 3, 2017

Hi, I have tried the modifications to the linemod and training files but am still having trouble detecting the coke can, majority of the time I just see "publishing to topic:/recognized_object_array". Very rarely I could catch a glimpse of the mesh, when it mistaken a random object for coke. If I use Kinect1 however, it detects fine. Just not sure what I am doing wrong with Kinectv2?

@TrinhTUHH

This comment has been minimized.

Show comment
Hide comment
@TrinhTUHH

TrinhTUHH Aug 6, 2017

Hi @JimmyDaSilva,
How could you get that precise detection. I am using both linemod and tabletop and when it detect the coke the position jumps around the image just like this problem:
https://groups.google.com/forum/#!searchin/object-recognition-kitchen/calibration%7Csort:relevance/object-recognition-kitchen/8LrJj2OLDKQ/bEjY45ytCgAJ . Do you have any idea why?

TrinhTUHH commented Aug 6, 2017

Hi @JimmyDaSilva,
How could you get that precise detection. I am using both linemod and tabletop and when it detect the coke the position jumps around the image just like this problem:
https://groups.google.com/forum/#!searchin/object-recognition-kitchen/calibration%7Csort:relevance/object-recognition-kitchen/8LrJj2OLDKQ/bEjY45ytCgAJ . Do you have any idea why?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment