Features : Color/Depth registration, World, IR

[elliotwoods]

Heya!
I'm teaching a workshop this weekend which relies on KinectV2 running on OSX
(It's a Rulr workshop, and Rhizomatiks is helping tto port Rulr to OSX for the mostly mac audience there)

I'm thinking to use this addon. Looks pretty neat and tidy and is top of the pops on ofxAddons

3 things I need seem to be missing right now:

1. Getting world XYZ chords from depth coords (both to render a live mesh which could be in a shader, and to perform calibration tasks which should be on cpu)
2. Depth <> Color registration and coordinate mappinh (ie individual points and all pixels). libfreenect2 seems to have a class Registration for these tasks
3. Getting the IR image

Any tips about getting these features would be great.
I might be able to hack a couple of these together and send you the code.
Is anybody else using those features with libfreenect2 / OF do you know?

Notes at:
OpenKinect/libfreenect2#41

Thanks!
Elliof

[elliotwoods]

The Registration class has pretty much everything I need:
https://github.com/OpenKinect/libfreenect2/blob/master/include/libfreenect2/registration.h

Are the libs in ofxKinectV2 up to date?
I had trouble building them myself before

[elliotwoods]

Hah....Just realised that it stripped the <strikethrough> I'd put around 'I demand you implement these functions immediately'.
And I can't edit existing issues from my phone

[mattfelsen]

Hey Elliot. That's awesome that Rhizomatiks is helping to port! There is already support for most of the stuff you mentioned in ofxMultiKinectV2 👍 I imagine it wouldn't be too hard to borrow some of the features, though my dream would be a unification on this front into a single libfreenect2-based addon at some point 😜

depth/color registration, in a shader. I took a quick glance and it looks like a port of libfreenect2's registration implementation
https://github.com/hanasaan/ofxMultiKinectV2/tree/2fa68722a14c10fd6311738ae6b27d5b1eb6e350/src/opt
https://github.com/hanasaan/ofxMultiKinectV2/blob/2fa68722a14c10fd6311738ae6b27d5b1eb6e350/example_registration/src/ofApp.cpp

depth -> world mapping:
https://github.com/hanasaan/ofxMultiKinectV2/blob/2fa68722a14c10fd6311738ae6b27d5b1eb6e350/src/ofxMultiKinectV2.cpp#L185-L199
https://github.com/hanasaan/ofxMultiKinectV2/blob/2fa68722a14c10fd6311738ae6b27d5b1eb6e350/example_pointcloud/src/ofApp.cpp
I actually added this code to ofxKinectV2 and had the point cloud working. It was a pretty quick addition (I had to add a getter for the protonect device I think), but I can submit a PR if that's helpful. I didn't profile it so I'm not sure how fast it was..

infrared:
https://github.com/hanasaan/ofxMultiKinectV2/blob/2fa68722a14c10fd6311738ae6b27d5b1eb6e350/example/src/ofApp.cpp

[ofTheo]

I actually have the getWorldCoords and getWorldCoord in another branch which I can push up today.

Also the registration/mapping is super easy to add in.

The only issue with using the shader approach of libfreenect is a bug in 10.10 and 10.11 which makes the glReadPixels call hang. Not sure if that's fixed or there is a workaround for that in ofxMultiKinect.

But that's why I switched to the OpenCL decoding pipeline.

[elliotwoods]

You mean the shader approach for depth unwrapping from raw images right
Theo?

I think Matt is talking about shader approach to registration and depth
unprojection (to XYZ), which is probably less risky
On Thu, 17 Dec 2015 at 23:26, Theodore Watson notifications@github.com
wrote:

I actually have the getWorldCoords and getWorldCoord in another branch
which I can push up today.

Also the registration/mapping is super easy to add in.

The only issue with using the shader approach of libfreenect is a bug in
10.10 and 10.11 which makes the glReadPixels call hang. Not sure if that's
fixed or there is a workaround for that in ofxMultiKinect.

But that's why I switched to the OpenCL decoding pipeline.

—
Reply to this email directly or view it on GitHub
#18 (comment).

[ofTheo]

I just meant I'm not sure if that glReadPixels is an issue for using ofxMultiKinect in general on 10.10

If it's not then it might be worth checking out. That Addon has moved a bit quicker than mine :)

Happy to add that stuff into ofxKinectV2 if needed though. It's on my to do list.

PS: huge thanks for your windows kinect addon, it was super awesome and easy to use. I recommend everyone to use it on windows.

[elliotwoods]

ahh you mean Yosemite? i see, that could be a problem..

i'll look into ofxMultiKinectV2 also

be great to work against something which will exist for a while
happy to PR some things back into ofxKinectV2 if I can write them nicely

i need to look at this tomorrow Japan time. will it be at all possible to get your getWorldCoords branch before then? or should i try something else?

thank you both!

[elliotwoods]

aha!
ofxMultiKinectV2 is by Yuya Hanai
who is teaching the workshop at Rhizomatiks with me! (i don't know how i didn't realise this before :)

i think i'll start with that then tomorrow when i'm with Yuya, and add any functionality Rulr needs over there.
and agreed, it'd be good for users if these 2 came together

[mattfelsen]

Ah perfect. Also as Theo said, thanks much for your addon – not only for its usefulness but I've picked up a bunch of tricks & techniques from your coding style as well 😄 Please give my thanks to Yuya too 👍 Wish I could be there to learn from you guys, but instead I'll have to settle for lurking on your repo

[ofTheo]

yeah totally happy to merge with @hanasaan
I think there is some nice things in both right now - but I think he has already got a lot of the advanced features. Could just make him an admin on this repo :)

[mattfelsen]

Didn't mean to leave you out Theo, this addon is the bomb and so are you. Tell Nick that Joe says hi.

[elliotwoods]

I worked with ofxMultiKinectV2 today. it's got a lot of fantastic functionality out of libfreenect2. almost as good as official SDK, but a little bit slower maybe

i think the API needs quite a bit of work to match the simplicity of ofxKinectV2
it's pretty much all there, just needs refactoring,

also it would be great to use a lazy pattern for things like registration / unprojection tasks as they can be quite heavy

[ghost]

Hi !
I don't see your getWorldCoords functions @ofTheo, did you pushed them ?
How is the merge with ofxMultiKinectV2 mentionned here ?
Did you / do you plan to add to your addon the kinect merge with point clouds feature you used in your Connected Worlds project ? I need to implement a similar method, that's why I need to access world coordinates

[ofTheo]

@BaptisteTheoriz here is the world coords branch
right now it runs whether or not world coords are requested - I should prob add a way to toggle it off, to speed things up a bit if your not using.

https://github.com/ofTheo/ofxKinectV2/tree/feature-worldcoord

[ofTheo]

also would be happy to add @hanasaan as admin on this repo and merge the two

[ghost]

Thanks @ofTheo
While waiting for your feature, I did it the same way as @hanasaan in ofxMultiKinectV2 (commit).
It seems to do the job, I don't know which method is the best

[ofTheo]

ahh I see - that also looks good.
I guess the only difference is the one in my PR is doing it on the registered coords so that the color data will correspond to the XYZ cloud data.

[uh]

thanks for this @ofTheo. i'm using your feature-worldcoord branch on something like the ofxKinect kinectExample that draws a pointCloud coloured by pixels from rgb, and i'm finding that the color data does not quite match the XYZ data, it's got the unregistered distortion (even after having corrected for the different fields of view and resolutions of the ir and rgb pixels). i created a basic getColorAt(x,y) method but there doesn't seem to be a setRegistration(true) that corresponds to anything in libfreenect2 -- unless i'm missing something obvious. got any pointers to set the registration and/or get them aligned?

[bakercp]

https://github.com/bakercp/ofxKinectV2 adds these features.

[ofTheo]

#34 should close this issue.