libuvc-based camera support

[d235j]

This is necessary for Mac OS X and Linux (for cross-platform camera access).

I have patched up libuvc to work correctly at https://github.com/d235j/libuvc but I have not yet had a chance to integrate libuvc support into OSVR-Core. The changes in #284 should make this easier.

[rpavlik]

I assume you mean OS X (not Mac since I was recently informed there is no such thing) and Linux...

[WiedemannD]

I assume OSVR-Core with libuvc support is mandatory for the HDK Camera to work/track position on OS X at all? If yes, I would up vote this, if I could!

[godbyk]

There's work in progress in the uvc-camera branch.

[gustavklopp]

[ChristophHaag]

I looked a bit at the code and so far it seems like the VideoBasedTracker plugin gets the image and extracts some points without much trouble and the debug window looks nice enough, but most of the leds are all labeled -3 most of the time and when dumping to csv most of the positions show -nan.

Can you describe what needs to be done there? Is it normal that the LEDs are flickering in that video? Are they not synchronized to the camera yet? Or is it something else?

https://www.youtube.com/watch?v=Be9PGokKSBI

[Conzar]

The numbers should be green. See this video for the correct behavior.
https://youtu.be/0faUZWFu6-E

[ChristophHaag]

Thanks for the reference video. Note that the Video Tracker Calibration Utility is only available for windows, and that in my video you see the output from this code

OSVR-Core/plugins/videobasedtracker/VideoBasedTracker.cpp

Line 209 in 3a449d7

if (m_params.debug) {

instead.

I take from your video that the flashing" effect is normal, so thanks for that.

It was a bit hard to see while flickering so I added some code to show ? for unidentified LEDs and ID: <led.getOneBasedID()> for identified LEDs and it looks like it's just identifying random LEDs as random numbers: https://www.youtube.com/watch?v=KEj_pv8p-QI
So my guess is still that the LEDs are not properly synchronized to the video framerate.

[Conzar]

How are you running VideoBasedTracker?

[ChristophHaag]

Well, it is run by the default osvr_server_config.HDK20ExtendedLandscape.sample.json. If you mean the window, it is shown when you edit this config file, go to the com_osvr_VideoBasedHMDTracker section and set showDebug to true.

[rpavlik]

So if you are in the show debug mode, and press B for blobs view, then you'll see both reprojected locations (if you get a lock) as well as the beacon ID values: negative are sentinel values. -3 means enough frames to identify but no pattern matched: I'd imagine this is a frame duplication or drop. (-1 is not enough data associated with a continuous blob).

There should be some pulsing, but not flickering, if you have the sync cable connected. (The video calibration tool, view cam, etc are just windows only because I kept the code brief and didn't abstract out the factory - should be easy to tweak.)

But yes, to get it to work, we do need every single frame, with some reliable amount of world to plugin latency.

[ChristophHaag]

So I have been playing with libuvc settings a bit: ChristophHaag@9d377e5

Nothing seemed to really help (and much is undocumented in the driver at https://github.com/ktossell/libuvc/blob/master/src/ctrl-gen.c so I ended up putting in a time measure every time libuvc calls the callback with a new frame:

7.869802ms since last frame = 127.068000 fps
11.954927ms since last frame = 83.647520 fps
7.986380ms since last frame = 125.213175 fps
11.984378ms since last frame = 83.441961 fps
7.908881ms since last frame = 126.440137 fps
12.015909ms since last frame = 83.223000 fps
7.863718ms since last frame = 127.166310 fps
11.998185ms since last frame = 83.345939 fps
7.970222ms since last frame = 125.467020 fps

So that's not smooth 100fps. Perhaps libuvc is batching/buffering stuff? Could this be the reason that it's not working properly?

[rpavlik]

Well, as long as it's getting every frame (is the device under test updated
with the latest IR board firmware to fix the sync timing?), it shouldn't
really matter the steadiness of the callback, though of course a more
steady callback is preferable.

You might consider looking at the "blobs-undo-bad" branch (which is the new
tracker branch), considering rebasing onto it, and using the uvbi-view-cam
app in its optional recording mode (compile-time ifdef) to see if it looks
like each frame has some distinctly brighter and some distinctly dimmer
LEDs.

On Mon, Sep 26, 2016 at 11:42 AM Christoph Haag notifications@github.com
wrote:

So I have been playing with libuvc settings a bit: ChristophHaag/OSVR-Core@9d377e5
ChristophHaag@9d377e5

Nothing seemed to really help (and much is undocumented in the driver at
https://github.com/ktossell/libuvc/blob/master/src/ctrl-gen.c so I ended
up putting in a time measure every time libuvc calls the callback with a
new frame:

7.869802ms since last frame = 127.068000 fps
11.954927ms since last frame = 83.647520 fps
7.986380ms since last frame = 125.213175 fps
11.984378ms since last frame = 83.441961 fps
7.908881ms since last frame = 126.440137 fps
12.015909ms since last frame = 83.223000 fps
7.863718ms since last frame = 127.166310 fps
11.998185ms since last frame = 83.345939 fps
7.970222ms since last frame = 125.467020 fps

So that's not smooth 100fps. Perhaps libuvc is batching/buffering stuff?
Could this be the reason that it's not working properly?

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
#339 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AADuyQreZgEHkoGCl9KiPGOVXFLGHYNPks5qt_XrgaJpZM4HHRjZ
.

Ryan A. Pavlik, Ph.D.
CTO - OSVR Platform
Sensics, Inc.
www.sensics.com

Latest news and blog posts (subscribe here
http://sensics.com/subscribe-to-our-mailing-list/ to get weekly updates):

Sep 13: Why WebVR? http://sensics.com/why-webvr/

Aug 30: TechCrunch: VR from the battlefield to the couch and back
https://techcrunch.com/2016/08/30/vr-on-the-battlefield-to-the-couch-and-back-again-sort-of/

Aug 24: OSVR comes to WebVR http://www.sensics.com/osvr-comes-webvr

[ChristophHaag]

To be honest, I don't know if the IR firmware is up to date. It's a HDK2 with whatever it came with and my crude try with the big cables didn't prove successful. Is there a way to check the version via software?

To rebasing. So many conflicts... Maybe I'll do that some time later.

[Conzar]

@ChristophHaag Version 1.97 seems to be the latest (at least that is what is in the git repos). You can update via linux here:
https://github.com/OSVR/OSVR-Docs/blob/master/Utilities/UpdateHDKLinux.md

[Conzar]

Also a question for the devs, why is libuvc required? Doesn't opencv have the necessary drivers for communicating with USB devices: example

[godbyk]

@Conzar The IR driver firmware is different than the main HDK firmware. While the main firmware can be upgraded via Linux, the IR driver firmware can't (as far as I know).

libuvc is required so we can specify that we want to open a specific camera. With OpenCV, you have to guess which index is associated with which camera and it could change if cameras are added or removed.

[rpavlik]

You actually can upgrade the IR driver board firmware in Linux, at least in theory. The one that you can't in Linux is the camera firmware.

Libuvc also would provide more accurate timing information, ability to change device parameters, etc. If you build the current code, it actually does use OpenCV capture on non Windows platforms but it's more for making sure the build isn't broken than actual usage.

[toastedcrumpets]

I'm having a similar issue on Linux. When running osvr_server, the beacons are not identified (I do see the id 37 on every beacon at some point using the debug window, see this video). I've run the calibration tool on windows and successfully calibrated it there, but on Linux I'm having no joy. I've merged the uvc-camera branch onto the current master and built it on Ubuntu 16.04. I'm using the HDK1.3 with v1.97 firmware, with the IR Camera firmware v7.

What is the current status of the uvc-camera branch and Linux support for the tracking camera? Are there known blockers which means developers should not try to use it at this point?

Is updating the IR driver board firmware required for Linux support/will it fix my issue? I'd have to buy a ST-link programmer but I'll get on this if this is a critical item.

[ChristophHaag]

The known blocker is that it doesn't identify the LEDs...

As I see, the original developer of the libuvc support didn't get it to work and had to focus on something else in the meantime and nobody else knows why it doesn't work yet either.

Maybe libuvc has to be set up in a specific way. I tried to play with some of the settings and some playground code is linked in my comment #339 (comment) but nothing really helped. I only found that the frames don't arrive very smoothly, but I don't know enough about the tracking to say whether that's a problem.

The improved tracking code will arrive soon (?), so someone will have to rebase the uvc-camera branch. Maybe we get lucky and it will just start working, but maybe not.

Maybe someone could just dump 100 frames or so both in windows and in linux with the uvc-camera branch, of course with lighting etc. as closely the same as possible and in a direct comparison something were to stand out?

[toastedcrumpets]

I actually resolved my problem with the original opencv based code, embarrassingly my camera's firmware was v5, but the windows update tool wouldn't upgrade it and kept saying I already had the correct version; however, using another system allowed me to update the firmware to v7 at last.

This means I should be able to dump what is being fed into the tracker algorithm from the (now working) opencv implementation and the libuvc implementation. I'll try to dump the OpenCV image just before processing (perhaps via the existing debug code on line 342 of plugins/videobasedtracker/VideoBasedTracker.cpp) with both branches and compare.

[ChristophHaag]

Dumping frames to disk is quite easy with opencv's imwrite(). Disclaimer: I don't really know anything about the tracking, I'm just poking in the dark and looking for significant differences, for example maybe something with contrast/gamma/brightness/sharpness settings etc. will turn up, or maybe not. Also check out the $155 bounty on this bug for anyone who figures it out and makes it work. :)

[rpavlik]

I'd suggest checking to make sure that every frame is being captured and that none are being dropped unexpectedly - drop a frame, you lose tracking. Could record the LED pattern in Windows to see what it should look like.

[toastedcrumpets]

Thanks for the hint rpavlik.

If dropping a frame makes you lose tracking, then the current UVCImageSource is insufficient as the libuvc camera thread can race ahead and delete frames before they are processed by the server thread. The fix I propose is to add a small queue to allow the system to process and receive frames in parallel. I have limited the size of this queue/buffer to only 100 frames to prevent it eating up all the memory on "slow" systems which aren't processing frames fast enough.

If the limit on buffered frames is reached, the entire queue is dropped (as we'll have to drop a frame, and if we do we may as well completely restart tracking from the latest image) and a warning is issued to the user to notify them their system is not processing frames fast enough. Finally, I have used condition variables to avoid spinlocking the cpu.

I have implemented this and will submit my pull request now. I now have tracking working under linux, and it appears to be much faster than the opencv path (at least for me).

[ChristophHaag]

In retrospect it seems kinda obvious. Thanks a lot!

Wouldn't it be a simpler method altogether to let the libuvc callback trigger the image processing? Of course on slow systems you'd still have the problem that if processing an image takes longer than the interval until the next image, it would have to drop frames and lose tracking completely again, but on those systems you'd have delayed tracking with the queue and erratic tracking when it drops the queue anyway.

[toastedcrumpets]

You can't let the camera thread hang around performing processing, as it needs to return to handling the usb traffic otherwise you'll drop (usb) frames leading to dropped camera frames or additional latency. Also, managing the shared memory of the plugin/server thread and the camera thread would require far more complexity.

That's why I also made sure everything is done outside the locks wherever possible (to allow the libuvc thread to return as fast as possible). One example is in retrieveColor, where the lock_guard scopes out immediately after grabbing the frame from the queue.

[rpavlik]

@toastedcrumpets Ah, cool! I'd be very surprised if you need anything close to 100 entries in that queue.

FWIW, on the new tracker branch (which this can probably be just ported to, rather than an attempt at a "rebase-onto"), we do have the Folly library vendored, specifically for lock-free single-producer single-consumer queues (used a few places in the new tracker plugin), and it does really make things smoother (at least on Windows).

The catch with doing stuff in the callback is that usually a: it's not typically good practice to put a lot of processing in a callback of any sort (unless you're specifically told you can) combined with b: the first step in the tracking, which is doing the image processing to extract blobs from the image, is the slowest and heaviest computationally. The new plugin does do that in a separate thread so it can also service IMU reports while waiting for an image to be captured and turn into blobs. There, I think, since there's only one image "in flight" from the plugins point of view, it's just condition variables saying whether the "time consuming image processing step" is done or there's IMU data to service.

[toastedcrumpets]

Yeah, 100 is "excessive" as I think the queue is reaching a size of 1-2 most of the time, but changing that number will make no performance difference on any normally loaded system, so I decided to pick something that wouldn't flood the console with error messages (i.e. this gives a warning every second in the worst-case that no frames are being processed).

Using lock free queues would be fine, but I understand that grab() must be blocking if there are no frames, so you'll still have to use the condition variables. That might need delicate handling to ensure that no frames are added while the server thread checks for frames then enters a wait, otherwise you'll get a lag of one frame before processing starts.

[ChristophHaag]

It's usable on osvr-core master, so good job for that.

As per the suggestion I also tried the blobs-undo-bad branch and it is a lot worse there. I've made this pull request with all the commits from the current pull request combined, even though I don't think it should be merged in the current state.
#493

[rpavlik]

@toastedcrumpets brought it home (though on the old plugin - but the hard work is done, just needs porting to the new plugin for full power) so I'm going to close this issue.

[rpavlik]

I created a new issue (see above) for the port between plugins.

[rpavlik]

Oh - I see that @d235j also contributed to this in addition to @godbyk, @ChristophHaag, and @toastedcrumpets . Ya'll will have to figure out how the bounty thing works, never used it before myself. I seem to recall @godbyk and @ChristophHaag telling me they were fine with others getting the bounty, correct me if I'm wrong or no longer correct.

[toastedcrumpets]

FWIW I'm happy to share, @godbyk got this all started which saved me a lot of time figuring out how to tie stuff in.

[godbyk]

@toastedcrumpets @rpavlik : I've already been paid for my part in this code, so I'm happy to see the bounty distributed to the others who carried it across the finish line.

Thanks, @toastedcrumpets and @ChristophHaag for your work fixing up this code and testing it!

[rpavlik]

As a follow-up, I am merging the version of this code that got ported to the new tracker plugin as well in #587

[godbyk]

@rpavlik We should try to resolve this in such a way that the bounty gets paid out, too.

[rpavlik]

Wait, that's still pending? Oh goodness sakes, I thought merging things was sufficient.