Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stereo mode error #218

Closed
guyver2 opened this issue May 18, 2015 · 15 comments

Comments

@guyver2
Copy link

commented May 18, 2015

Stereo mode doesn't work on the compute module with two cameras.

This minimal piece of code fails :

import picamera
import sys
import traceback

try :
   camera = picamera.PiCamera(stereo_mode='side-by-side')
   camera.start_preview()
   # Camera warm-up time
    time.sleep(2)
    camera.capture('foo.jpg')
except Exception as e:
    print "Error:"
    print "############"
    print e
    print "############"
    traceback.print_exc(file=sys.stdout)

It gives the following output :

pi@computePI ~/stream $ python test_stereo.py
Error:
############
Unable to set stereoscopic mode on output 0: Argument is invalid
############
Traceback (most recent call last):
  File "test_stereo.py", line 7, in <module>
    picamera.PiCamera(stereo_mode='side-by-side')
  File "/usr/lib/python2.7/dist-packages/picamera/camera.py", line 419, in __init__
    self.STEREO_MODES[stereo_mode], stereo_decimate)
  File "/usr/lib/python2.7/dist-packages/picamera/camera.py", line 525, in _init_camera
    prefix="Unable to set stereoscopic mode on output %d" % p)
  File "/usr/lib/python2.7/dist-packages/picamera/exc.py", line 133, in mmal_check
    raise PiCameraMMALError(status, prefix)

Here are some information on my system :

pi@compute ~ $ uname -a 
Linux compute 3.18.13+ #784 PREEMPT Sat May 9 15:49:42 BST 2015 armv6l GNU/Linux

pi@compute ~ $ vcgencmd version
May 13 2015 14:58:12 
Copyright (c) 2012 Broadcom
version 8e0e0dbfe92be77d6355082451280d32f5bf0ff3 (clean) (release)

Also, the latest version of raspistill/raspivid do provide a working support for stereo through the -3d option

@waveform80

This comment has been minimized.

Copy link
Owner

commented Jun 27, 2015

Sorry I haven't had much time to look at picamera tickets lately! Stereo mode's especially tricky given I don't have the hardware to test on. That said I can certainly compare exactly what raspistill/vid do and what picamera does and I'll try and do this before the next release to ensure picamera does effectively the same thing when stereo_mode gets set in the constructor

@waveform80 waveform80 added the bug label Jun 27, 2015

@waveform80 waveform80 added this to the 1.11 milestone Jun 27, 2015

@waveform80 waveform80 self-assigned this Jun 27, 2015

@guyver2

This comment has been minimized.

Copy link
Author

commented Jun 29, 2015

Thanks Dave,

If I can be of any help let me know. I do have the hardware and I'm more than willing to test any modification you would like to try.

@6by9

This comment has been minimized.

Copy link

commented Jul 16, 2015

Daft as it sounds, I suspect it is because you've set the camera number at https://github.com/waveform80/picamera/blob/master/picamera/camera.py#L534 before setting the stereoscopic mode at line 583.

Setting the camera number triggers the loading and opening of the selected camera driver as a startup optimisation (some things had to wait for numbers to be available from the camera driver, so if the loading of that can be triggered early, then you save time. That's also why a parameter changed event can be triggered off the camera number - it triggers when the camera number has been changed and that driver is loaded and available).
Selecting stereoscopic mode needs a different set of parameters when opening that driver - specifying that you want to channels and two camera drivers being the main things. If it's already open when you try to select stereo mode, then it throws an error.

guyver2 added a commit to guyver2/picamera that referenced this issue Jul 22, 2015

@guyver2

This comment has been minimized.

Copy link
Author

commented Jul 23, 2015

Following 6by9 comment I came up with this solution (see above) that fixes the issue for the stereo mode. Now I can access synchronized streams on my compute module.

Maybe the stereo part of the documentation could be made clearer. Especially this part :

If the stereo_decimate parameter is True, the resolution of the two cameras will be halved so that the resulting image has the same dimensions as if stereoscopic mode were not being used.

The resolution of the image we get from stereoscopic mode is the same as the given (or default) resolution parameter, and if a regular 4/3 ratio is given, the two images will be halved so that their combination fits into a 4/3 image.

For example, to get one stereo image with two full (480x320) images inside, I have to use something like this :

camera = picamera.PiCamera(stereo_mode='side-by-side', resolution=(960,320), stereo_decimate=False)
@waveform80

This comment has been minimized.

Copy link
Owner

commented Mar 18, 2016

Well, I've now got hold of a compute module dev-kit - yay! Unfortunately I've only got a single camera adapter at the moment (as it only comes with one...). I'll see if I can borrow another one next week to finally test this!

@waveform80

This comment has been minimized.

Copy link
Owner

commented Apr 17, 2016

Got a spare camera adapter (generously lent by @lesp!), and got it all set up with both cameras working under raspivid/still ... but my god, I'd forgotten how long it took to compile numpy on a Pi 1 (the Pi2/3 has ruined me ;) ... an hour in and it's still going. Think I'll leave this overnight.

@6by9

This comment has been minimized.

Copy link

commented Apr 18, 2016

but my god, I'd forgotten how long it took to compile numpy on a Pi 1

:-) I had the same when doing some V4L2 development for multiple cameras the other week. I now have CM3 working though :-o

@waveform80

This comment has been minimized.

Copy link
Owner

commented Apr 18, 2016

Grrr! Definitely need to get my hands on one (or more) of those (my slice would definitely appreciate one!).

Started having a play with stereoscopic support - definitely a fair amount I need to fix up here. @guyver2's patch is a start but doesn't permit selection of the primary camera (probably not that important, but nice to support if we can). Then there's the issue that picamera constantly runs the preview at the full camera resolution which simply won't work with a stereoscopic full resolution still capture (out-of-mem when initializating with res 5184x1944 - hardly surprising :), so I need to have a bit of a think about how to separate out camera and preview resolution without breaking the API and keeping things relatively simple. And I ought to add the swap-eyes feature. Hmmm.

(unfortunately the only two spare cameras I had lying around were a regular and a noir ... which makes for some amusingly unbalanced stereo captures given one camera does the exposure/awb sampling :)

@6by9

This comment has been minimized.

Copy link

commented Apr 18, 2016

A better fix would be to move the section (line 581)

             port = self._camera[0].output[p]
             # Don't attempt to set this if stereo mode isn't requested as it'll
             # break compatibility on older firmwares
             if stereo_mode != mmal.MMAL_STEREOSCOPIC_MODE_NONE:
                 mp = mmal.MMAL_PARAMETER_STEREOSCOPIC_MODE_T(
                     mmal.MMAL_PARAMETER_HEADER_T(
                         mmal.MMAL_PARAMETER_STEREOSCOPIC_MODE,
                         ct.sizeof(mmal.MMAL_PARAMETER_STEREOSCOPIC_MODE_T),
                     ),
                     mode=stereo_mode,
                     decimate=stereo_decimate,
                     swap_eyes=False,
                     )
                 mmal_check(
                     mmal.mmal_port_parameter_set(port, mp.hdr),
                     prefix="Unable to set stereoscopic mode on output %d" % p)

to before setting MMAL_PARAMETER_CAMERA_NUM, but still set CAMERA_NUM. That would give you selection of primary camera via camera_num.

Memory use should be controllable, but depends on where you can make use of OPAQUE buffers - I seem to recall a brief exchange over pipeline architecture a fair while back.

Another potential nasty: not all components may respect the stereoscopic representation.
camera, video_encode, image_encode, and video_render all do. I fixed a bug with the MJPEG codec the other week where it didn't understand the signalling. resize or video_splitter being presented with opaque buffers may well result in dropping the second image. Let me know if so and I'll try to fix it.

@waveform80

This comment has been minimized.

Copy link
Owner

commented Apr 18, 2016

A better fix would be to move the section (line 581) ... to before setting MMAL_PARAMETER_CAMERA_NUM, but still set CAMERA_NUM. That would give you selection of primary camera via camera_num.

Yup - done that in my test clone on the CM - works nicely.

Memory use should be controllable, but depends on where you can make use of OPAQUE buffers - I seem to recall a brief exchange over pipeline architecture a fair while back.

Yeah - I'm just re-acquainting myself with all this (been so long since I looked at it!). I vaguely recall there's a difference in the video pipeline (picamera uses I420) because I couldn't get the splitter working with OPQV way back when. I should have a go at doing that though, as it sounds vastly more efficient than throwing around whole copies of YUV data.

I think the still port defaults to I420 as well, but that was purely a temporary work-around for the buffer-size issues which were fixed ages ago, and it switches to OPQV when doing anything encoded like JPEG/PNG. Still, I should probably change the default to OPQV instead.

Still pondering the separation of preview and camera resolution at the moment though...

@waveform80

This comment has been minimized.

Copy link
Owner

commented Apr 18, 2016

Okay, partly for my own reference -still encoding's a little more complex:

  • I420 is the default (mostly likely for historical reasons relating to buffer allocation)
  • When an encoded output format (JPEG/PNG/GIF/BMP) is requested for capture from the still port, without resizing in the pipeline, still port switches to OPQV
  • When an unencoded output (YUV/RGB) is requested or encoded+resizer, I420 is used (resizer doesn't work with OPQV perhaps?)
  • Starting a recording forces still port to I420 as well - not quite sure why yet - maybe simultaneous capture+record doesn't work with OPQV still encoding?
@waveform80

This comment has been minimized.

Copy link
Owner

commented Apr 18, 2016

Ah, I see now - the video port is set to I420 because of the resizer functionality. With the existing picamera API it's valid for someone to start an unresized recording, then request to start another resized recording on another splitter port (in fact, it's explicitly listed in the recipes).

Now, given that:

  • Resizer only works with I420 encoding, not OPQV
  • All splitter ports share a format
  • Changing splitter port format requires stopping all existing processes using those ports

If the splitter started out in OPQV, the second recording would require us to shut down the first recording, reconfigure the splitter, then restart it (i.e. break the first recording). Alternatively, I could require users that wish to do this to start the resized recording first (giving us the hint that the video splitter needs to be running in I420) and start their other recordings after. It's technically an API break - but I'm rather tempted by it as it allows OPQV encoding on the splitter+video port.

@6by9

This comment has been minimized.

Copy link

commented Apr 18, 2016

raspberrypi/firmware#440 added supporting opaque into resize, although it seems it never got tested in anger.

I do have another largely untested patch on my firmware tree for "video_splitter: support format conversion on buffer in and out - previously it would only convert on opaque in, buffers out". I'm not sure if that helps you out or not.
Video_splitter output ports should all be independent for format.

So it sounds like you're after:

camera->splitter->(resize)->dest1
                ->(resize)->dest2
                ->video_encode->dest3
                ->dest4

So:

  • opaque camera->splitter
  • opaque splitter->resize,
  • chosen format resize->dest
  • chosen format splitter->dest4

should work.

edit: I should add, if there is a pipeline configuration that would make life easier for you, let me know and I'll see how much effort it is to fix up the necessary components to get it working.
Opaque into the video encoder may be the tricky one, as if it isn't direct from the camera video port then it's missing one of the images.

@waveform80

This comment has been minimized.

Copy link
Owner

commented Apr 18, 2016

Oooh - very interesting stuff, thanks! I'll have a play with that and see what I can come up with. On the downside that means this is probably going to become a rather large patch as it'll involve a fair bit of re-plumbing but it looks like picamera's rather overdue for that anyway if we want to take advantage of all the new firmware goodies.

@waveform80

This comment has been minimized.

Copy link
Owner

commented Apr 18, 2016

And yes, the pipeline you've outlined is an extreme but nonetheless plausible use-case in picamera's API.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.