Raw capture at full res on still port fails

[waveform80]

Raw (YUV/RGB/RGBA/etc.) captures against the still port fail when resolution is 2592x1944 with an out of memory error in the latest firmware (656). This doesn't happen in raspiyuv but the pipeline in picamera is rather more complex: raspiyuv sets the preview res to 1296x972 (rounded up) while picamera has one res for everything (in this case 2592x1944), and picamera uses a resizer (in this case redundantly) to perform conversion to the appropriate raw format rather than reconfigure the camera's ports (which would break capture-while-recording for the video ports).

Not sure what the solution here is - it's probably soluble in picamera but only by breaking backwards compatibility by changing various things about raw capture. For now I'm tempted to leave it as a known issue in the release and see if anybody complains about it? (i.e. does anyone really care about full res raw captures or is this a niche bit of functionality)

[waveform80]

Okay, definitely need to fix this as it seems some people are relying on this functionality (had two reports of people encountering it now)

Fixing YUV captures wouldn't be hard as we can just ditch the resizer for those, but what about the other raw formats?

[waveform80]

Okay, after much hacking at the weekend I've concluded the following:

1. It's not the resizer
2. It's not the video splitter either
3. It's not the discrepancy in preview resolution (it turns out raspiyuv works with a full-res preview anyway)

Unfortunately I don't know what it is yet. I'm beginning to think I might've run into some obscure memory leak or something else in the firmware, but I've no conclusive proof of anything yet. Well ... back to the debugging ...

[waveform80]

Edit: Ignore this comment - it's wrong (got confused about which port was which encoding!). The proper workaround is a couple of comments down...

Finally! After what seems like days of debugging I've tracked down exactly what the difference is between picamera and raspiyuv which causes the ENOMEM error when performing full-res raw capture. That's the good news. Here's the bad: it's not something I can fix without breaking serious amounts of functionality in picamera. However, I'm pretty convinced it's a firmware bug so I'll report it upstream and see what comes of it.

In the meantime, here's what's going on and how to work around it: picamera sets the camera's video port to YUV 420 encoding because we need it to work with the splitter (for video output, video-based encoded and raw captures, etc. etc.). Meanwhile, raspiyuv just copies the preview port's format to the video port, which means the video port winds up with opaque encoding (this is perfectly reasonable since raspiyuv doesn't use the video port at all - it just needs to give it a valid format). For some reason (and this is why I suspect a firmware bug), setting the video port to opaque encoding means still-port-based full-res raw captures work fine, but set it to YUV 4:2:0 encoding and an ENOMEM (out of memory error) is reported (despite the fact we're not using the video port at all).

To work around this in the current version of picamera, change line 319 of picamera/camera.py from this:

fmt[0].encoding = mmal.MMAL_ENCODING_I420 if p == self.CAMERA_VIDEO_PORT else mmal.MMAL_ENCODING_OPAQUE

To this:

fmt[0].encoding = mmal.MMAL_ENCODING_OPAQUE

WARNING: doing this will break all video related functions (i.e. any calls to start_recording, record_sequence or any of the capture methods with "use_video_port=True" will fail).

[waveform80]

Argh! Sorry folks, now even the work-around doesn't appear to work (I could've sworn it was earlier). There's something I'm missing here ... back to the drawing board!

[waveform80]

Right, ignore everything in my comment above (yes, I could edit it but I think I'd just confuse myself further if I start modifying the history on this!). Here's the situation: it's nothing to do with the video port encoding (confirmed that after building a modified version of raspiyuv). It's down to the still port encoding after all, but it depends on some state that can't be observed from MMAL...

Firstly a quick bit of history. In all versions of picamera, the preview port's encoding is fixed on opaque, and the video port's encoding is fixed on I420 (aka YUV 4:2:0). However, the still port's encoding has varied over time as it appears somethings work in some encodings and not in others. Specifically, full exif output (for JPEGs) only works when the still port's encoding is opaque, not when it's I420. For raw captures, the still port's encoding has to be I420 (for YUV) or BGR24 (for RGB) etc. so when required we disable the camera, switch the port's encoding and re-enable the camera.

Now it turns out that if the camera's still port is ever set to opaque encoding (even if it's reset to I420 later), an out-of-memory error occurs when a full-res raw capture is attempted. In order to avoid this the entire camera component has to be destroyed and recreated. I've finally managed to confirm this with a slightly tweaked version of raspiyuv so I should be able to report this upstream now (I'm pretty confident this is a firmware bug).

As to fixing it in picamera ... firstly, here's the workaround I'll be implementing in a bit. Change line 319 of picamera/camera.py from this:

fmt[0].encoding = mmal.MMAL_ENCODING_I420 if p == self.CAMERA_VIDEO_PORT else mmal.MMAL_ENCODING_OPAQUE

To this:

fmt[0].encoding = mmal.MMAL_ENCODING_I420 if p != self.CAMERA_PREVIEW_PORT else mmal.MMAL_ENCODING_OPAQUE

This will mean that when an instance of PiCamera is created its still port will initially have I420 encoding. This means full-res YUV captures should work. However, if at any point you take a JPEG (or PNG or anything that switches the port to opaque encoding), subsequent YUV captures will fail until you recreate the instance. Obviously this isn't a long term solution (for starters I'm going to have to modify the test suite temporarily to recreate the camera instance between calls), but I figure most users won't be switching between YUV and JPEG encoding so it should fix things for most users...

[waveform80]

Another workaround, discovered by a diligent user (thanks Jason!) is to set gpu_mem=256 in /boot/config.txt. This will allow picamera 1.3 to take full-res YUV captures without alteration, but obviously isn't an option on Model A's which only have 256Mb of RAM to start with. I haven't yet had time to experiment with values between 128 and 256 to see where the cut-off might be - this may be left as an exercise for adventurous Model A users!

[waveform80]

Looks like there's a couple of things here which contribute to this error: firstly, picamera should be modifying buffer_num and buffer_size after a still port encoding change and secondly we should ignore buffer_num_recommended as there seems to be a minor upstream issue with it staying too high after the port's been set to opaque.

So, as suggested by 6by9 in raspberrypi/userland#167 we should set buffer_num to buffer_num_min and buffer_size to buffer_size_recommended after we've fiddled with the still port encoding (i.e. in PiCamera._set_still_encoding).

Note to self: once this is fixed, remember to switch the scope on the camera fixture in the tests to ensure it's actually worked properly!