New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MJPEG recording locks up camera #47

Closed
waveform80 opened this Issue Jan 25, 2014 · 20 comments

Comments

Projects
None yet
5 participants
@waveform80
Owner

waveform80 commented Jan 25, 2014

Under certain circumstances (resolution related?), MJPEG recording fails to stop. Looks like an explicit end capture signal is required (opposite of PiCamera._enable_port)

@ghost ghost assigned waveform80 Jan 25, 2014

@waveform80

This comment has been minimized.

Owner

waveform80 commented Jan 25, 2014

This is looking more like a firmware issue - in the lockup cases the encoder callbacks just stop and MMAL hangs when disabling the port ... weird!

@waveform80

This comment has been minimized.

Owner

waveform80 commented Jan 25, 2014

Reasonably confident this is a firmware issue now - when recording at 1080p the encoder callback gets called for about a dozen frames and then suddenly doesn't get called anymore. Attempts to disable the port don't return and a reboot is required to recover. Marking as an upstream issue for now.

@waveform80 waveform80 closed this Jan 25, 2014

@waveform80

This comment has been minimized.

Owner

waveform80 commented Mar 6, 2014

This looks awfully similar to #62...

@waveform80 waveform80 removed this from the 1.1 milestone Mar 19, 2014

@waveform80

This comment has been minimized.

Owner

waveform80 commented Mar 19, 2014

This shouldn't be closed as it's still an issue. Need to track down the exact resolution it occurs at and report it upstream. After updating the tests for #65 it appears even 720p locks up the MJPEG encoder.

@waveform80 waveform80 reopened this Mar 19, 2014

@rsacchettini

This comment has been minimized.

rsacchettini commented Jul 17, 2014

Same issue while trying to do a still image capture when in video capture mode (works like perfect in h264):

def captureToFile(camera, file):
    #camera.resolution = (2592, 1944)
    camera.capture(file,format="jpeg", use_video_port=True)

def streamVideo(camera, connection):
    print "start recording"
    camera.resolution = (640, 480)
    camera.framerate = 25
    camera.start_recording(connection, format='mjpeg')
    #

    #camera.stop_recording()
    #time.sleep(2)

    print "recording stopped"

if __name__ == "__main__":

    server_socket = socket.socket()
    server_socket.bind(('0.0.0.0', 8000))
    server_socket.listen(0)

    try:
        with picamera.PiCamera() as camera:
            camera.resolution = (640, 480)
            camera.framerate = 24

            # Start a preview and let the camera warm up for 2 seconds
            print "preview"
            #camera.start_preview()
            time.sleep(2)

            # Accept a single connection and make a file-like object out of it
            connection = server_socket.accept()[0].makefile('wb')
            print "connection made"
            captureStream = io.BytesIO()

            streamVideo(camera, connection)
            for i in range(0, 3):
                camera.wait_recording(8)
                t0 = time.clock()
                filename = "sample-%d_%s.jpeg" % (i, time.strftime("%H:%M:%S"))
                print "filename: {}", filename
                captureToFile(camera, captureStream)
                t1 = time.clock()
                diff = t1 - t0

                print ("time end of streaming => end of capture = %s" % (str(diff)))

    except KeyboardInterrupt:
        # quit
        sys.exit()

    finally:
        connection.close()
        server_socket.close()
@rsacchettini

This comment has been minimized.

rsacchettini commented Jul 17, 2014

A simpler code to reproduce the issue:

import picamera

with picamera.PiCamera() as camera:
    camera.resolution = (800, 600)
    camera.start_preview()
    camera.start_recording('foo.mjpeg', format='mjpeg')
    camera.wait_recording(10)
    camera.capture('foo.jpg', use_video_port=True)
    print " recorded foo.jpg"
    camera.wait_recording(10)
    camera.stop_recording()
    print "stopped recording"
@waveform80

This comment has been minimized.

Owner

waveform80 commented Aug 5, 2014

The script posted by @rsacchettini above produces the following output in sudo vcdbg log msg (I suspect only the last line is pertinent):

...
003240.919: TV service:host side not connected, dropping notification 0x00000002, 0x00000002, 0x0000001c
145303.210: camsubs: Looking for camera 0: i2c_port = 0, led gpio = 5, power enable gpio = 21
145304.832: camsubs: Camera found OK
145306.878: gpioman: gpioman_get_pin_num: pin CAMERA_LED not defined
146048.103: mmal: mmal_vll_load: could not load VLL 'videnc.vll': 
191142.236: mmal: fail_destroy: Timed out waiting to destroy ril.video_encode

After this message is produced, killing the python process leaves the camera operational and a reboot is required to restore functionality. Interestingly, reducing the resolution to 640x480 permits the script to work. MJPEG generally seems to have issues when the resolution is beyond VGA. Currently all picamera tests with MJPEG are xfailed where the vertical resolution exceeds 480; however all tests at VGA res or below succeed (including capture-while-recording, split-recording, etc). Whether the issue is the actual resolution or the bandwidth required by high resolutions, I'm not sure.

@waveform80 waveform80 changed the title from MJPEG recording fails to stop to MJPEG recording locks up camera Aug 5, 2014

@nikoder

This comment has been minimized.

nikoder commented Sep 8, 2014

For what it's worth, I am having no problems recording 1080p MJPEG through the v4l2 interface (using python-v4l2capture) with the Pi Camera, although I am having the same problems described here with Picamera. I'm not sure if this is helpful or not, because I am not sure if they rely on the firmware in the same way.

Also, it might be worth noting that I did have to make sure that only minimal buffers were allocated by v4l2. Allocating more than 2 frames of buffer at 1080 resolution caused memory issues with my "original" model B Pi. I'm not sure if the allocation of buffers is within the control of Picamera?

Any news from upstream? I'm not sure where to look for a potential upstream issue.

If anyone happens to be aware of a firmware/picamera version combination where this works, I would be grateful to know!

@waveform80 waveform80 added this to the 1.9 milestone Sep 8, 2014

@waveform80

This comment has been minimized.

Owner

waveform80 commented Sep 8, 2014

Interesting - I'm beginning to wonder if this has something to do with picamera's usage of a splitter after reading 6by9's fascinating comments in this thread. Stripping out the splitter isn't an option (too much functionality relies on it) but there's a possibility I might be able to make it optional (obviously in cases where it wasn't used things like multi-res recording and capture-while-recording wouldn't work).

I'll mark this for 1.9 to remind me to look into it.

@waveform80

This comment has been minimized.

Owner

waveform80 commented Sep 23, 2014

More information in this response too.

@mparkachov

This comment has been minimized.

mparkachov commented Sep 29, 2014

I came across the same issue. Currently I have model A + camera 1.3. For me even 640x480 doesn't work. The biggest resolution, which I can get working is 512x384. But when it is working, it seems to be stable. I used the same code as @rsacchettini to reproduce issue with file set to '/dev/null' to avoid interference with SD.

I don't know if this is relevant or not, but I can set camera.resolution to max values without problem, but I have to set then resize=(512,384) in start_recording() call.

@andyg24

This comment has been minimized.

andyg24 commented Oct 30, 2014

This is still an issue on a B+ board with latest firmware, although recording at 640x480 does work reliably.

Quick question -- how does MJPEG recording differ from continuous_capture(use_video_port=True) in JPEG format? My understanding is that MJPEG is just a concatenation of JPEG images with a few bytes of extra headers on each frame. continous_capture() seems to work reliably even at high resolutions. Would it be possible to emulate start_recording(format='mjpeg') using capture_continuous() under the hood? Would there be a performance penalty in this approach?

@andyg24

This comment has been minimized.

andyg24 commented Nov 1, 2014

A couple more observations:

  1. I downloaded the latest version of userland, modified RaspiVid.c to output in MJPEG format instead of H264 and ran some tests. Exactly the same thing happens there. Video encodes just fine at 640 x 480. If I change resolution to 800 x 600, the camera hangs and a reboot is required. So I doubt that eliminating the splitter in picamera will change anything -- the problem appears more fundamental than that.

  2. The problem is not resolution but the amount of data processed by the encoder. If I turn off the light in my room so that jpeg data compresses very well, 800 x 600 mode works just fine. As soon as I turn on the light, encoding fails.

@waveform80

This comment has been minimized.

Owner

waveform80 commented Nov 1, 2014

First of all, many thanks to @andyg24 for testing a modified RaspiVid.c - that's been on my todo list for weeks (as a means of determining whether removing the splitter will make a difference, and in order to have something to submit upstream). That's also a very interesting observation on whether compressibility is a factor in the crash; I wonder if @nikoder's success at 1080p with the V4L2 driver had anything to do with that?

I've also just realized I never responded to the comment above about whether picamera can control buffers, sorry! Via MMAL it can control the number and size of buffers allocated at each stage of the pipeline. At the moment it does what raspistill/raspivid do, which is to set them to the minimum or recommended size (the reason for minimum/recommended is there's a minor upstream bug: it's mentioned at the end of #73). Still, if circumstances demanded (i.e. if buffer allocation made a difference) I'd be happy to add a special case for MJPEG output.

@andyg24 - sounds like you've got plenty for an upstream bug report there, if you fancy opening one? That said, given 6by9's comments on the state of the MJPEG codec I wouldn't hold my breath waiting for a fix!

@andyg24

This comment has been minimized.

andyg24 commented Nov 1, 2014

Sure, I can file a bug with the userland project on github (is that the best way to report this problem upstream?)

I will share the steps to reproduce the issue here anyway.

  1. Download userland from github:

wget https://github.com/raspberrypi/userland/archive/master.zip

  1. Unzip the archive and go to userland-master/host_applications/linux/apps/raspicam

  2. Edit RaspiVid.c:

  • line 1384: change

encoder_output->format->encoding = MMAL_ENCODING_H264;
to
encoder_output->format->encoding = MMAL_ENCODING_MJPEG;

  1. Comment out setting of H264 video profile (lines 1465 - 1477)

  2. Compile. On Raspbian, this should work:

gcc -O3 -o mjpeg_test RaspiVid.c RaspiCamControl.c RaspiCLI.c RaspiPreview.c -I/opt/vc/include/ -I/opt/vc/include/interface/vcos/pthreads -I/opt/vc/include/interface/vmcs_host/linux -L/opt/vc/lib/ -lmmal -lmmal_components -lmmal_core -lmmal_util -lrt -lbcm_host -lvcos -lpthread

  1. Now, this works:

./mjpeg_test -w 640 -h 480 -fps 15 -o test.mjpeg

  1. This hangs the camera

./mjpeg_test -w 800 -h 600 -fps 15 -o /dev/null

@andyg24

This comment has been minimized.

andyg24 commented Nov 4, 2014

I've opened issue #208 in userland project:

raspberrypi/userland#208

There is not much activity in that project, and many bug reports haven't been acknowledged for months. I wouldn't hold my breath for this to be fixed quickly. Additionally, I don't see any MJPEG-related code in userland. Perhaps this functionality is part of the closed-sourced Broadcom driver?

@andyg24

This comment has been minimized.

andyg24 commented Nov 7, 2014

In the upstream discussion, a user suggested setting encoder buffer_size to a higher value. This seems to work: if I change line 465 in encoders.py to

self.output_port[0].buffer_size = 500000;

then MJPEG encoding seems to work at 1024 x 768 without wedging the camera. Don't know if this fix is reliable or if there are any downsides to a large buffer, but it's definitely an improvement.

Another suggestion in the upstream thread was to stop capture and wait for a few milliseconds before closing the encoder. I haven't tried this yet.

@andyg24

This comment has been minimized.

andyg24 commented Nov 10, 2014

#179 seems to do the trick. No lockups since I started testing this patch three days ago.

@waveform80

This comment has been minimized.

Owner

waveform80 commented Nov 29, 2014

PR #179 merged. Note to self: run the results through the test suite to see how far we can push it with the bigger buffer and then we can close this one!

@nikoder

This comment has been minimized.

nikoder commented Dec 19, 2014

That's also a very interesting observation on whether compressibility is a factor in the crash; I wonder if @nikoder's success at 1080p with the V4L2 driver had anything to do with that?

No, I don't think compressibility is the cause of my success with V4L2: although I don't have the best performance (since I'm streaming through the network and the default MJPG quality level does not give enough compression for that), it does work without locking up.

But I guess this is not so relevant, now that we know that the buffer size helps (sorry for my late response :S ).

I have hacked the buffer fix, which was merged, into my Picamera installation and that allowed me to get similar results as with the v4l2 driver (including at 1080p). I have only tested it once at that resolution though. It worked many times at 720p.

There were still some problems but they might be unrelated:

  • occasionally, it did not release the camera properly after recording
  • when I came back to do more tests with 1080p, it failed to start recording with any resolution (red light came on and then no data arrived, but killing the python process freed it up) -> probably something else going really wrong, but unfortunately it stopped me from doing further testing

waveform80 added a commit that referenced this issue Jan 1, 2015

Minor tweak for #47
Ensure that, if the recommended size ever exceeds our guesstimate of
512k, we accept that as the correct buffer size

waveform80 added a commit that referenced this issue Jan 1, 2015

Push the MJPEG tests to 1080p
With #47 fixed, these all seem to pass happily now

@waveform80 waveform80 closed this Jan 1, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment