Skip to content

image/jpeg: Unable to decode concatenated JPEGs (MIME-less "MJPEG") #22170

Open
@hnnsgstfssn

Description

@hnnsgstfssn

What did you do?

Wrapping ffmpeg -i <url> -c:v mjpeg -f image2pipe - in a exec.Cmd and continuously decoding the command output in a loop with jpeg.Decode fails while doing the same thing with ffmpeg -i <url> -c:v png -f image2pipe - and png.Decode works.

The following example illustrates the error by creating an MJPEG stream using a sample image: https://play.golang.org/p/lppyHZftWA

The same program but using a PNG stream and png.Decode shows successful decoding and a clean exit: https://play.golang.org/p/c54-hc6JRK

The following is a test that is expected to pass:

package test

import (
	"bytes"
	"image/jpeg"
	"image/png"
	"io"
	"testing"
)

// pngFrame is a PNG produced with
// `convert -size 1x1 xc:white png:- | xxd -c 1 -ps | sed -e 's/^/\\x/' |tr -d '\n'`
const pngFrame = "\x89\x50\x4e\x47\x0d\x0a\x1a\x0a\x00\x00\x00\x0d\x49\x48\x44\x52\x00\x00\x00\x01\x00\x00\x00\x01\x01\x00\x00\x00\x00\x37\x6e\xf9\x24\x00\x00\x00\x04\x67\x41\x4d\x41\x00\x00\xb1\x8f\x0b\xfc\x61\x05\x00\x00\x00\x20\x63\x48\x52\x4d\x00\x00\x7a\x26\x00\x00\x80\x84\x00\x00\xfa\x00\x00\x00\x80\xe8\x00\x00\x75\x30\x00\x00\xea\x60\x00\x00\x3a\x98\x00\x00\x17\x70\x9c\xba\x51\x3c\x00\x00\x00\x02\x62\x4b\x47\x44\x00\x01\xdd\x8a\x13\xa4\x00\x00\x00\x07\x74\x49\x4d\x45\x07\xe1\x0a\x06\x15\x06\x25\x79\xa4\x37\xa0\x00\x00\x00\x0a\x49\x44\x41\x54\x08\xd7\x63\x68\x00\x00\x00\x82\x00\x81\xdd\x43\x6a\xf4\x00\x00\x00\x25\x74\x45\x58\x74\x64\x61\x74\x65\x3a\x63\x72\x65\x61\x74\x65\x00\x32\x30\x31\x37\x2d\x31\x30\x2d\x30\x36\x54\x32\x31\x3a\x30\x36\x3a\x33\x37\x2b\x30\x31\x3a\x30\x30\xeb\x24\x00\x4a\x00\x00\x00\x25\x74\x45\x58\x74\x64\x61\x74\x65\x3a\x6d\x6f\x64\x69\x66\x79\x00\x32\x30\x31\x37\x2d\x31\x30\x2d\x30\x36\x54\x32\x31\x3a\x30\x36\x3a\x33\x37\x2b\x30\x31\x3a\x30\x30\x9a\x79\xb8\xf6\x00\x00\x00\x00\x49\x45\x4e\x44\xae\x42\x60\x82"

// jpegFrame is a JPEG frame produced with
// `convert -size 1x1 xc:white jpg:- | xxd -c 1 -ps | sed -e 's/^/\\x/' |tr -d '\n'`
const jpegFrame = "\xff\xd8\xff\xe0\x00\x10\x4a\x46\x49\x46\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00\xff\xdb\x00\x43\x00\x03\x02\x02\x02\x02\x02\x03\x02\x02\x02\x03\x03\x03\x03\x04\x06\x04\x04\x04\x04\x04\x08\x06\x06\x05\x06\x09\x08\x0a\x0a\x09\x08\x09\x09\x0a\x0c\x0f\x0c\x0a\x0b\x0e\x0b\x09\x09\x0d\x11\x0d\x0e\x0f\x10\x10\x11\x10\x0a\x0c\x12\x13\x12\x10\x13\x0f\x10\x10\x10\xff\xc0\x00\x0b\x08\x00\x01\x00\x01\x01\x01\x11\x00\xff\xc4\x00\x14\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x09\xff\xc4\x00\x14\x10\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xff\xda\x00\x08\x01\x01\x00\x00\x3f\x00\x54\xdf\xff\xd9"

func TestMultiPngDecode(t *testing.T) {

	var expected int = 3 // expected success count
	var got int          // decoded frame count
	var no int           // frame number

	stream := bytes.NewBuffer(bytes.Repeat([]byte(pngFrame), expected))

	for no = 1; no <= expected; no++ {
		_, err := png.Decode(stream)
		if err != nil {
			if err != io.EOF {
				t.Errorf("Unexpected error when decoding frame #%d: %s", no, err)
			}
			break
		}
		got++
	}

	if expected != got {
		t.Errorf("Expected %d decoded frames, got %d", expected, got)
	}
}

func TestMjpegDecode(t *testing.T) {

	var expected int = 3 // expected success count
	var got int          // decoded frame count
	var no int           // frame number

	stream := bytes.NewBuffer(bytes.Repeat([]byte(jpegFrame), expected))

	for no = 1; no <= expected; no++ {
		_, err := jpeg.Decode(stream)
		if err != nil {
			if err != io.EOF {
				t.Errorf("Unexpected error when decoding frame #%d: %s", no, err)
			}
			break
		}
		got++
	}

	if expected != got {
		t.Errorf("Expected %d decoded frames, got %d", expected, got)
	}
}

Grabbing a number of frames from a video and inspecting the output we can verify that there's no unexpected bytes after the EOI marker (ff d9) and before the next SOI marker (ff d8) and that we get the expected number of frames:

ffmpeg -v quiet -y -i <video> -frames:v 3 -c:v mjpeg -f image2pipe - | hexdump | grep "ff d8"
0000000 ff d8 ff e0 00 10 4a 46 49 46 00 01 02 00 00 01
00054d0 14 9e 80 7f ff d9 ff d8 ff e0 00 10 4a 46 49 46
000ca60 ad d0 2b 8a 29 69 29 68 7a 05 cf ff d9 ff d8 ff

ffmpeg -v quiet -y -i <video> -frames:v 3 -c:v mjpeg -f image2pipe - | hexdump | grep "ff d9"
00054d0 14 9e 80 7f ff d9 ff d8 ff e0 00 10 4a 46 49 46
000ca60 ad d0 2b 8a 29 69 29 68 7a 05 cf ff d9 ff d8 ff
0014180 ff d9

What did you expect to see?

The MJPEG stream is a concatenation of JPEG images and the decoder should be able to successfully decode multiple images in sequence without forcing caller to keep track of and seek to next frame.

The program https://play.golang.org/p/lppyHZftWA is expected to not give any output but decode all stream frames correctly and exit cleanly when hitting io.EOF

What did you see instead?

The program prints an error message as it hits io.ErrUnexpectedEOF

Does this issue reproduce with the latest release (go1.9.1)?

Yes

System details

go version go1.9 darwin/amd64
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/hnnsgstfssn/go"
GORACE=""
GOROOT="/usr/local/Cellar/go/1.9/libexec"
GOTOOLDIR="/usr/local/Cellar/go/1.9/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/md/xyw1fqr954lfbtnfc_9ym_0r0000gn/T/go-build236075526=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="0"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOROOT/bin/go version: go version go1.9 darwin/amd64
GOROOT/bin/go tool compile -V: compile version go1.9
uname -v: Darwin Kernel Version 16.6.0: Fri Apr 14 16:21:16 PDT 2017; root:xnu-3789.60.24~6/RELEASE_X86_64
ProductName:	Mac OS X
ProductVersion:	10.12.5
BuildVersion:	16F73
lldb --version: lldb-370.0.42
  Swift-3.1

Metadata

Metadata

Assignees

No one assigned

    Labels

    NeedsInvestigationSomeone must examine and confirm this is a valid issue and not a duplicate of an existing one.

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions