Add default Huffman table if huffman table uninitialized #493

kim-mishra · 2023-04-11T14:02:19Z

Description

Certain MJPEG frames do not include Huffman tables, this causes an uninitialized Huffman table error during jpeg.Decode(). These changes add a default Huffman Table in the case that one was not provide, so that the jpeg can be processed and transformed into an image. I tested these changes locally with cameras that provide and do not provide Huffman tables to ensure that the error is avoided, and for frames with initialized Huffman tables there is no change in behavior.

Reference issue

Fixes an uninitialized Huffman error that is returned from jpeg.Decode() when a jpeg Frame omits the Huffman Table

edaniels

nice fix! can we get some tests in for this behavior. I mentioned one specifically in the comments to prevent a pathological case.

edaniels · 2023-04-11T18:33:38Z

pkg/frame/compressed.go

 func decodeMJPEG(frame []byte, width, height int) (image.Image, func(), error) {
 	img, err := jpeg.Decode(bytes.NewReader(frame))
+
+	if err != nil && err.Error() == "invalid JPEG format: uninitialized Huffman table" {


can you return early if there is no error?
also, it would be better to use errors.As with https://cs.opensource.google/go/go/+/refs/tags/go1.20.3:src/image/jpeg/reader.go;l=18 to then check just the "uninitialized Huffman table" error so we can be a bit more specific

also, it would be better to use errors.As with https://cs.opensource.google/go/go/+/refs/tags/go1.20.3:src/image/jpeg/reader.go;l=18 to then check just the "uninitialized Huffman table" error so we can be a bit more specific

Could you explain this a little more? We can check that its a FormatError but still need to check the contents of the message to make sure it isn't a different type of FormatError. Also, how can I extract only "uninitialized Huffman table", without the "invalid JPEG format: " prefix added in the function?

we do but we can eliminate the first part of that string to check

edaniels · 2023-04-11T18:34:26Z

pkg/frame/compressed.go

@@ -6,7 +6,25 @@ import (
 	"image/jpeg"
 )

+// Thank you to https://github.com/filiptc/gorbit/blob/fa87ff39b68a6706306f34c318e0b9a5a3c97110/image/overlay.go#L37-L40 for addMotionDht, dhtMarker, dht, and sosMarker


licensing checks out 👍

Strictly speaking, when OSS licensed code is copied, we should include the full original license statements according to

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

in case of MIT.
https://github.com/filiptc/gorbit/blob/fa87ff39b68a6706306f34c318e0b9a5a3c97110/LICENSE

edaniels · 2023-04-11T18:37:19Z

pkg/frame/compressed.go

+
+func addMotionDht(frame []byte) []byte {
+	jpegParts := bytes.Split(frame, sosMarker)
+	return append(jpegParts[0], append(dhtMarker, append(dht, append(sosMarker, jpegParts[1]...)...)...)...)


this should check that the split worked. otherwise a malicious input could crash the caller. this would be a good test

edaniels · 2023-04-11T18:38:29Z

pkg/frame/compressed.go

+
+func addMotionDht(frame []byte) []byte {
+	jpegParts := bytes.Split(frame, sosMarker)
+	return append(jpegParts[0], append(dhtMarker, append(dht, append(sosMarker, jpegParts[1]...)...)...)...)


this is copying bytes in a potentially expensive way. can you make the slice in advance with the new size and then copy in order. it will also make these appends a tad easier to read!

at-wat · 2023-04-12T02:36:06Z

pkg/frame/compressed.go

+
+var (
+	dhtMarker = []byte{255, 196}
+	dht       = []byte{1, 162, 0, 0, 1, 5, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 1, 0, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 16, 0, 2, 1, 3, 3, 2, 4, 3, 5, 5, 4, 4, 0, 0, 1, 125, 1, 2, 3, 0, 4, 17, 5, 18, 33, 49, 65, 6, 19, 81, 97, 7, 34, 113, 20, 50, 129, 145, 161, 8, 35, 66, 177, 193, 21, 82, 209, 240, 36, 51, 98, 114, 130, 9, 10, 22, 23, 24, 25, 26, 37, 38, 39, 40, 41, 42, 52, 53, 54, 55, 56, 57, 58, 67, 68, 69, 70, 71, 72, 73, 74, 83, 84, 85, 86, 87, 88, 89, 90, 99, 100, 101, 102, 103, 104, 105, 106, 115, 116, 117, 118, 119, 120, 121, 122, 131, 132, 133, 134, 135, 136, 137, 138, 146, 147, 148, 149, 150, 151, 152, 153, 154, 162, 163, 164, 165, 166, 167, 168, 169, 170, 178, 179, 180, 181, 182, 183, 184, 185, 186, 194, 195, 196, 197, 198, 199, 200, 201, 202, 210, 211, 212, 213, 214, 215, 216, 217, 218, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 17, 0, 2, 1, 2, 4, 4, 3, 4, 7, 5, 4, 4, 0, 1, 2, 119, 0, 1, 2, 3, 17, 4, 5, 33, 49, 6, 18, 65, 81, 7, 97, 113, 19, 34, 50, 129, 8, 20, 66, 145, 161, 177, 193, 9, 35, 51, 82, 240, 21, 98, 114, 209, 10, 22, 36, 52, 225, 37, 241, 23, 24, 25, 26, 38, 39, 40, 41, 42, 53, 54, 55, 56, 57, 58, 67, 68, 69, 70, 71, 72, 73, 74, 83, 84, 85, 86, 87, 88, 89, 90, 99, 100, 101, 102, 103, 104, 105, 106, 115, 116, 117, 118, 119, 120, 121, 122, 130, 131, 132, 133, 134, 135, 136, 137, 138, 146, 147, 148, 149, 150, 151, 152, 153, 154, 162, 163, 164, 165, 166, 167, 168, 169, 170, 178, 179, 180, 181, 182, 183, 184, 185, 186, 194, 195, 196, 197, 198, 199, 200, 201, 202, 210, 211, 212, 213, 214, 215, 216, 217, 218, 226, 227, 228, 229, 230, 231, 232, 233, 234, 242, 243, 244, 245, 246, 247, 248, 249, 250}


Is this data standardized or specific to some device of some vendor?

This should work across devices in the case that the Huffman encoding info has been omitted for bandwidth optimization

Can you point where this comes from though?

The JPEG wiki lists dhtMarker & sosMarker as dht & sos respectively.
This RFC discusses a "Standard Huffman Table" and location of the citation is behind a paywall. This thread is where I got the table from. It works on cameras I have tested with and has been reported to work on other cameras as well.

@at-wat is that cool with you?

This RFC discusses a "Standard Huffman Table" and location of the citation is behind a paywall.

https://www.w3.org/Graphics/JPEG/itu-t81.pdf
W3C seems putting it in public.

kim-mishra · 2023-04-17T13:54:23Z

pkg/frame/compressed.go

+/* addMotionDht, dhtMarker, dht, and sosMarker is protected under the following license:
+
+The MIT License (MIT)
+
+Copyright (c) 2016 Philip Thomas Casado
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+*/


@at-wat like this?

Maybe better to move the copied part to separated file (if code inside decodeMJPEG and addMotionDht is also copied, current structure is fine)
Could you add a URL of the original repo as well?

Other things looks good to me!

Link is added! parts of the functions you mentioned are taken from the repo

edaniels

Done with another pass. Thanks for adding tests!

edaniels · 2023-04-17T14:15:19Z

pkg/frame/compressed.go

+		return img, func() {}, err
+	}
+
+	var target jpeg.FormatError = jpeg.FormatError("uninitialized Huffman table")


this can be pulled outside of this function so it's not recomptued

edaniels · 2023-04-17T14:16:39Z

pkg/frame/compressed.go

+	var target jpeg.FormatError = jpeg.FormatError("uninitialized Huffman table")
+	if errors.As(err, &target) {
+		if err.Error() == target.Error() {
+			img, err = jpeg.Decode(bytes.NewReader(addMotionDht(frame)))


this should check if the error is not nil and return the error. Otherwise it can return the image. It's a good practice to not return any potentially initialized values (img) in the case of failure cases.

Could that break other things? The function did not originally ever return nil

it would not

edaniels · 2023-04-17T14:17:04Z

pkg/frame/compressed.go

+	correctedFrame := make([]byte, len(jpegParts[0])+len(dhtMarker)+len(dht)+len(sosMarker)+len(jpegParts[1]))
+	correctedFrameOffset := 0
+
+	for indx, item := range jpegParts[0] {


indx -> idx for each of these

edaniels · 2023-04-17T14:17:28Z

pkg/frame/compressed.go

+	if len(jpegParts) != 2 {
+		return frame
+	}
+	correctedFrame := make([]byte, len(jpegParts[0])+len(dhtMarker)+len(dht)+len(sosMarker)+len(jpegParts[1]))


len(dhtMarker)+len(dht)+len(sosMarker) can be calculated once outside of this function which will simplify reading this

edaniels · 2023-04-17T14:18:12Z

pkg/frame/compressed.go

+	correctedFrameOffset += len(jpegParts[0])
+
+	for indx, item := range dhtMarker {
+		correctedFrame[indx+correctedFrameOffset] = item


slightly more readable to have it be the offset + the idx for each of these

edaniels · 2023-04-17T14:19:34Z

pkg/frame/compressed.go

+	correctedFrameOffset := 0
+
+	for indx, item := range jpegParts[0] {
+		correctedFrame[indx] = item


instead of these loops I think you can do copy(correctedFrame[offset:], jpegParts[0]) for each of these. that should make things smaller and simple to read!

edaniels · 2023-04-17T14:19:59Z

pkg/frame/compressed_test.go

+	// Decode fails with an uninitialized Huffman table error for sample input
+	expectedErrorMessage := "invalid JPEG format: uninitialized Huffman table"
+	if err.Error() != expectedErrorMessage {
+		t.Errorf("Wrong decode error result,\nexpected:\n%+v\ngot:\n%+v", expectedErrorMessage, err)


I believe you need to use FailNow as well

edaniels · 2023-04-17T14:50:47Z

pkg/frame/compressed.go

+	correctedFrame := make([]byte, len(jpegParts[0])+huffmanTableInfoLength+len(jpegParts[1]))
+	correctedFrameOffset := 0
+
+	copy(correctedFrame[correctedFrameOffset:], jpegParts[0])


so much better!

edaniels · 2023-04-17T14:51:36Z

pkg/frame/compressed_test.go

+	// Decode passes after adding default Huffman table to
+	defaultHuffmanTableFrame, err := jpeg.Decode(bytes.NewReader(addMotionDht(UninitializedHuffmanTable)))
+	if err != nil {
+		t.Errorf("Expected decode function to pass after adding default Huffman table. Failed with %v\n", err)


all of these need the failnow too but you can just do Fatalf instead

kim-mishra · 2023-04-17T14:54:44Z

pkg/frame/compressed_test.go

+func TestAddMotionDht(t *testing.T) {
+	uninitializedHuffmanTableFrame, err := jpeg.Decode(bytes.NewReader(UninitializedHuffmanTable))
+
+	// Decode fails with an uninitialized Huffman table error for sample input
+	expectedErrorMessage := "invalid JPEG format: uninitialized Huffman table"
+	if err.Error() != expectedErrorMessage {
+		t.Fatalf("Wrong decode error result,\nexpected:\n%+v\ngot:\n%+v", expectedErrorMessage, err)
+	}
+
+	// Decode passes after adding default Huffman table to
+	defaultHuffmanTableFrame, err := jpeg.Decode(bytes.NewReader(addMotionDht(UninitializedHuffmanTable)))
+	if err != nil {
+		t.Fatalf("Expected decode function to pass after adding default Huffman table. Failed with %v\n", err)
+	}
+
+	// Adding default Huffman table to a valid frame without a Huffman table changes the table
+	if uninitializedHuffmanTableFrame == defaultHuffmanTableFrame {
+		t.Fatalf("Expected addMotionDht to update frame. Instead returned original frame")
+	}
+
+	// Check that an improperly constructed frame does not get updated by addMotionDht
+	randomBytes := []byte{1, 2, 3, 4}
+	frame1, err := jpeg.Decode(bytes.NewReader(randomBytes))
+	if err == nil {
+		t.Fatalf("Expected decode function to fail with random bytes but passed.")
+	}
+
+	frame2, err := jpeg.Decode(bytes.NewReader(addMotionDht(randomBytes)))
+	if err == nil {
+		t.Fatalf("Expected decode function to fail with random bytes but passed.")
+	}
+
+	if frame1 != frame2 {
+		t.Fatalf("addMotionDht updated the frame despite being improperly constructed")
+	}
+}
+
+func TestDecodeMJPEG(t *testing.T) {
+	_, _, err := decodeMJPEG(UninitializedHuffmanTable, 640, 480)
+	if err != nil {
+		t.Fatalf("Expected decode function to pass. Failed with %v\n", err)
+	}
+}


@edaniels like this?

edaniels

LGTM. Will wait on @at-wat for other approval

at-wat

LGTM!

codecov · 2023-04-18T03:45:00Z

Codecov Report

Patch coverage: 83.87% and project coverage change: +0.25 🎉

Comparison is base (bf290b0) 58.85% compared to head (c8312f9) 59.11%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #493      +/-   ##
==========================================
+ Coverage   58.85%   59.11%   +0.25%     
==========================================
  Files          62       62              
  Lines        3753     3784      +31     
==========================================
+ Hits         2209     2237      +28     
  Misses       1416     1416              
- Partials      128      131       +3

Impacted Files	Coverage Δ
pkg/frame/compressed.go	`82.35% <83.87%> (+82.35%)`	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

check if huffman table error occurred, and if so add in necessary data

cff0a8f

kim-mishra changed the title ~~check if huffman table error occurred, and if so add in necessary data~~ Add default Huffman table if huffman table uninitialized Apr 11, 2023

edaniels self-requested a review April 11, 2023 18:30

edaniels requested changes Apr 11, 2023

View reviewed changes

addressing PR reviews

ad4bc37

at-wat reviewed Apr 12, 2023

View reviewed changes

check if error is format error

2f64779

kim-mishra requested a review from edaniels April 12, 2023 14:23

Merge branch 'master' into uninitialized-huffman-table-fix

67353f1

kim-mishra requested a review from at-wat April 13, 2023 14:18

Update compressed.go

0731cb4

kim-mishra commented Apr 17, 2023

View reviewed changes

edaniels requested changes Apr 17, 2023

View reviewed changes

addressing PR reviews

f99d1a0

kim-mishra requested a review from edaniels April 17, 2023 14:44

edaniels reviewed Apr 17, 2023

View reviewed changes

edaniels requested changes Apr 17, 2023

View reviewed changes

t.Errorf -> t.Fatalf

a3980d3

kim-mishra commented Apr 17, 2023

View reviewed changes

Merge branch 'master' into uninitialized-huffman-table-fix

6a535f4

kim-mishra requested a review from edaniels April 17, 2023 16:27

edaniels approved these changes Apr 17, 2023

View reviewed changes

add link to comment

c8312f9

at-wat approved these changes Apr 18, 2023

View reviewed changes

edaniels merged commit 52a080b into pion:master Apr 18, 2023
8 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add default Huffman table if huffman table uninitialized #493

Add default Huffman table if huffman table uninitialized #493

kim-mishra commented Apr 11, 2023 •

edited

edaniels left a comment

edaniels Apr 11, 2023

kim-mishra Apr 11, 2023

edaniels Apr 11, 2023

edaniels Apr 11, 2023

at-wat Apr 17, 2023

edaniels Apr 11, 2023

edaniels Apr 11, 2023

at-wat Apr 12, 2023

kim-mishra Apr 12, 2023

edaniels Apr 12, 2023

kim-mishra Apr 12, 2023

edaniels Apr 12, 2023

at-wat Apr 17, 2023

kim-mishra Apr 17, 2023

at-wat Apr 18, 2023

kim-mishra Apr 18, 2023

edaniels left a comment

edaniels Apr 17, 2023

edaniels Apr 17, 2023

kim-mishra Apr 17, 2023

edaniels Apr 17, 2023

edaniels Apr 17, 2023

edaniels Apr 17, 2023

edaniels Apr 17, 2023

edaniels Apr 17, 2023

edaniels Apr 17, 2023

edaniels Apr 17, 2023

edaniels Apr 17, 2023

kim-mishra Apr 17, 2023

edaniels Apr 17, 2023

edaniels left a comment

at-wat left a comment

codecov bot commented Apr 18, 2023

Add default Huffman table if huffman table uninitialized #493

Add default Huffman table if huffman table uninitialized #493

Conversation

kim-mishra commented Apr 11, 2023 • edited

Description

Reference issue

edaniels left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

edaniels left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

edaniels left a comment

Choose a reason for hiding this comment

at-wat left a comment

Choose a reason for hiding this comment

codecov bot commented Apr 18, 2023

Codecov Report

kim-mishra commented Apr 11, 2023 •

edited