Frames Are Displayed With Delay of One Frame

[Firionus]

How to Reproduce

• Start with commit 599e2a5
• pip install -r requirements.txt
• Modify camera.py to log which image is being shown. Also make the loop slower to make it easier to observe:

        while True:
            time.sleep(5)
            image_ind = int(time.time()) % 3
            print(f"yielding image {image_ind+1}", flush=True)
            yield Camera.imgs[image_ind]

• Run the server with python3 app.py
• Access the served page at localhost:5000 in Firefox (94)
• Observe the server logs and compare the number of the last served image with the image being shown in the browser

Observed Behavior

Annotated log dump (arrows show what can be seen in browser afterwards):

Starting camera thread.
yielding image 3 -> Browser shows nothing
yielding image 2 -> Browser shows image 2
yielding image 1 -> Browser still shows image 2
yielding image 3 -> Browser shows image 1
yielding image 2 -> Browser shows image 3

Observations:

• The first frame is never shown.
• The second frame is shown too long.
• Starting from the third frame, all images are only shown when the next one was sent. I.e., there is a 1 frame delay.

Expected Behavior

• The first frame should be shown.
• The second frame should only show for one tick.
• The successive frames should be shown shortly after being sent, not one frame late.

Reason for Unexpected Behavior

When capturing network traffic with Wireshark and looking at the content of the HTTP Continuation packets in question, we can see what's going on:

packet 1:

--frame 
Content-Type: image/jpeg 
 
···· ·JFIF ··· H H  ·· ·Created with GIMP·· C ·····················
···· 
[truncated JPEG]

packet 2:

--frame 
Content-Type: image/jpeg 
 
···· ·JFIF ··· H H  ·· ·Created with GIMP·· C ·····················
···· 
[truncated JPEG]

packet 3:

--frame 
Content-Type: image/jpeg 
 
···· ·JFIF ··· H H  ·· ·Created with GIMP·· C ·····················
···· 
[truncated JPEG]

(and then repeating again)

The issue is that the encapsulation boundary --frame only appears at the beginning of each message, so the browser/middleware can't be certain that the image is finished yet at the end of the messages, so they always waits for the next one.

Solution

The first message should start and end with an encapsulation boundary. The consecutive messages, should only have an encapsulation boundary at the end (see https://stackoverflow.com/a/65604493/15471654).

I will put in a PR shortly.

Further Chrome Bug

Even when you fix this issue in Firefox, Chrome will still show the frames one frame too late. This is due to https://bugs.chromium.org/p/chromium/issues/detail?id=1250396. The workaround mentioned only works for immediately displaying the first frame. In my testing, consecutive frames weren't be displayed properly in Chrome with the workaround.

The only functioning workaround I know is to send each image twice (https://stackoverflow.com/a/67506325/15471654).