Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HTML: Add test for <video> dispatching resize event and displaying variable video track width and height #17821

Open
guest271314 opened this issue Jul 13, 2019 · 32 comments · May be fixed by #19030
Labels

Comments

@guest271314
Copy link
Contributor

@guest271314 guest271314 commented Jul 13, 2019

Consider a WebM video where the input video track contains variable width and height (differing pixel dimensions)

<!DOCTYPE html>
<html>
  <head>
    <title>Chromium bug - HTMLVideoElement does not resize or display correct videoWidth or videoHeight when source media contains video tracks having variable width and height</title>
  </head>
  <body>
    <video controls autoplay muted src="" onresize="console.log(this.videoWidth, this.videoHeight)"></video>
    <script>
      fetch("https://gist.githubusercontent.com/guest271314/6a024dd09db307bbb7fe7be718070364/raw/8bbd9cd34a15ef5ea3ffc29778abc33780f75f0a/variable_width_height_webm_input.txt")
      .then(response => response.text())
      .then(WebMdataURL => 
        document.querySelector("video").src = WebMdataURL
      ).catch(console.error)
    </script>
  </body>
</html>

Firefox 70 dispatches resize event three times, once for the initial width and height and once each for the subsequent different pixel dimensions encoded into the WebM file and displays the correct videoWidth and videoHeight corresponding to the variable input pixel dimensions.

Chromium 77 dispatches resize event one time, once for the initial width and height of the video track and does not display the correct videoWidth and videoHeight corresponding to the variable input pixel dimensions.

Add tests for HTMLVideoElement to verify that

  1. resize event is dispatched for each change to width and height of video track of media file; and
  2. <video> element displays the correct videoWidth and videoHeight corresponding the potentially variable input pixel dimensions encoded within the media file.
@guest271314

This comment has been minimized.

Copy link
Contributor Author

@guest271314 guest271314 commented Sep 2, 2019

@guest271314

This comment has been minimized.

Copy link
Contributor Author

@guest271314 guest271314 commented Sep 2, 2019

@foolip How to determine which specification owns resizing of HTMLVideoElement?

@foolip

This comment has been minimized.

Copy link
Contributor

@foolip foolip commented Sep 2, 2019

@guest271314 that's defined in https://html.spec.whatwg.org/multipage/media.html, searcg for "named resize" to find the two places the event is dispatched.

One way to find the relevant spec is by following spec links at the bottom of MDN pages, in this case https://developer.mozilla.org/en-US/docs/Web/API/HTMLVideoElement.

@foolip foolip added the html label Sep 2, 2019
@foolip

This comment has been minimized.

Copy link
Contributor

@foolip foolip commented Sep 2, 2019

Regarding the topic at hand, I didn't know it was even possible to construct a WebM video with variable dimensions, since the width and height is part of the metadata in the beginning of the file. It is of course possible to put video of a different size in the file and for the size to change at any keyframe, but is this a valid WebM file?

Ogg is a format where the up-front metadata is optional and which I think would also trigger this problem.

@guest271314

This comment has been minimized.

Copy link
Contributor Author

@guest271314 guest271314 commented Sep 2, 2019

@foolip Yes, the WebM file is valid, in brief see https://gitlab.com/mbunkus/mkvtoolnix/issues/2582#note_189520756; https://mkvtoolnix.download/doc/NEWS.md

Version 36.0.0 "Is That Jazz?" 2019-08-10

New features and enhancements

  • mkvmerge: mkvmerge now allows appending AV1, VP8, VP9, H.264/AVC and
    H.265/HEVC tracks whose pixel dimensions differ. Implements #2582.

It is argued that H264 and AVC1 codecs are invalid for WebM files, though that has not stopped Chromium authors from implemented the codec for MediaRecorder https://bugs.chromium.org/p/chromium/issues/detail?id=980822.

In fact, the only way that have been able to get Chromium, using a WebM file encoded at Chromium, to display variable pixel dimensions at Chromium is to use H264 or AVC1 codecs (openh264).

@foolip

This comment has been minimized.

Copy link
Contributor

@foolip foolip commented Sep 2, 2019

Alright, I would have thought that producing a video stream like that using VP8 or VP9 would be straightforward, but maybe the decoder rejects it?

@guest271314

This comment has been minimized.

Copy link
Contributor Author

@guest271314 guest271314 commented Sep 2, 2019

@foolip The issue is that Chromium does not display the variable pixel dimension video at HTML <video> that its own implementation of MediaRecorder outputs.

This https://plnkr.co/edit/Axkb8s?p=info is the only code that have been able to compose which outputs variable pixel dimensions at HTML <video> element at Chromium browser https://plnkr.co/edit/Axkb8s?p=info. Since the codecs are "non-standard" for a WebM file and Mozilla does not currently decode Matroska files the video does not play at Firefox or Nightly.

This demonstrates the issue with HTML <video> element at Chromium even when the input video is VP8 or VP9 and srcObject of a <video> is set to a MediaStream where the media source is a video created by libwebm https://plnkr.co/edit/sMA653?p=info; et. al.

If you follow the links in the issues you will see that it is not immediately clear which specification owns and is responsible for HTML <video> element resizing.

The decoder at Chromium apparently only reads the initial frame dimensions when the codec is VP8 or VP9, though not for H264 or AVC1.

Again, when trying to determine which specification owns and controls resizing of HTML <video> element (to file issues at the appropriate specification, where am not banned from, in order to mandate that the <video> is resized), the answer is not immediately clear.

@guest271314

This comment has been minimized.

Copy link
Contributor Author

@guest271314 guest271314 commented Sep 2, 2019

@foolip Re

I didn't know it was even possible to construct a WebM video with variable dimensions

see https://github.com/guest271314/native-messaging-mkvmerge.

@guest271314

This comment has been minimized.

Copy link
Contributor Author

@guest271314 guest271314 commented Sep 2, 2019

@foolip Interestingly Chromium does display variable pixel dimensions and dispatch resize event for a WebM video recorded with MediaRecorder at Mozilla with codecs set to VP8. Diving in to Chromium source code there are sections which evaluate pixel dimensions and frame rate https://cs.chromium.org/chromium/src/third_party/blink/renderer/modules/mediarecorder/media_recorder_handler.cc?dr=CSs&g=0&l=38 (see also https://bugs.chromium.org/p/webm/issues/detail?id=1642.) which could explain (though not sure exactly how) the output at https://plnkr.co/edit/4JxS4O?p=info using webm-writer.js
https://github.com/thenickdude/webm-writer-js.

The consequence is that videoWidth and videoHeight of HTML <video> element (and MedisStreamTrack of kind "video" width and height from getSettings(), to the extent applicable where getSettings() is defined for captureStream() in specifications that use and refere toMediaStream though are not specifically Media Capture and Streams https://bugzilla.mozilla.org/show_bug.cgi?id=1537986) values are not accurate relevant to the actual underlying encoded frame during playback at Chromium, e.g., https://plnkr.co/edit/uCnx1R?p=info.

It has been somewhat challenging to determine which specification and implemented API is responsible for the issue of resizing HTML <video> element at Chromium; MediaRecorder; HTML <video> element; or CSS?

Presently am not certain by reading the current iteration of the relevant specifications; even the new ResizeObserver API does not explicitly state that it owns resizing HTML elements; i.e., from w3c/csswg-drafts#4266

@foolip

This comment has been minimized.

Copy link
Contributor

@foolip foolip commented Sep 2, 2019

It's the resize event in https://html.spec.whatwg.org/multipage/media.html that's relevant here. HTML together with container format specs like WebM are the specs that define what should happen.

If both WebM and MPEG-4 support resizing video streams, then adding a test for this would be straightforward.

@guest271314

This comment has been minimized.

Copy link
Contributor Author

@guest271314 guest271314 commented Sep 2, 2019

@foolip

It's the resize event in https://html.spec.whatwg.org/multipage/media.html that's relevant here.

Do we disregard or edit the language at MDN https://developer.mozilla.org/en-US/docs/Web/API/Window/resize_event?

However, resize events are only fired on the window object (i.e. returned by document.defaultView). Only handlers registered on the window object will receive resize events.

Am currently banned (evidently indefinitely) from WHATWG. Am still willing to help or write the tests if a member of WPT files the PR.

@foolip

This comment has been minimized.

Copy link
Contributor

@foolip foolip commented Sep 2, 2019

Yes, https://developer.mozilla.org/en-US/docs/Web/API/Window/resize_event should be disregarded for two reasons. First, its concerns the resize event fired at the window object, which isn't defined in the same place as the resize event fired at media elements. Second, nothing MDN says has any bearing on what browsers should implement or what wpt should test, since it's documentation intended for web developers.

If you can produce media files for this, then getting a test into wpt should be straightforward.

guest271314 added a commit to guest271314/MediaFragmentRecorder that referenced this issue Sep 2, 2019
HTML: Add test for <video> dispatching resize event and displaying variable video track width and height #17821
web-platform-tests/wpt#17821

All files produced using the code at https://github.com/guest271314/MediaFragmentRecorder/blob/webrtc-replacetrack/MediaFragmentRecorder.html

The codecs used are

- "video/webm;codecs=vp8,opus"
- "video/webm;codecs=vp9,opus"
- "video/webm;codecs=h264"
- "video/x-matroska;codecs=h264" (see https://bugs.chromium.org/p/chromium/issues/detail?id=999580; https://plnkr.co/edit/WUVbjz?p=info)

Width and height of the encoded frames in the files in order

WidthxHeight

768x576
480x240 
640x360
400x300
1280x720
@guest271314

This comment has been minimized.

Copy link
Contributor Author

@guest271314 guest271314 commented Sep 2, 2019

@foolip

If you can produce media files for this, then getting a test into wpt should be straightforward.

guest271314/MediaFragmentRecorder@ca44c12

@foolip

This comment has been minimized.

Copy link
Contributor

@foolip foolip commented Sep 2, 2019

I guess MediaRecorder only supports WebM/Matroska, but is it possible to have an MPEG-4 container with video that changes its size mid-stream?

The ideal test video files would be one that is red and 400x300 for 1 second, and then green and 300x150 for 1 second. The exact size isn't very important, but something smallish is good. The size of the videos in bytes should be as small as possible, it should be possible to get them below 100kb I think?

Large test files makes the tests slower and makes it more likely that failures are due to something on the network side rather than the media decoding pipeline.

@guest271314

This comment has been minimized.

Copy link
Contributor Author

@guest271314 guest271314 commented Sep 2, 2019

I guess MediaRecorder only supports WebM/Matroska, but is it possible to have an MPEG-4 container with video that changes its size mid-stream?

Cannot fathom why that would not be possible. Would that be dissimilar from the Matroska container width H264 (openh264) codec at the previous link?

The ideal test video files would be one that is red and 400x300 for 1 second, and then green and 300x150 for 1 second. The exact size isn't very important, but something smallish is good. The size of the videos in bytes should be as small as possible, it should be possible to get them below 100kb I think?

Created the various files that have previously tested to demonstrate the issue with interoperability. The variation of the media files and display of those files at HTML <video> element output by the browsers. There are workarounds to acheieve the expected result at both browsers, though as yet
not using the same code. Use them or not as the individual test description requires. The plnkr https://plnkr.co/edit/uCnx1R?p=preview at the linked Chromium issue https://bugs.chromium.org/p/chromium/issues/detail?id=972470 provides a means to create the file chromium_mediarecorder_1s_red_1s_green_variable_pixel_dimensions.webm.zip

@guest271314

This comment has been minimized.

Copy link
Contributor Author

@guest271314 guest271314 commented Sep 2, 2019

@foolip

      const imageData = [{
        width: 400,
        height: 300,
        fillStyle: "red"
      }, {
        width: 300,
        height: 150,
        fillStyle: "green"
      }];
@guest271314

This comment has been minimized.

Copy link
Contributor Author

@guest271314 guest271314 commented Sep 9, 2019

Compare the output of
firefox_mediarecorder_1s_red_1s_green_variable_pixel_dimensions.webm.zip at Chromium to the output of the WebM file produced by the same code at Chromium. The WebM file produced at Chromium does not resize at HTML <video> element at Chromium, does resize at Firefox. The WebM file produced Firefox resizes HTML <video> element at Chromium and Firefox.

@foolip

This comment has been minimized.

Copy link
Contributor

@foolip foolip commented Sep 12, 2019

There are a lot of video files flying around now :) Is there a pair of small video files, one WebM and the other something that Safari supports, that could be used to test the resize event behavior? Please point directly to which ones you're proposing.

@guest271314

This comment has been minimized.

Copy link
Contributor Author

@guest271314 guest271314 commented Sep 12, 2019

The WebM with VP8 codec file is in the .zip archive at previous post.
chromium_mediarecorder_1s_red_1s_green_variable_pixel_dimensions.mp4.zip. Do not have access to Safari to test playback. The video was recorded at Chromium using h264 codec the converted to MP4 using this command ffmpeg -i video.webm -c:v copy video.mp4 (Converting WebM to MP4 Using FFmpeg).

@guest271314

This comment has been minimized.

Copy link
Contributor Author

@guest271314 guest271314 commented Sep 12, 2019

This was recorded at Firefox https://github.com/web-platform-tests/wpt/files/3588824/firefox_mediarecorder_1s_red_1s_green_variable_pixel_dimensions.webm.zip and should dispatch resize event at Chromium. The file recorded at Chromium (at this post #17821 (comment)) currently will not dispatch resize event at Chromium (TL;DR https://bugs.chromium.org/p/chromium/issues/detail?id=1001948).

foolip added a commit that referenced this issue Sep 12, 2019
Fixes #17821.

New media files are from that issue.
@foolip

This comment has been minimized.

Copy link
Contributor

@foolip foolip commented Sep 12, 2019

Cool, thanks! Putting that in a video element and playing in Chrome, I actually do see resize events. I don't think this is tested in wpt however, so turning it into a test is a good idea. I don't have time to finish it now, but #19030 is the structure for a test.

foolip added a commit that referenced this issue Sep 13, 2019
Fixes #17821.

New media files are from that issue.
@foolip

This comment has been minimized.

Copy link
Contributor

@foolip foolip commented Sep 13, 2019

Alright, I finished #19030, but it seems to work in Chrome, so not covering what you'd hoped for I think.

@guest271314

This comment has been minimized.

Copy link
Contributor Author

@guest271314 guest271314 commented Sep 13, 2019

@foolip

but it seems to work in Chrome, so not covering what you'd hoped for I think.

Do not entertain "hope".

The expected result is for the video produced by Chromium to resize at Chromium.

You must have used the WebM file recorded at Firefox for the test, correct?

@guest271314

This comment has been minimized.

Copy link
Contributor Author

@guest271314 guest271314 commented Sep 13, 2019

The .webm video in the test was created at Firefox

$ mkvmerge -J 400x300-red-resize-300x150-green.webm
{
  "attachments": [],
  "chapters": [],
  "container": {
    "properties": {
      "container_type": 17,
      "is_providing_timestamps": true,
      "muxing_application": "QTmuxingAppLibWebM-0.0.1",
      "writing_application": "QTwritingAppLibWebM-0.0.1"
    },
    "recognized": true,
    "supported": true,
    "type": "Matroska"
  },
  "errors": [],
  "file_name": "400x300-red-resize-300x150-green.webm",
  "global_tags": [],
  "identification_format_version": 12,
  "track_tags": [],
  "tracks": [
    {
      "codec": "VP8",
      "id": 0,
      "properties": {
        "codec_id": "V_VP8",
        "codec_name": "VP8",
        "codec_private_length": 0,
        "default_track": true,
        "display_dimensions": "400x300",
        "display_unit": 0,
        "enabled_track": true,
        "forced_track": false,
        "language": "eng",
        "minimum_timestamp": 0,
        "number": 1,
        "pixel_dimensions": "400x300",
        "uid": 403731017
      },
      "type": "video"
    }
  ],
  "warnings": []
}

Have you tried running the test with the WebM file generated by Chromium implementation of MediaRecorder included at #17821 (comment)?

@foolip

This comment has been minimized.

Copy link
Contributor

@foolip foolip commented Sep 13, 2019

I'll have to double-check, I think I used the file in #17821 (comment)

@guest271314

This comment has been minimized.

Copy link
Contributor Author

@guest271314 guest271314 commented Sep 13, 2019

The file was generated by Firefox https://bugzilla.mozilla.org/show_bug.cgi?id=1177767, see

      "muxing_application": "QTmuxingAppLibWebM-0.0.1",
      "writing_application": "QTwritingAppLibWebM-0.0.1"

For a comprehensive test would include the file produced by Chrome as well.

@guest271314

This comment has been minimized.

Copy link
Contributor Author

@guest271314 guest271314 commented Sep 21, 2019

@foolip

Alright, I finished #19030, but it seems to work in Chrome, so not covering what you'd hoped for I think.

This demonstrates the issue at Chromium https://plnkr.co/edit/n1rGNe?p=info.

@guest271314

This comment has been minimized.

Copy link
Contributor Author

@guest271314 guest271314 commented Sep 24, 2019

@guest271314

This comment has been minimized.

Copy link
Contributor Author

@guest271314 guest271314 commented Sep 25, 2019

@foolip

It's the resize event in https://html.spec.whatwg.org/multipage/media.html that's relevant here.

From your reading of HTML Standard does the specification

Whenever the intrinsic width or intrinsic height of the video changes (including, for example, because the selected video track was changed), if the element's readyState attribute is not HAVE_NOTHING, the user agent must queue a task to fire an event named resize at the media element.

mandate that resize event must be fired when the intrinsic width or intrinsic height - in this case the fact that the developer encodes variable resolution frames directly affecting intrinsic width and intrinsic height?

Or, does the HTML specification (WHATWG or W3C) not mandate that the resize event MUST be dispatched when the underlying encoded frame resolution (intrinsic width and intrinsic height) changes, for example, from 400x300 to 300x150?

@foolip

This comment has been minimized.

Copy link
Contributor

@foolip foolip commented Sep 25, 2019

The resize event should be fired when the intrinsic size changes, but it comes down to the definition of intrinsic width and intrinsic height, which is ultimately a question for the container and video format. I don't know much about this, but there are probably video codecs that internally downscale to improve quality/bit, where a correct player should upscale the video frames. This is somewhat similar to the video having a different aspect ratio than the raw video frames.

So, what should happen with the video streams you've produced depends on other specs that HTML, but my best guess is that resizing and firing a resize event is the right thing.

@guest271314

This comment has been minimized.

Copy link
Contributor Author

@guest271314 guest271314 commented Sep 25, 2019

@foolip The reason asked the question are the contents of this comment https://bugs.chromium.org/p/chromium/issues/detail?id=992235#c29

The change means that it is not possible to display variable resolution video at Chromium using the Blob that Chromium produced at MediaRecorder

Variable resolution videos can still be played, but the video element will not dynamically resize.

The HTML element does display variable resolution when the video (VP8) is recorded at Mozilla Firefox or Nightly, or an encoder other than Chromium https://plnkr.co/edit/8J61Rw?p=preview.

This is working as expected. The video plays through ffmpeg unless you're part of the 1% libvpx experiment. It probably also plays through ffmpeg in firefox. If we launch the libvpx experiment it will cease to do resolution changes in Chrome.

and specifically 3)

It is reasonable to expect that variable resolution input frames to MediaStreamTrack and recorded with MediaRecorder should display the underlying encoded frames at HTML element.

I agree with the high level statement, but I the details of paint a different picture.

  1. Given the crashes are CHECKs, its unclear that MediaRecorder really supports this use case (I ack that it works for some codecs)
  2. Even when an implicit resolution change file could be produced by MediaRecorder, we will still play it back, we just may not resize the element for the later resolutions.
  3. No spec mandates that we change the video size for implicit resolution changes, and we are already mostly not doing it for other codecs.

which implies that HTML does not consider or currently unambiguously define "intrinsic width" and "intrinsic height" as encompassing variable resolution frames encoded in the container which the developer is reasonably being expected to be decoded and displayed at HTML <video> element, else the end-user would not have encoded the variable resolution frames. From perspective here intentionally encoded variable resolution frames MUST be included within the defintion of "intrinsic width" and "intrinsic height" - there is notthing else an "intrinsic width" could refer to relevant to updating videoWidth and firing resize event. If the terms means something else which excludes the end-user intentionally encoding variable resolution frames to the container that needs to be clarified at HTML specification.

The containers Matroska and WebM and the codecs VP8 and VP9 do support encoding variable resolution frames. There is no dispute of facts there. The issue is the arbitrary decision by implementer of HTML <video> to prevent decoding of variable resolution frames and the display those frames at <video> element - perhaps for the purpose of targeting certain devices.

The question is simple: where the specification currently states

the user agent must queue a task to fire an event named resize at the media element.

do the terms "intrinsic width" and "intrinsic height" in the HTML specification apply to the underlying variable resolution encoded video frames?

foolip added a commit that referenced this issue Nov 8, 2019
Fixes #17821.

New media files are from that issue.
foolip added a commit that referenced this issue Nov 14, 2019
Fixes #17821.

New media files are from @guest271314:
#19030 (comment)
foolip added a commit that referenced this issue Nov 15, 2019
Fixes #17821.

New media files are from @guest271314:
#19030 (comment)
#19030 (comment)
foolip added a commit that referenced this issue Nov 15, 2019
Fixes #17821.

New media files are from @guest271314:
#19030 (comment)
#19030 (comment)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.