Make requestFrame() observable as a Promise resolving with frame metadata

[guest271314]

Currently requestFrame() is not observable.

At some implementations of captureStream() it might be necessary to execute requestFrame() twice to get the current frame drawn onto the canvas captured in the MediaStreamTrack, see https://bugs.chromium.org/p/chromium/issues/detail?id=1047984.

Given requestFrame() operation necessarily asynchronous make requestFrame() a Promise that resolves with a plain JavaScript object containing metadata about the frame request and image, for example, from HTMLVideoElement.requestVideoFrameCallback() we find this language

Drawing operations (e.g. drawing a video frame to a canvas via drawImage()) made through this API will be synchronized as a best effort with the video playing on screen. Best effort in this case means that, even with a normal work load, a callback can occasionally be fired one v-sync late, relative to when the new video frame was presented. This means that drawing operations might occasionally appear on screen one v-sync after the video frame does. Additionally, if there is a heavy load on the main thread, we might not get a callback for every frame (as measured by a discontinuity in the presentedFrames).

where best effort is a realistic description of drawing frames onto the <canvas> that will make it to the MediaStreamTrack, as there is no guarantee that requestFrame() will be executed in the current "event loop" or "tick" or "v-sync".

2. VideoFrameMetadata

dictionary VideoFrameMetadata {
  required DOMHighResTimeStamp presentationTime;
  required DOMHighResTimeStamp expectedDisplayTime;

  required unsigned long width;
  required unsigned long height;
  required double mediaTime;

  required unsigned long presentedFrames;
  double processingDuration;

  DOMHighResTimeStamp captureTime;
  DOMHighResTimeStamp receiveTime;
  unsigned long rtpTimestamp;
};

and

3. VideoFrameRequestCallback

callback VideoFrameRequestCallback = void(DOMHighResTimeStamp now, VideoFrameMetadata metadata);

so that users may gain some observable metadata about the result of the transactional procedure of drawing an image onto the <canvas> and capturing that image into a MediaStreamTrack.

The resulting change could look something like

await track.requestFrame()
/*
[
  3104.6,
  {
    "expectedDisplayTime": 3089.1200000000003,
    "height": 150,
    "mediaTime": 2.805395,
    "presentationTime": 3089.1200000000003,
    "presentedFrames": 3,
    "width": 300
  }
]
*/

[guest271314]

Related #28

[alvestrand]

I think this passes the limit for "substantive contribution", which we cannot accept from someone who has not agreed to the IPR rules for the WG. Closing.

[guest271314]

@alvestrand Actually did sign up for W3C. The W3C revoked membership for the fraudulent reason that guest271314 is not a "real name" though nowhere in W3C documentation could find a definition of "real name". There are publication by W3C with "real name" that is no different than this users' "real name" web-platform-tests/rfcs#21 (comment). Thus, that rational has no credibility with regard to W3C besides proving they have the capacity to be hypocrites, even when advising them there is no such thing as "real name" per their own lack of a defition and publications.

The "IPR" question is interesting. Had to file an issue just to get W3C to actually state in the repository that "IPR" means intellectual property rights.

When file an issue it is not for some accredidation. It is a concept in an effort to solve a problem. Thus, any Member, or who has their "IPR" badge per W3C can certain take ownership of the issue.

Making requestFrame() observable is in the interests of all parties, not just a party who might happen to have signed some contract with W3C.

In any event, all code that write and issues that raise in W3C repositories are PUBLIC DOMAIN.