Eye gaze correction #56

eehakkin · 2022-01-28T22:26:48Z

Eye gaze correction corrects the eye gaze then a user is looking at the display and not the camera while the camera is far from the point the user is looking at. This problem is widespread especially with large displays. For an example and background, see #46.

Please see #49 (comment) why a constrainable property is most suitable for the purpose.

Preview | Diff

eehakkin · 2023-02-27T13:51:22Z

@jan-ivar, @aboba: Is it OK to merge this PR?

eehakkin · 2023-03-10T11:12:51Z

@alvestrand: Could you merge this PR?

youennf

LGTM, we should unify our usage of User Agent, user agent and UA in this document

index.html

jan-ivar

Eye gaze correction is still a rather new and novel feature, which makes me wonder where the web platform should draw the line. Mozilla is still clarifying its position on this API in mozilla/standards-positions#706 so I'd like another week to run this past some of my colleagues, if that's ok.

index.html

jan-ivar · 2023-03-23T20:38:30Z

From #46 (comment):

Okay, if your team has conducted user studies with evidence of people feeling "something wrong", we can put this on the back burner and re-visit later when users are more comfortable with this. I thought that if Microsoft and Apple had launched this feature on their native platform, there must be a user need. Do let us know if subsequent user studies suggest the perception of "uncanny valley" among majority of users has changed.

Sorry if I missed the answer to this in a more recent presentation, but has information surfaced that "users are more comfortable with this" now than in 2021?

Also, could you clarify why an application would need to control this feature? For blur, I think a valid argument was made that it was to avoid double-blurring, but that doesn't seem to apply here as much.

Isn't this something users can configure in their user agent or OS today and leave it on? Same question for face framing.

youennf · 2023-03-23T21:09:36Z

I think a valid argument was made that it was to avoid double-blurring

If face framing is on in the OS, the web page UI might not want to show this option (or might want to show it but unmodifiable).

index.html

jan-ivar · 2023-03-30T14:33:15Z

index.html

+const videoCapabilities = videoTrack.getCapabilities();
+if ((videoCapabilities.eyeGazeCorrection || []).includes(true)) {
+  await videoTrack.applyConstraints({eyeGazeCorrection: {exact: true}});
+} else {
+  // Eye gaze correction is not supported by the platform or by the camera.
+  // Consider falling back to some other method.
+}
+


Suggested change

const videoCapabilities = videoTrack.getCapabilities();

if ((videoCapabilities.eyeGazeCorrection || []).includes(true)) {

await videoTrack.applyConstraints({eyeGazeCorrection: {exact: true}});

} else {

// Eye gaze correction is not supported by the platform or by the camera.

// Consider falling back to some other method.

}

const capabilities = videoTrack.getCapabilities();

if (("eyeGazeCorrection" in capabilities) {

await videoTrack.applyConstraints({eyeGazeCorrection: true});

} else {

// Eye gaze correction is not supported by the platform or by the camera.

// Consider falling back to some other method.

}

That's not the same thing.

if ((videoCapabilities.eyeGazeCorrection || []).includes(true)) tests if eye gaze correction can be enabled and the else branch can then fall back to some other method.

if ("eyeGazeCorrection" in capabilities) tests only if the UA and the track know about eye gaze correction but the capability can be either [false], [false, true] or [true]. The await videoTrack.applyConstraints({eyeGazeCorrection: true}); will of course always succeed regardless the capability of the lack of it because only optional constraints are used. But now the else branch is reached only if there are no eye gaze correction capabilities but not if the eye gaze correction capability is [false] which contradicts the comment in and the intent of that else branch.

youennf · 2023-03-30T14:33:17Z

We should fix the test example and probably file an issue about quality of implementation.

dontcallmedom-bot · 2023-05-03T15:15:37Z

This issue was mentioned in WEBRTCWG-2023-02-21 (Page 46)

jan-ivar · 2023-09-07T14:27:18Z

@eehakkin is there still interest in this one? After reviewing the editors feel this PR should have a note stating the feature lacks consensus if it is to merge ahead of a CfC.

riju · 2023-09-14T15:31:32Z

@eehakkin is there still interest in this one? After reviewing the editors feel this PR should have a note stating the feature lacks consensus if it is to merge ahead of a CfC.

Yes. We are still interested to continue with this feature.

I have put up an Explainer also. There are 2 modes of Eye Gaze Correction on Windows, so it also an option to have a enum as shown in the Explainer.
If there is a consensus among chairs/editors on having a simple ON/OFF we can start with that also, as in this PR.

Happy to discuss here at TPAC how to move forward on this.

youennf · 2023-09-21T15:01:15Z

Editor's meeting: @riju could you rebase it and we will plan to merge it next week?

eehakkin · 2023-10-10T22:39:56Z

Editor's meeting: @riju could you rebase it and we will plan to merge it next week?

I rebased it.

riju · 2023-10-30T15:28:19Z

Gentle ping @jan-ivar @alvestrand

aboba · 2023-11-02T16:02:26Z

Should we start a Call for Consensus?

alvestrand · 2023-11-02T16:05:46Z

Yes.

…

On Thu, Nov 2, 2023 at 5:02 PM Bernard Aboba ***@***.***> wrote: Should we start a Call for Consensus? — Reply to this email directly, view it on GitHub <#56 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADVM7MZGCVAYC5BHW5R3YLYCO7Z3AVCNFSM5NB45VT2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCNZZGEYDEMJTHEYA> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>

eehakkin mentioned this pull request Jan 28, 2022

Eye contact correction #52

Closed

This was referenced Feb 5, 2022

Face detection, background blur and eye gaze correction example #57

Closed

Add face detection constraints and VideoFrame attributes #48

Closed

jan-ivar marked this pull request as draft October 13, 2022 14:56

aboba mentioned this pull request Oct 13, 2022

Extensibility: VideoFrame and encoded chunk metadata w3c/media-pipeline-arch#5

Open

eehakkin force-pushed the feature/eye-gaze-correction branch from 58442d8 to a87b208 Compare November 29, 2022 15:38

eehakkin marked this pull request as ready for review February 24, 2023 08:31

eehakkin requested a review from alvestrand March 14, 2023 11:11

youennf approved these changes Mar 17, 2023

View reviewed changes

index.html Outdated Show resolved Hide resolved

index.html Outdated Show resolved Hide resolved

index.html Outdated Show resolved Hide resolved

eehakkin force-pushed the feature/eye-gaze-correction branch from 6c43f7a to 6e12e1e Compare March 17, 2023 12:41

jan-ivar reviewed Mar 23, 2023

View reviewed changes

index.html Outdated Show resolved Hide resolved

eehakkin force-pushed the feature/eye-gaze-correction branch from b439256 to bb24084 Compare March 29, 2023 07:57

henbos reviewed Mar 30, 2023

View reviewed changes

index.html Outdated Show resolved Hide resolved

youennf added the Editors can integrate label Mar 30, 2023

jan-ivar reviewed Mar 30, 2023

View reviewed changes

jan-ivar removed the Editors can integrate label Sep 7, 2023

aboba added the TPAC 2023 label Sep 7, 2023

Add support for eye gaze correction

14b2e44

eehakkin force-pushed the feature/eye-gaze-correction branch from 78f35ac to 14b2e44 Compare October 10, 2023 22:38

henbos approved these changes Nov 2, 2023

View reviewed changes

alvestrand merged commit 429fc20 into w3c:main Nov 2, 2023
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Eye gaze correction #56

Eye gaze correction #56

eehakkin commented Jan 28, 2022 •

edited by pr-preview bot

Loading

eehakkin commented Feb 27, 2023

eehakkin commented Mar 10, 2023

youennf left a comment

jan-ivar left a comment

jan-ivar commented Mar 23, 2023

youennf commented Mar 23, 2023

jan-ivar Mar 30, 2023

eehakkin Mar 31, 2023 •

edited

Loading

youennf commented Mar 30, 2023

dontcallmedom-bot commented May 3, 2023

jan-ivar commented Sep 7, 2023

riju commented Sep 14, 2023 •

edited

Loading

youennf commented Sep 21, 2023

eehakkin commented Oct 10, 2023

riju commented Oct 30, 2023

aboba commented Nov 2, 2023

alvestrand commented Nov 2, 2023 via email

Eye gaze correction #56

Eye gaze correction #56

Conversation

eehakkin commented Jan 28, 2022 • edited by pr-preview bot Loading

eehakkin commented Feb 27, 2023

eehakkin commented Mar 10, 2023

youennf left a comment

Choose a reason for hiding this comment

jan-ivar left a comment

Choose a reason for hiding this comment

jan-ivar commented Mar 23, 2023

youennf commented Mar 23, 2023

jan-ivar Mar 30, 2023

Choose a reason for hiding this comment

eehakkin Mar 31, 2023 • edited Loading

Choose a reason for hiding this comment

youennf commented Mar 30, 2023

dontcallmedom-bot commented May 3, 2023

jan-ivar commented Sep 7, 2023

riju commented Sep 14, 2023 • edited Loading

youennf commented Sep 21, 2023

eehakkin commented Oct 10, 2023

riju commented Oct 30, 2023

aboba commented Nov 2, 2023

alvestrand commented Nov 2, 2023 via email

eehakkin commented Jan 28, 2022 •

edited by pr-preview bot

Loading

eehakkin Mar 31, 2023 •

edited

Loading

riju commented Sep 14, 2023 •

edited

Loading