Notes taken by Harald Alvestrand.
Introduction (5 minutes)
Controlling captured surfaces (25 minutes)
Extensions to CapturedMouseEvents (20 minutes)
Update on AV1 Screen Coding and Content Hints (15 minutes)
Annotations (25 minutes)
Element Capture (25 minutes)
Administrative matters (5 minutes)
- Local:
- Nicola Tommasi - Google Chrome
- Theodore Polsaukas-Warren - Chrome
- Harald Alvestrand - Google WebRTC
- Elad Alon - Google
- Youenn Fablet - Apple
- Eric Carlson - Apple
- Jean - Chrome media
- Henrik Boström - Googe WebRTC
- Andreas Bovens - Whereby
- Simon Hangel - ChromeOS
- Erik Språng - Google WebRTC
- Dominique Hazael-Massieux - W3C
- Camille Lamy - Google Chrome
- Theodore Olsauskas-Warren - Google Chrome
- Remote:
- Philipp Hancke - Microsoft
- Tove Peterson - Google
- Laurent Denoue - Google
- Mark Foltz - Chrome
- Bernard Aboba - Microsoft
- Christoph Gutlandin
Going over the agenda.
No recording.
Elad presents the 3 options (only the 3rd is going to be discussed today)
Option #1: Limited user control of captured surface (expose control widget to user)
Option #2: Unlimited user control of captured surface (“video portal”, redirect user input to capturee). High complexity and contentiousness.
Option #3: Limited app control of captured surface. This option iwll be presented in detail.
Scroll & Zoom is possible.
App can generate input - which means that we can delegate to remote.
Youenn: Is this a benefit or a concern?
Elad: Both.
Youenn: The prompt “you can scroll” is easier to understand than “let the app scroll”. Safer?
Camille: Not sure we can craft a permission message that makes the user understand the risk. “A random person on a VC can scroll my GDoc - is that understandable?”
Theo: Reminds us of element capture - gives away control from the user to the application.
Elad: Pprobably most of the information is already visible... but there’s a permission prompt. Need to also allow quick revocation.
Henrik: Would a heuristic like “center” work 99% of the time for the “scroll the right part” problem?
Elad: This has been considered. It won't work often enough, and we expect users to be confused and frustrated. The widget is also a poor user experience when it does work.
Youenn: Captioning is a case where user wants to put something on top of the capture from an iframe.
Elad: An inferior experience
Youenn: I was mentioning some other benefits <of approach 1>
Henrik: reminds me of the control-button that allows you to scroll the map in GoogleMaps.
Elad: A specific modifier key? Maybe.
Youenn thinks that option 2 (lots of restrictions, but unlimited control) sounds like a good idea.
Elad: One approach allows delegating limited control to a remote user, and the other approach unlimited control to the local user. Both have their uses. Makes sense to eventually build both.
Elad presents API for option 3.
Harald: Is the app (not user) ability to control important or not?
Elad: Important - allows delegation to remote users.
Andreas: Would expect scroll & click (zoom seems to be an optional extra)
Elad: Think that the video portal is the only one that would make click not-risky.
Youenn: who needs zoom?
Elad: Some web developers are adamant that they need it.
Mark Foltz: Can you add an explainer?
Elad: https://github.com/screen-share/captured-surface-control (explainer already, spec tbd)
Elad presents.
Tracking the cursor with OCR is possible but expensive. New API would monitor mouse and send events. “Most people we spoke to thought it was not too dangerous”. Thus mouse events (previously adopted).
Extension is to expose clicks. (Not produce them, of course.) Not observable otherwise - are there concerns with it?
Laurent presents use cases and “we don’t see a big concern” - fallback: can we permit via dialogue?
Elad: Prompting and checkboxes challenges discoverability. Would like to allow without one.
Henrik: What’s the benefit of the app solving the coordinate transform vs the UA solving it?
Laurent: Would have to add media processing (expensive). And the one who wants the effect may not be the presenter.
Elad elaborates: The viewer w/the effects may be remote (in space or in time) - different users can need different things.
Harald: In the case of multiple remote viewers with different needs, transform cannot be done locally - sender doesn’t know what transform is being made.
Youenn: What is being exposed?
Elad: All clicks and scrolls.
Randell: I think this sounds like a reasonable proposal. Will have to consult Mozilla folks for a formal position, of course.
Elad: A specific shape is proposed, but left out of the slides for now. event.button/buttons follows MouseEvent.
Youenn: I don’t have a webkit position - would like to start small. Wonder about the synchronization.
Elad: We’re exposing a timestamp of some kind. Also, not capturing the cursor is an opportunity for frame decimation - saving power.
Bernard presents what Chromium currently does with VideoCodecMode::kScreensharing.
Content-hint doesn’t go into it.
Hancke: WebCodecs exposes a new control for “force text tools” - sees improvements. I2S soon. Artifact reduction, 10% BW reduction. BUT: Sharing a video makes significantly larger frames. So don’t turn on by default.
Q: Do you see any change in power? A: Not looked at yet.
SuggestedContentHint control was preexisting.
Elad: Youenn had previously suggested that the UA should be able to tell - any data?
Hancke: Would prefer an explicit hint. Bernard: ML approaches tried - very expensive in CPU/thermals.
Youenn: Thinks originating website could set a hint?
Henrik: Track effects will strip out necessary information - explicit signals preferred.
Elad/Youenn: Segues into content-hint structure (object, user-defined JSON, known attributes required for platform action). Thread can be revived - on mediacapture-handle
Mark: WRT detection - Chrome’s compositor has some info that could be a useful signal (animation percentage).
Skipped due to absence of Zoom presenters; miscommunication around which of today's two meetings featured their presentation.
Elad introduces. Differences from Region Capture have been emphasized - CropTo(CropTarget) vs RestrictTo(RestrictionTarget).
Youenn: Like the two different types.
Elad: cropTo() / restrictTo() interaction - throw on interaction?
Youenn: Is there an use case for combining?
Elad: Throw now seems like the conservative choice.
Jordan Bayles: What is eligible for restriction? A number of rules.
what is the background color for semitransparent objects?
Elad thinks the apps are cooperative, so targets will be written to be capturable.
Jordan explains issue with bounding box - some stuff overflows it.
Elad asks if we can bound to something other than the bounding box.
Youenn: need to consult with the renderer guys to speak about this.
Mark: Some of the issues are not practical issues - web authors will restrict themselves - but these are potential interop issues, and should be treated as such.
Mark explains the issue with detecting capturability - the value might vary.
Harald: Why wouldn’t you just mint a target when you want to test capturability?
Elad: Don’t think we need isCapturable for an MVP.
Mark: Rejection might be a way to solve the problem, might enforce certain design patterns (make capturable first, mint target afterwards). Not a big deal either way.
Elad: Declare it followup work?
Mark: Waiting for developer feedback seems OK to me.
Elad: Forcing capturability (by CSS?) might be an easier sell to developers.
Youenn: Not necessary for MVP. Privacy issue - fingerprinting - with detecting user CSS?
Elad: Making detection only available on the captured track would mitigate fingerprinting.
Thank you all for joining!
Further discussion goes to github repos - or the mailing list.