Minutes 2022 04 27

GPU Web 2022-04-27

Chair: Corentin

Scribe: Ken

Location: Google Meet

Tentative agenda

• Administrivia
• CTS Update
• “Tacit Resolution” Queue
• All the canvas issues!
  • createRenderPipeline with preferred format is dependent on canvas #2747
    • proposal: Move preferred format queries to the adapter #2771
  • The preferred format is not always storage-capable #2748
  • rgba8unorm canvas textures and IOSurface #2535
  • WebGPURenderingContext should support preserveDrawingBuffer: true #2743
    • related: Does compositingAlphaMode affect using the canvas as an image source? #1847 (comment)
  • Handling out-of-memory when resizing canvas backing store #2520
• (stretch) Internationalization Metadata for Human-readable Messages #2780
• Agenda for next meeting

Attendance

• Apple
  • Myles C. Maxfield
• Google
  • Austin Eng
  • Brandon Jones
  • Corentin Wallez
  • Gregg Tavares
  • Kai Ninomiya
  • Ken Russell
  • Loko Kung
• Mozilla
  • Jim Blandy
  • Kelsey Gilbert
• Mehmet Oguz Derin

Administrivia

• CW: TAG review open for a year. Sent a semi-strongly worded comment that we can’t wait forever. Heard out-of-band that there IS a review, they just haven’t managed to edit/send it yet.
• CW: “Polish post-v1” milestone. How to figure out what to put in it? Should we propose things to the milestone or have a special section in the meeting for it?
  • KG: Think we can do it like we do with post-v1 issues in this meeting. So someone proposes postponing the milestone and we chat about it.
• Viewport rect must lie entirely within the current attachment size? #373
  • Want to move this to post-V1-polish
  • Silent agreement to do so.
• KN: in the past, have put something in tacit resolution queue, and said that that resolution belongs to another milestone
• KR/KG: WebGL+WebGPU Meetup yesterday - 50+ attendees
  • Lot of excitement from the audience
  • Question about Unity and availability of WebGPU backend
  • Folks want to evaluate NeRFs in compute shaders
  • Would anyone like to present work at a future one?
  • Feel free to reach out to kbr@ or events@khronos.org to sign up

CTS Update

• AE: looking at changing infra for reporting results
  • Many sub-cases -> many back-and-forths
  • Trying to pipeline it
  • Some platforms, looks like it improves things a lot. Some, not as much.
  • Still TBD.
• CW: lot of good CTS contribs from Intel this week.
• CW: CTS keeps finding bugs. Found bug in Metal, Intel HW (feedback report filed)
• CW: WebGPU CTS will be a great GPU API stress test suite. Found many issues in Vulkan as well - all the APIs, really.

“Tacit Resolution” Queue

• https://github.com/gpuweb/gpuweb/pull/2784 from 2 weeks ago
  • Clarify type and use case for object labels
  • BJ: Broken off from convo at prev meeting. Resolving intl'ization.
  • BJ: getting rid of (USVString or undefined) union. Give notes why using USVString not DOMString. Notes clarifying that labels are provided by developer.
  • KN: fundamental thing here - changing from objects may or may not have labels, to they always have labels, defaulting to the empty string.
• Change ArrayBuffer arguments to ArrayBuffer + srcOffset · Issue #244 · gpuweb/gpuweb · GitHub
  • Some discussion - not very webby. Bunch of discussion
  • Tacit resolution - keep what we have, keep the offsets. Avoids creating views all the time.
  • MM: did we ever respond to Dominic's question? He asked for numbers.
  • CW: Jukka from Unity provided numbers. When they did the optimization can be in the double digit percentages. Dominic was saying it was different - Kai's version of the polyfill shows you always need to create garbage. A 15% CPU perf improvement for this tiny thing is very much worth it.
  • MM: not going to push back.
  • CW: tacit resolution is to just close.
• https://github.com/gpuweb/gpuweb/pull/2796
  • CW: VideoFrames the browsers use can be cropped. H.264 encodes 16x16 blocks for example, but can crop borders.
  • Sampling an ExternalTexture will always use clamped addressing mode. Clamp args before doing sample operation.
  • MM: doesn't sample operation correspond to a lot of code already? Why can't we implement the other addressing mode?
  • CW: texture sample level on external texture takes a sampler. Because addressing mode is forced, sampler's only used for filtering - linear or not. You saying - can't we pass sampling parameters when creating ExternalTexture? That's fair, we could do that.
  • MM: we already are required to know which samplers are used by which textures. At compile time, we'd know whether to pass this extra data. I don't quite see the need for this.
  • BJ: do we know the samplers at compile time? Don't think so.
  • MM: we know which samplers are used with this texture.
  • CW: we don't know the content of sampler at compile time, only draw time. Would have to inject data from the texture into a uniform buffer.
  • MM: yup. Telling Metal, please attach this sampler to this render pass - would have to check to see, is this sampler used with an external texture? if so, send additional info. Is there a way to inspect sampler in shader?
  • KN: we'd need sampler, and that extra data. No, no way to inspect sampler in a shader.
  • CW: in general - think this removes this discussion from tacit res queue. If we wanted to do that- would rather not inspect shader/pipeline at runtime. Instead, would be easier to pass the sampler as a creation parameter of the external texture.
  • BJ: think it'll be easier to loosen this over time if we start with the restriction: samplers can only be clamped. Then later, have a feature loosening the restriction. Already have a lot of restrictions on use of ExternalTextures, don't think people will balk at one more.
  • BJ: concretely: when you create the BindGroup.
  • CW: I have an AI to explore the various options, can discuss it again.
  • KN: combined texture/sampler object?
  • MM: if just for ExternalTexture, seems reasonable.
  • KN: yes, only for those textures.
  • CW: I'll take an AI to assemble the options.

All the canvas issues!

createRenderPipeline with preferred format is dependent on canvas #2747

• proposal: Move preferred format queries to the adapter #2771
• BJ: seems safe assumption - for any system, we're probably not going to have different preferredFormats per adapter, nor per canvas.
• BJ: desire: don't scare developers into thinking they need to compile different versions of their pipelines for every canvas. Nor every adapter.
• BJ: can we hoist getPreferredFormat to higher level? Adapter, or navigator.gpu?
• BJ: ideally would help put developers at ease - can look at this once, not reconsider every time creating new context.
• KR: concern over CanvasColorSpaceProposal, HDR canvases.
• BJ: can't silently switch between HDR/LDR formats. The browser won't do this. Seems plausible - when you want to do HDR rendering, may still be preferred format - but you'll have to opt into it. Right now it doesn't take this into account. Method, instead of attribute, to extend this in the future? Yes, want to report diff formats for diff configurations. Wouldn't switch behind the user's back though.
• KG: for VR HMDs we do have different preferred formats. Concerns there?
• BJ: that's a possibility. Likely you'll have the same format. Standalone system - browser displayed in same space as VR content. Or, have something connected to PC. Doubtful you'll be using significantly separate format for that.
• BJ: when creating rendering surfaces for headset for WebXR - goes through separate interface, not the canvas here. Can have separate preferred format reported there.
• KN: it's one of the reasons we have the word "canvas" here - getPreferredCanvasFormat.
• CW: how do we deal with HDR vs. not? Extra arg added later?
• KN: my logic - app will check for HDR support & doing lots of stuff differently. So, later, getPreferredFormat({hdr: true}) or similar. maybe finer grained formats, RGBA16F vs. RGB10_A2.
• CW: so, expand with dictionary of options in the future.
• RC: slight preference for putting it on navigator, or window. Windows ecosystem - people take laptop, plug in external monitors - have different setup, some HDR, some not. Requires broader changes to web platform - monitor config has changed. Having it on window is more forward-looking.
• KN: have been operating under assumption - RGBA vs BGRA is fixed in the OS. I'm more confident that's true, not having to do with display. It would be complicated because you can move windows between displays. You'd need signal to the app - check for new preferred format. For SDR, convinced this won't happen. HDR, I think it will - needs more checks. Win/Mac/Linux want BGRA, Android wants RGBA. Can only get one of those 2 formats here.
• RC: BGRA vs RGBA I agree. More bitness, how wide it is, is more of the concern. Need to be able to handle this somehow. Bigger than this group. Forward-looking thing, on navigator, would be more forward-looking. People can move between adapters, etc.
• BJ: choosing between window/navigator - want to choose navigator because it's on workers.
• CW: if we put it on navigator we might as well as put it on navigator.gpu so it's namespaced. A point Brandon made. If the web ever grows an API to discuss monitors, they're added/removed - would be easy to say, give me the format for this display.
  • KR: this API is being developed: https://web.dev/multi-screen-window-placement/
• MM: even dragging window from one display to another - canvas doesn't change its bit depth. If we wanted to drag window from one monitor to another & start doing HDR stuff - that's a different thing. Shouldn't tie ourselves to a monitor, but to a canvas. App should recreate canvas if window was dragged.
• KN: I exactly agree with that.
• KG: not so quick to agree it should be on navigator instead of window. For WebRender - deciding bit depth will be a document.window decision, not a navigator decision. Don't want to upgrade all of your background tabs. Not sure, still working on this.
• KN: if that turns out to be true - we just won't extend this, and find a different way to handle HDR. Concerning only SDR - I think navigator's the right place.
• KG: 10-bit SDR is a thing also.
• KN: if we're considering only RGBA8 and BGRA8, which are the common cases. Think it makes sense to put it here. More advanced use cases, can add them somewhere else if needed.
• KN: about HDR - we may not even turn out to have preferred formats for HDR. Has been proposed that HDR's always float16. Lot less tied to display hardware - requires tone mapping. One possible future, only one HDR format anyway.
• KR: from conversations in the ColorWeb CG - likely that RGB10_A2 is likely to be required on TV sets. The hardware's not powerful enough to do a full-screen blit at the end of the pipeline.
• KG: I'm OK, reluctantly, putting this on navigator.
• KN: problem with what we're doing today - there are concrete reasons this is wrong, even if we have multiple HDR formats, because canvases can move around and you can be on a different display than the adapter you're using. (canvas,adapter) pair isn't future-proof.
• KG: sure.
• CW: let's call it reasonable, can live with, consensus, and go forward. Figure out how to add HDR later.

The preferred format is not always storage-capable #2748

• CW: Kai outlined 3 options.
• 0. do nothing. People can use RGBA8unorm, incurs a blit, but they can use storage.
• 1. optional feature flags for BGRA8Unorm storage.
• 2. Add magic formats that look like RGBA8unorm storage, but may be BGRA8unorm under the hood. Not copyable, because impl wouldn't have to paper over difference of byte order.
• CW: my strong suggestion - this is a post-V1 issue.
• BJ: discussing with Kai earlier - magic format sounds mildly interesting. Doesn't solve fundamental problem. If someone goes in, blindly gets preferred format, use it everywhere, try to use it as storage - may work on dev machine, works fine, then don't realize this fails as soon as they get onto a Mac, for example. Magic format, unless forcing people to use it all the time, doesn't solve it for us. If there's the possibility that people can stumble into a failing situation, not helping people.
• MM: preferred formats are useful for displaying the final image. Using intermediary framebuffer - can use whatever you want. Preferred format has no relation to that. Using preferred format for last render pass - you won't be using it for storage anyway. Preferred format not having storage binding seems fine to me.
• CW: a couple people at least asking about this for webgpu - doing compute-shader-driven rendering. Agreed, since doing heavy CS, should do a blit at the end.
• KG: just because they're in CS doesn't mean they can eat the bandwidth. Maybe need low overhead. Even outside CS, using storage texture as output - will work on half of machines. Unfortunate. Hard to get around it. Except, when requesting texture, have to provide usage flags.
• RC: do we have this problem as a general problem with the other formats we support? If so, getting it from preferred format enum, or elsewhere, you should know you can/can not do storage.
• KN: if not opting into add'l caps - you'll get the spec'd behavior. RGBA - storage will work. Put on system that uses BGRA - things will break.
• MM: only way to write correct WebGPU program using preferred format - ask for preferred format, feed it into a switch statement? That's the problem.
• KN: if you want to use storage, yes.
• CW: ties into another common issue, can we have an API for texture caps in WebGPU? One day, will probably have that, and switch statement will disappear from user code. My belief: getPreferredFormat() guaranteed to only return rgba8unorm or bgra8unorm today, so you just have a condition. Likely developing on desktop, bgra8unorm, and you'll test that.
• KG: wouldn't work on mobile.
• KN: it would, because rgba8unorm is strictly more capable than bgra8unorm. Assuming you get the rest of your caps matching correct.
• BJ: always in favor of warning when user does something risky. Heuristic: if config'ing canvas, querying storage, getting preferred format, use preferred format - warn you might be doing dangerous thing. More extreme, not advocating for: getPreferredFormat always returns special enums with "preferred" in them. Equivalent, but if just taking something and shoving it in somewhere else, there's an additional bit. Using preferred format with storage = warn, or maybe even fail. Seems like a lot of hoop-jumping to avoid a minority case.
• MS: seems like we want to be able to query texture caps. We basically had to implement this table in C++. If you can query it, should always query it, will never run into this situation.
• MM: on mac/ios - you can use the preferred texture as storage. reason it's not storage-capable is because of other platforms. Dealing with false problem - there's no OS that can't use this preferred format as storage. Can't we change the API to fix this?
• KN: difficulty - we'd introduce a cap that automatically shows up on a format. BGRA format sometimes would have storage support, sometimes not. Windows, will never have it. OK to add usage capability to this format for this reason. Would otherwise say, have to enable device feature flag to make it work.
• MM: so the idea is you'd specify the feature flag on Mac and get it, and on Windows and not get it?
• KN: Windows prefers BGRA but doesn't support storage.
• MM: I see.
• BJ: beyond preferred storage - people could prefer to use it as their texture format.
• JN: don't see strong reason why Windows would need to prefer BGRA. D3D12 perspective - once in swap chain, should be equivalent. Used to prefer BGRA for GDI interop, but web platforms' not doing GDI interop anyway.
• CW: we tend to promote canvases into overlays - think that's why we use BGRA.
• JN: anyone who exposes color RGB overlays on Windows can do the channel swap dynamically.
• CW: no perf cost?
• JN: AFAIK, but I haven't heard of one and am not display expert.
• KG / KR: vote for #0.
• MM: core of problem - author taking preferred format and creating textures with it - there's nowhere in their program they're typing "bgra". Option 3 - change the enum for preferred format, return a new enum with two values in it? Author has to translate those two enums to pixel formats, they end up spelling the pixel formats themselves.
• BJ: don't know that addresses the issue long-term. Forcing them to do the translate, everyone will have the same utility function. Pass value in/out, get into same situaiton easily. Could have variant - preferred-specific format, but can be passed into places where other formats are accepted. Could warn / disallow that specific format to be used. Simple case, passing it around everywhere - will be more likely to hit barriers we can put in place.
• MS: might cause more issues - create new format like opaque format, always the preferred format, never supports storage.
• CW: nor copies?
• MS: whatever you'd need to do to it.
• MM: similar to option 2.
• CW: can't rely on BGRA8 unorm storage caps. Metal storage tier 1 only supports floats. (?)
• KN: that's write-only cap. Just doesn't have read-write.
• CW: ah, sorry.
• JN: D3D12 is similar, RW storage is harder.
• KR: can we please agree on option 0 for now to move forward? We can resolve this better in the future, and we can warn developers about this.
• CW: can we say that? Or does someone still want to discuss this?
• MM: think if we go with option 0 now, then at least say we'll do option 1 "soon".
• CW: as optional feature, sure.
• CW: ok, let's do this, and post-V1-polish BGRA8unorm storage feature. We could also say that if preferred format is BGRA8unorm, option 1 will be supported.
• MM: that definition of "soon" should be really soon.
• KN: I"m OK saying we can do option 1. Option 2's tempting, but too restrictive - think it won't work well.
• CW: option 1's just one optional feature. Guarantee that the extension's available on BGRA8 systems. Then close this, have resolution.

rgba8unorm canvas textures and IOSurface #2535

WebGPURenderingContext should support preserveDrawingBuffer: true #2743

• related: Does compositingAlphaMode affect using the canvas as an image source? #1847 (comment)

Handling out-of-memory when resizing canvas backing store #2520

(stretch) Internationalization Metadata for Human-readable Messages #2780

Agenda for next meeting

• APAC-timed meeting
• More canvas stuff.
• KG: next week not APAC for WGSL. Think Shanghai's OOO. Back on May 5, May 4 our time? Not sure.
• CW: I'll ask. Assume APAC timed, but if not, will email.