Effective MLComputePolicy exposure

[anssiko]

Opening this fresh issue per:

RESOLUTION: Close issue #836 and open a new issue for Effective MLComputePolicy exposure discussion.

Effective MLComputePolicy is the policy actually used by the implementation. To that end, this effective policy can be exposed only after compilation, likely on MLGraph.

Based on today's discussion, the group believes the policy is a better abstraction than devices, thus we resolved that this to-be-defined policy-centric mechanism will supersede the earlier device-centric graph.devices proposal that was discussed in issue #836 and PR #854 that was closed.

The group's next step is to make sure the effective policy is exposed in a way that allows developers to understand it.

@handellm owns the use case and a product that would benefit from this feature, and can enlighten us on the preferred design.

For your reference, here's the latest MLComputePolicy that's passed in at context creation time (from #923):

enum MLComputePolicy {
  "default",
  "high-performance",
  "low-power",
  "fallback"
};

[handellm]

Thanks for filing the issue, @anssiko!

Here are the use-cases that I think effective compute policy would help solve:

1. Real-Time Media Constraints Audio models need to be executed with low and reasonably predictable latency and can typically not run well on GPUs and NPUs. Video models have unacceptable power or latency characteristics when executed on CPU. A reported mismatching effective compute policy enables an execution router to consider alternatives before ruining user experience by simply trying.

2. Dynamic execution routing The system must account for real-time device conditions when considering how to run a new workload (locally or in the cloud). If effective compute policies indicate potential hardware contention, the router can rearrange workloads to avoid degrading the user experience.

3. QoS monitoring There could be usage of ops and data types in a model that force unintended effective quality of service causing poor user experience. Trying to figure out all incompatibilities at development time is impossible due to the cartesian zoo of implementations (hardware, drivers, browsers, OSes). The effective compute policy enables aggregation of outcomes.

The concrete proposal (which is similar to already discussed alternatives) is to expose a set of MLComputePolicies that match the effective compiled graph's compute characteristics. A set of policies seems to better express the situation for the implementation than a single policy, for example to be able to express partial fallback.

interface MLGraph {

  // Set-like
  readonly attribute FrozenArray<MLComputePolicy> effectiveComputePolicies;

}

Interpretations

• ["high-performance", "fallback"] could indicate partial hardware acceleration
• ["low-power", "high-performance"] could indicate inference is using a device which is both high performance and low power
• ["fallback"] there won't be hardware acceleration when dispatching this onw

There are also non-sensical combinations, not sure how to deal with them.

Examples of use:

Use case	Requested policy	Criteria for using WebNN
Audio denoising	"fallback"	"fallback" in effectiveComputePolicies AND effectiveComputePolicies does not contain "low-power" or "high-performance"
Video processing + dynamic routing example	"low-power"	"fallback" not in effectiveComputePolicies AND (no significant NPU workload OR "low-power" not in effectiveComputePolicies) AND (no significant GPU workload OR "high-performance" not in effectiveComputePolicies)

Wdyt?

[zolkis]

This proposal works well with the current opaque design for MLGraph. Needs to be checked against others' use cases, though.

There are also non-sensical combinations, not sure how to deal with them.

Implementations will own the sense of them, in this proposal.

I assume the "Criteria for using WebNN" are checked inside the application. Would there be something WebNN could do to make these checks simpler?

[handellm]

Would there be something WebNN could do to make these checks simpler?

The end result of our use cases will result in us either using or not using the compiled graph, so an alternative solution could be to pass some kind of "acceptable" representation when compiling the graph and fail it if it doesn't go the right way. That could impact the rejection time positively.

[anssiko]

RESOLUTION: Review Dynamic AI Offload Protocol in context of the effective MLComputePolicy proposal.

@jonathanding, the WebML WG discussed dynamic execution routing today and wants to invite you to share an update on the Dynamic AI Offloading Protocol (DAOP) incubation to understand synergies.

The group would like to hear your learnings from the DAOP incubation, both web and native side, to understand how to shape the effective MLComputePolicy proposal.

@handellm represents the Google Meet product that would benefit from dynamic execution routing, a use case that is increasingly more important given the global compute shortage.

[handellm]

@mwyrzykowski what do you think of the proposal?

[mwyrzykowski]

Examples of use:

Use case	Requested policy	Criteria for using WebNN
Audio denoising	"fallback"	"fallback" in effectiveComputePolicies AND effectiveComputePolicies does not contain "low-power" or "high-performance"
Video processing + dynamic routing example	"low-power"	"fallback" not in effectiveComputePolicies AND (no significant NPU workload OR "low-power" not in effectiveComputePolicies) AND (no significant GPU workload OR  "high-performance" not in effectiveComputePolicies)

@handellm it seems like sites should not be trying to rely on this

E.g., site requests fallback, but UA implements no such fallback, similar to WebGPU's fallback which is optional https://www.w3.org/TR/webgpu/#fallback-adapter but rather a UA reports high-performance, site chooses not to run the model which would have otherwise ran great.

Similarly, high-performance and low-power are not necessarily mutually exclusive on some hardware.

I will try to make the next meeting the the WG time slot is lately a conflict after day light savings pushed the meeting back an hour.

It would seem some type of high performance timer could achieve the same with far greater reliability than exposing an effective compute policy.

[handellm]

E.g., site requests fallback, but UA implements no such fallback

Why would it not implement fallback? Is there an implementation that doesn't implement it?

but rather a UA reports high-performance, site chooses not to run the model which would have otherwise ran great.
It would seem some type of high performance timer could achieve the same with far greater reliability than exposing an effective compute policy.

"high-performance" would typically not be an acceptable result in the case of Audio models. Although high performance timers are not possible on the web platform due to security, you might still be able to use normal performance.now() for typical Audio dispatch speeds. But regardless - the problem with "trying anyway" is still that on a device with unsuitable implementation the absence of an API will cause bad experience for the user before detected and falling back to working solutions. Why would that be desireable?

[mwyrzykowski]

E.g., site requests fallback, but UA implements no such fallback

Why would it not implement fallback? Is there an implementation that doesn't implement it?

WebGPU has the concept of fallback adapters which are optional. Their target use case is often dealing with non-conformant GPU drivers whose bugs are too challenging or not practical to work around.

There is no requirement to provide such an adapter. A site requests it and the UA is free to ignore it, for instance on GPU hardware with few bugs to work around.

WebGL and WebGPU both have power preference enums, WebGLPowerPreference and powerPreference, again a UA is free to ignore these as they are hints.

"high-performance" would typically not be an acceptable result in the case of Audio models.

Currently the WebNN specification defines high-performance as:

"high-performance"
Prioritizes execution speed over power consumption.

One would think prioritizing execution speed is precisely what is desired for audio processing. A UA reports high-performance and then the site decides not to run the model due to anticipated latency => that does not make sense since the UA never said there was going to be latency.

Although high performance timers are not possible on the web platform due to security, you might still be able to use normal performance.now() for typical Audio dispatch speeds. But regardless - the problem with "trying anyway" is still that on a device with unsuitable implementation the absence of an API will cause bad experience for the user before detected and falling back to working solutions. Why would that be desireable?

It is a reliable method handling system load from other actively running sites / applications / system operations and performance.now() should be sufficient. We could also consider exposing some graph compilation metrics regarding rough latency and runtime estimates to avoid requiring such timings.

I don't think we should build some matrix of hints to latency / runtime / chip-the-model-runs-on expectations, that is not the purpose of hints and would be objectionable.

[fdwr]

One would think prioritizing execution speed is precisely what is desired for audio processing.

Note low-latency differs from throughput - a horse runs faster than a rabbit, but for short distances, a rabbit beats a horse because it takes longer for the horse to gain speed. We saw that multiple times with DirectML (RIP) where customers would complain that their tiny audio models took too long to upload the audio data to the GPU, process it, and then download data back. So we recommended to just keep the audio model on the CPU instead (or these days, perhaps NPU's if UMA and beefier than they used to be). Now if they passed a whole mp3 to the GPU, yes, that would certainly beat the CPU, but audio filters like live noise cancellation during meetings have really tiny time slices and need immediate results to avoid audio lag.

[handellm]

One would think prioritizing execution speed is precisely what is desired for audio processing.

The reality is that on many platforms, hardware execution is associated with queueing delays. For Audio this is a killer reason to avoid it.

This is one example where the UAs view of "it should fit" might not match actual reality for app developers. There are others, like hybrid graphs which are likely to not match realtime performance expectations for video.

When the UA makes the wrong choice, the user has to endure poor performance before the system notices. This is what we want to avoid.

Also interested in perspectives from @RafaelCintron and @huningxin on the matter.

[zolkis]

We again hit a resource optimization problem seen and discussed before, also in this group: "high-performance" is a compound term and depends also on the application (model): whether audio, video, generative, etc. I'd assume that for audio a "low-latency" description would be more generic and more relevant. Or, we could create more hint classes (compute policies) describing the intended inputs and outputs.

These (counter-)examples don't necessarily invalidate the proposed design (that returns an array of compute policy descriptors), but we need to de-confuse the combinations by choosing a higher resolution "sampling" of the use case descriptors, which are as much orthogonal to each other as possible, in order to reduce overlaps. If we introduce more descriptors that are overlapping then we'll face exponentially bigger confusion space when these overlaps happen.

Fortunately we don't need to solve the whole world, then the use cases, but first focus on the use cases.

Still the question remains, @handellm - for your use cases, could you come up with a single descriptor that would quantify your needs, or if it must be a combination, is there an orthogonal set of descriptors that would solve your use case, and could be also used to decompose other use cases?

[handellm]

@zolkis re: low-latency. For the case of realtimeish Audio processing, use of hardware is a no-go so the present "fallback" perfectly serves the purpose. Not saying that there might be other use cases of low-latency or in what sense, just I don't know what those would be. Maybe low-latency can be separately discussed?

It looks like the primary question is whether exposing scheduled compute devices to the app is necessary. So as I mentioned above, since the anticipated end result would be about not using the compiled graph, one alternative proposal could be that the UA fails MLGraphBuilder build if the results turns out to be incompatible with a specified MLContext computePolicy. That pattern kind of aligns with the existing pattern of WebRTC getUserMedia constraints whose promise is rejected if constraints are not met.

Examples needed to solve use cases 1-3 (drafty):

MLComputePolicy	Description
low-power, high-performance	Prefer [low-power, high-performance] device but accept ops falling back
strict-low-power	Fail build if some op is not scheduled to a low-power equivalent device
strict-high-performance	Fail build if some op is not scheduled to a high-performance equivalent device
strict-fallback	Fail build if ops are scheduled on HW
strict-accelerated	Fail build if some op falls back to CPU
strict-accelerated-prefer-low-power	Fail build if some op falls back to CPU, prefer scheduling to low-power devices

Wdyt?