[WebNN] Better int64 integration #23831

Honry · 2025-02-27T03:01:22Z

This PR adds some workarounds to enable int64 support for some WebNN backends which don't support int64 data type.

Do not fallback ops that are specifically due to the int64 limitation.
Convert all int64 initializer and input values to int32 and handle potential overflow errors.
Register all int64 model inputs and outputs as int32 ml-tensor.
Handle ONNX ops that need inputs or outputs conversion between int64 and int32. e.g. ArgMax, ArgMin, Cast, etc.
Convert int64 output data back to int32.
Disallow int64 outputs as 'ml-tensor' preferredOutputLocation.

Fixed #21401

Honry · 2025-02-27T03:02:05Z

@fdwr, @guschmue, @egalli, PTAL, thanks!

guschmue · 2025-02-27T16:22:46Z

/azp run ONNX Runtime Web CI Pipeline,Windows GPU CI Pipeline,Linux Android Emulator QNN CI Pipeline

guschmue · 2025-02-27T16:22:54Z

/azp run Linux CPU CI Pipeline,Linux CPU Minimal Build E2E CI Pipeline,Linux GPU CI Pipeline,Linux GPU TensorRT CI Pipeline, Linux OpenVINO CI Pipeline,Linux QNN CI Pipeline,MacOS CI Pipeline,Windows ARM64 QNN CI Pipeline,Windows CPU CI Pipeline

guschmue · 2025-02-27T16:23:01Z

/azp run Windows GPU TensorRT CI Pipeline,onnxruntime-binary-size-checks-ci-pipeline,orttraining-linux-ci-pipeline,orttraining-linux-gpu-ci-pipeline,orttraining-ortmodule-distributed,Windows x64 QNN CI Pipeline,Big Models

azure-pipelines · 2025-02-27T16:23:02Z

Azure Pipelines successfully started running 2 pipeline(s).

guschmue · 2025-02-27T16:23:07Z

/azp run Windows GPU CUDA CI Pipeline,Windows GPU DML CI Pipeline,Windows GPU Doc Gen CI Pipeline, Win_TRT_Minimal_CUDA_Test_CI

azure-pipelines · 2025-02-27T16:23:19Z

Azure Pipelines successfully started running 4 pipeline(s).

azure-pipelines · 2025-02-27T16:23:24Z

Azure Pipelines successfully started running 4 pipeline(s).

azure-pipelines · 2025-02-27T16:23:30Z

Azure Pipelines successfully started running 9 pipeline(s).

fdwr

Thanks Wanming. It's a tough problem to think about, but the change isn't as large as I was expecting.

onnxruntime/core/providers/webnn/builders/model_builder.cc

onnxruntime/core/providers/webnn/builders/model_builder.h

onnxruntime/core/providers/webnn/builders/impl/cast_op_builder.cc

js/web/lib/wasm/jsep/webnn/tensor-manager.ts

onnxruntime/core/providers/webnn/builders/impl/argmax_min_op_builder.cc

fdwr · 2025-03-04T02:58:24Z

js/web/lib/wasm/jsep/webnn/tensor-manager.ts

+  public shouldConvertInt64toInt32: boolean = false;
+  public isInt64ToInt32Converted: boolean = false;


I recall hearing from Austin Sullivan some limitations in CoreML about what data types you could bind to the graph inputs/outputs that differed from what operators could internally use (IIRC, the model input/output types were a subset of the operator types), meaning we might need upcasts/downcasts for other data types too. So would it be cleaner to just store a "dataType" and an "originalDataType"? I think both of these booleans could be trivially derived from that (where dataType == 64 and originalDataType == 32, or vice versa).

Aah, here is @philloooo's comment in webmachinelearning/webnn#675

CoreML only supports int32, int16, uint16, int8, uint8 . And at model boundary, it only supports int32, float32, float16

So if we need to send a uint8 tensor (a common pixel format) to CoreML graph inputs, will we need to upcast to int32? Of course u/int8 inputs don't need to be part of this change, but I want to check whether this approach of a number of booleans will scale cleanly.

(imo, the WebNN model builder would ideally just handle uint8 in the backend rather than each caller needing to repeat such logic which varies across OS's 🙃, or CoreML would just accept uint8 directly, but handling it in ORT is our option for now)

Good point!

We still need an additional boolean flag to indicate whether the int64 data has been converted, specifically for cases where the user invokes the download method in this file. If it has been converted, we should converted it back to int32.

Besides, from the perspective of the getCachedTensor method, introducing a boolean flag (shouldConvertInt64toInt32) would provide better clarity.

My inquiry was whether we even need a bunch of boolean flags (and consider the future if we add int8 -> int32, how many more boolean flags there will be to test and pass around 🙃), if we instead just had a field for the "coerced data type" or "fallback data type" in the tensor wrapper.

Honry

@fdwr, thanks much for your careful review, addressed most comments, PTAL again. ❤

js/web/lib/wasm/jsep/webnn/tensor-manager.ts

onnxruntime/core/providers/webnn/builders/impl/cast_op_builder.cc

onnxruntime/core/providers/webnn/builders/model_builder.h

Honry · 2025-03-04T05:35:53Z

js/web/lib/wasm/jsep/webnn/tensor-manager.ts

+  public shouldConvertInt64toInt32: boolean = false;
+  public isInt64ToInt32Converted: boolean = false;


Good point!

We still need an additional boolean flag to indicate whether the int64 data has been converted, specifically for cases where the user invokes the download method in this file. If it has been converted, we should converted it back to int32.

Besides, from the perspective of the getCachedTensor method, introducing a boolean flag (shouldConvertInt64toInt32) would provide better clarity.

js/web/lib/wasm/jsep/webnn/tensor-manager.ts

onnxruntime/core/providers/webnn/builders/impl/argmax_min_op_builder.cc

onnxruntime/core/providers/webnn/builders/model_builder.cc

fdwr

👀 Would be good for @guschmue or @egalli to look at the JSEP and shared WASM parts. The rest looks alright to me.

js/web/lib/wasm/jsep/webnn/tensor-manager.ts

fdwr · 2025-03-05T00:31:58Z

js/web/lib/wasm/jsep/webnn/tensor-manager.ts

+  public shouldConvertInt64toInt32: boolean = false;
+  public isInt64ToInt32Converted: boolean = false;


My inquiry was whether we even need a bunch of boolean flags (and consider the future if we add int8 -> int32, how many more boolean flags there will be to test and pass around 🙃), if we instead just had a field for the "coerced data type" or "fallback data type" in the tensor wrapper.

fdwr · 2025-03-05T02:42:12Z

/azp run ONNX Runtime Web CI Pipeline,Windows GPU CI Pipeline,Linux Android Emulator QNN CI Pipeline

fdwr · 2025-03-05T02:42:15Z

/azp run Linux CPU CI Pipeline,Linux CPU Minimal Build E2E CI Pipeline,Linux GPU CI Pipeline,Linux GPU TensorRT CI Pipeline,Linux OpenVINO CI Pipeline,Linux QNN CI Pipeline,MacOS CI Pipeline,Windows ARM64 QNN CI Pipeline,Windows CPU CI Pipeline

fdwr · 2025-03-05T02:42:17Z

/azp run Windows GPU CUDA CI Pipeline,Windows GPU DML CI Pipeline,Windows GPU Doc Gen CI Pipeline,Win_TRT_Minimal_CUDA_Test_CI

fdwr · 2025-03-05T02:42:19Z

/azp run Windows GPU TensorRT CI Pipeline,onnxruntime-binary-size-checks-ci-pipeline,orttraining-linux-ci-pipeline,orttraining-linux-gpu-ci-pipeline,orttraining-ortmodule-distributed,Windows x64 QNN CI Pipeline,Big Models

azure-pipelines · 2025-03-05T02:42:26Z

Azure Pipelines successfully started running 2 pipeline(s).

azure-pipelines · 2025-03-05T02:42:32Z

Azure Pipelines successfully started running 4 pipeline(s).

azure-pipelines · 2025-03-05T02:42:37Z

Azure Pipelines successfully started running 4 pipeline(s).

azure-pipelines · 2025-03-05T02:42:48Z

Azure Pipelines successfully started running 9 pipeline(s).

fdwr · 2025-03-06T07:05:59Z

/azp run Big Models (Llama2_7B_ONNX Llama2_7B_ONNX)

azure-pipelines · 2025-03-06T07:06:05Z

No pipelines are associated with this pull request.

fdwr · 2025-03-06T07:08:52Z

The failure for Big Models (Llama2_7B_ONNX Llama2_7B_ONNX) appears to be a device space issue (link):

/opt/rh/gcc-toolset-11/root/usr/bin/ranlib: libprotocd.a: No space left on device

And there are linter issues:

/mnt/vss/_work/1/s/js/web/lib/wasm/jsep/backend-webnn.ts
  291:5   error  Type boolean trivially inferred from a boolean literal, remove type annotation  @typescript-eslint/no-inferrable-types
  376:41  error  ["input"] is better written in dot notation                                     dot-notation
  376:50  error  ["dataTypes"] is better written in dot notation                                 dot-notation

/mnt/vss/_work/1/s/js/web/lib/wasm/jsep/webnn/tensor-manager.ts
   92:3   error  Type boolean trivially inferred from a boolean literal, remove type annotation  @typescript-eslint/no-inferrable-types
   93:3   error  Type boolean trivially inferred from a boolean literal, remove type annotation  @typescript-eslint/no-inferrable-types
  154:25  error  'convertInt32ToInt64' was used before it was defined                            @typescript-eslint/no-use-before-define
  168:11  error  Redundant use of `await` on a return value                                      no-return-await
  169:11  error  Redundant use of `await` on a return value                                      no-return-await
  221:58  error  ["input"] is better written in dot notation                                     dot-notation
  221:67  error  ["dataTypes"] is better written in dot notation                                 dot-notation
  223:7   error  Assignment to function parameter 'dataType'                                     no-param-reassign
  267:9   error  Assignment to function parameter 'data'                                         no-param-reassign
  267:16  error  'convertInt64ToInt32' was used before it was defined                            @typescript-eslint/no-use-before-define
  290:12  error  'convertInt32ToInt64' was used before it was defined                            @typescript-eslint/no-use-before-define
  431:5   error  Type boolean trivially inferred from a boolean literal, remove type annotation  @typescript-eslint/no-inferrable-types
  469:8   error  Use const or class constructors instead of named functions                      prefer-arrow/prefer-arrow-functions
  498:1   error  Use const or class constructors instead of named functions                      prefer-arrow/prefer-arrow-functions

Silly linter, complaining about explicit boolean. :/

This PR adds some workarounds to enable int64 support for some WebNN backends which don't support int64 data type. - Do not fallback ops that are specifically due to the int64 limitation. - Convert all int64 initializer and input values to int32 and handle potential overflow errors. - Register all int64 model intputs and outputs as int32 ml-tensor. - Handle ONNX ops that need intputs or outputs conversion between int64 and int32. e.g. ArgMax, ArgMin, Cast, etc. - Convert int64 output data back to int32. - Disallow int64 outputs as 'ml-tensor' preferredOutputLocation. Fixed microsoft#21401

Honry · 2025-03-07T05:37:54Z

The failure for Big Models (Llama2_7B_ONNX Llama2_7B_ONNX) appears to be a device space issue (link):

/opt/rh/gcc-toolset-11/root/usr/bin/ranlib: libprotocd.a: No space left on device

And there are linter issues:

/mnt/vss/_work/1/s/js/web/lib/wasm/jsep/backend-webnn.ts
  291:5   error  Type boolean trivially inferred from a boolean literal, remove type annotation  @typescript-eslint/no-inferrable-types
  376:41  error  ["input"] is better written in dot notation                                     dot-notation
  376:50  error  ["dataTypes"] is better written in dot notation                                 dot-notation

/mnt/vss/_work/1/s/js/web/lib/wasm/jsep/webnn/tensor-manager.ts
   92:3   error  Type boolean trivially inferred from a boolean literal, remove type annotation  @typescript-eslint/no-inferrable-types
   93:3   error  Type boolean trivially inferred from a boolean literal, remove type annotation  @typescript-eslint/no-inferrable-types
  154:25  error  'convertInt32ToInt64' was used before it was defined                            @typescript-eslint/no-use-before-define
  168:11  error  Redundant use of `await` on a return value                                      no-return-await
  169:11  error  Redundant use of `await` on a return value                                      no-return-await
  221:58  error  ["input"] is better written in dot notation                                     dot-notation
  221:67  error  ["dataTypes"] is better written in dot notation                                 dot-notation
  223:7   error  Assignment to function parameter 'dataType'                                     no-param-reassign
  267:9   error  Assignment to function parameter 'data'                                         no-param-reassign
  267:16  error  'convertInt64ToInt32' was used before it was defined                            @typescript-eslint/no-use-before-define
  290:12  error  'convertInt32ToInt64' was used before it was defined                            @typescript-eslint/no-use-before-define
  431:5   error  Type boolean trivially inferred from a boolean literal, remove type annotation  @typescript-eslint/no-inferrable-types
  469:8   error  Use const or class constructors instead of named functions                      prefer-arrow/prefer-arrow-functions
  498:1   error  Use const or class constructors instead of named functions                      prefer-arrow/prefer-arrow-functions

Thanks @fdwr, lint issues fixed, and I also rebased the code to latest main to solve a conflict.

fdwr

👍

fdwr · 2025-03-07T06:33:06Z

/azp run ONNX Runtime Web CI Pipeline,Windows GPU CI Pipeline,Linux Android Emulator QNN CI Pipeline

fdwr · 2025-03-07T06:33:10Z

/azp run Linux CPU CI Pipeline,Linux CPU Minimal Build E2E CI Pipeline,Linux GPU CI Pipeline,Linux GPU TensorRT CI Pipeline,Linux OpenVINO CI Pipeline,Linux QNN CI Pipeline,MacOS CI Pipeline,Windows ARM64 QNN CI Pipeline,Windows CPU CI Pipeline

fdwr · 2025-03-07T06:33:16Z

/azp run Windows GPU CUDA CI Pipeline,Windows GPU DML CI Pipeline,Windows GPU Doc Gen CI Pipeline,Win_TRT_Minimal_CUDA_Test_CI

fdwr · 2025-03-07T06:33:18Z

/azp run Windows GPU TensorRT CI Pipeline,onnxruntime-binary-size-checks-ci-pipeline,orttraining-linux-ci-pipeline,orttraining-linux-gpu-ci-pipeline,orttraining-ortmodule-distributed,Windows x64 QNN CI Pipeline,Big Models

azure-pipelines · 2025-03-07T06:33:22Z

Azure Pipelines successfully started running 2 pipeline(s).

azure-pipelines · 2025-03-07T06:33:33Z

Azure Pipelines successfully started running 4 pipeline(s).

azure-pipelines · 2025-03-07T06:33:36Z

Azure Pipelines successfully started running 4 pipeline(s).

azure-pipelines · 2025-03-07T06:33:48Z

Azure Pipelines successfully started running 9 pipeline(s).

guschmue added the ep:WebNN label Feb 27, 2025

fdwr reviewed Mar 4, 2025

View reviewed changes

Honry commented Mar 4, 2025

View reviewed changes

Honry force-pushed the int64 branch from 331f768 to d9b9f08 Compare March 4, 2025 12:35

fdwr previously approved these changes Mar 5, 2025

View reviewed changes

Honry added 3 commits March 7, 2025 11:12

Address comments

9372c93

Fix lint errors

f92f0bd

Honry dismissed fdwr’s stale review via f92f0bd March 7, 2025 05:35

Honry force-pushed the int64 branch from d9b9f08 to f92f0bd Compare March 7, 2025 05:35

fdwr approved these changes Mar 7, 2025

View reviewed changes

		public shouldConvertInt64toInt32: boolean = false;
		public isInt64ToInt32Converted: boolean = false;

[WebNN] Better int64 integration #23831

Are you sure you want to change the base?

[WebNN] Better int64 integration #23831

Conversation

Honry commented Feb 27, 2025 • edited by fdwr Loading

Honry commented Feb 27, 2025

guschmue commented Feb 27, 2025

guschmue commented Feb 27, 2025

guschmue commented Feb 27, 2025

azure-pipelines bot commented Feb 27, 2025

guschmue commented Feb 27, 2025

azure-pipelines bot commented Feb 27, 2025

azure-pipelines bot commented Feb 27, 2025

azure-pipelines bot commented Feb 27, 2025

fdwr left a comment

Choose a reason for hiding this comment

fdwr Mar 4, 2025

Choose a reason for hiding this comment

fdwr Mar 4, 2025 • edited Loading

Choose a reason for hiding this comment

Honry Mar 4, 2025

Choose a reason for hiding this comment

fdwr Mar 5, 2025

Choose a reason for hiding this comment

Honry left a comment

Choose a reason for hiding this comment

Honry Mar 4, 2025

Choose a reason for hiding this comment

fdwr left a comment • edited Loading

Choose a reason for hiding this comment

fdwr Mar 5, 2025

Choose a reason for hiding this comment

fdwr commented Mar 5, 2025

fdwr commented Mar 5, 2025

fdwr commented Mar 5, 2025

fdwr commented Mar 5, 2025

azure-pipelines bot commented Mar 5, 2025

azure-pipelines bot commented Mar 5, 2025

azure-pipelines bot commented Mar 5, 2025

azure-pipelines bot commented Mar 5, 2025

fdwr commented Mar 6, 2025

azure-pipelines bot commented Mar 6, 2025

fdwr commented Mar 6, 2025 • edited Loading

Honry commented Mar 7, 2025

fdwr left a comment

Choose a reason for hiding this comment

fdwr commented Mar 7, 2025

fdwr commented Mar 7, 2025

fdwr commented Mar 7, 2025

fdwr commented Mar 7, 2025

azure-pipelines bot commented Mar 7, 2025

azure-pipelines bot commented Mar 7, 2025

azure-pipelines bot commented Mar 7, 2025

azure-pipelines bot commented Mar 7, 2025

Honry commented Feb 27, 2025 •

edited by fdwr

Loading

fdwr Mar 4, 2025 •

edited

Loading

fdwr left a comment •

edited

Loading

fdwr commented Mar 6, 2025 •

edited

Loading