Cherry pick telemetry changes from win-onnxruntime #24957
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Description
This change adds the following additional telemetry fields to existing ONNX Runtime telemetry events:
1. Model file name
2. Model weight data type
3. Model graph hash
4. Model weight hash
5. Framework name
Motivation and Context
In order to improve telemetry data collection for WCR + ONNX Runtime, we need to include additional information in the telemetry events.
1. The model graph and weight hash will help us better differentiate between the different models.
2. The model weight data type will help us identify if we are running quantized models.
3. The framework name is set to WinAI to help us identify the source of the telemetry events.
Testing
1. With the hashing algorithm used, we see hash calculation take between 35 – 50 microseconds when profiling SquezeNet, GoogleBert and Llama3.2.
2. Tested with the following unit tests while verifying hash reproducibility:
Case 1: Empty Graph
Case 2: Graph with Duplicate Node Names
Case 3: Graph with Nodes Having No Inputs or Outputs
Case 4: Empty Initializer Set
Case 5: Duplicate Tensor Names
Case 6: Tensors with No Data
Case 7: Large Graph with Random Nodes (10000 nodes)
Case 8: Large Initializer Set (10000 nodes)
This change generates the telemetry hash before initializing the session to ensure that the telemetry hash is generated correctly and consistently without any issues related to accessing invalid memory. The issue is seen in cases where the initializers are cleared after the session is initialized if the tensor was not pre-allocated with some other tensors in a single 'allocate' call or if the context is not to be saved. ORT does this to remove weights from the graph to save memory. Tested with OnnxRuntimeLoad, CppConsoleDesktop, CppOnnxRuntimeConsoleDesktop
This commit checks the ORT_CALLER_FRAMEWORK macro to determine if the caller is WinAI. If it is, the code will only calculate the graph and weight hash of the model. This is done to avoid unnecessary overhead when the caller is not WinAI. Tested the change with: CppModelEvaluation, OnnxRuntimeLoad, OnnxRuntimeThunkTest, WcrServices, CppConsoleDesktop, CppOnnxRuntimeConsoleDesktop
This change limits the parameters required by windows telemetry to only compile for windows builds.
ashrit-ms
force-pushed
the
ashritms/cherry-pick-telemetry-changes
branch
from
b36d04c to
bd78f5c
Compare
snnn
reviewed
Jun 13, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This change cherry-picks telemetry changes from win-onnxruntime to improve telemetry data collection for ONNX Runtime on Windows.
Motivation and Context
These changes are already present in win-onnxruntime, so cherry-picking these changes here for Windows use cases that rely on public ONNX Runtime.